Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip all known flaky tests #6634

Merged
merged 4 commits into from
Oct 10, 2024
Merged

Skip all known flaky tests #6634

merged 4 commits into from
Oct 10, 2024

Conversation

bergundy
Copy link
Member

@bergundy bergundy commented Oct 9, 2024 •
edited
Loading

What changed?

Skipped all known flaky tests.

go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchChildWorkflow
go.temporal.io/server/tests TestWorkflowTestSuite/TestWorkflowTaskAndActivityTaskTimeoutsWorkflow
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchCron
go.temporal.io/server/tests TestHttpApiTestSuite/TestHTTPAPIHeaders
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchActivityFailOld
go.temporal.io/server/tests/xdc TestNexusStateReplicationTestSuite/TestNexusOperationEventsReplicated
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchActivityTimeoutOld
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchContinueAsNewUpgrade
go.temporal.io/server/tests TestMaxBufferedEventSuite/TestBufferedEventsMutableStateSizeLimit
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestWorkflowTaskRedirectInRetryFirstTask
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchActivityOld
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchContinueAsNewOld
go.temporal.io/server/tests TestVersioningFunctionalSuite/estDispatchActivityFailOld/ForceTaskForwardNoPollForwardForceAsync
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchCronOld
go.temporal.io/server/tests/xdc TestHistoryReplicationSignalsAndUpdatesTestSuite/TestAcceptedUpdateCanBeCompletedAfterFailoverAndFailback
go.temporal.io/server/tests TestDescribeTaskQueueSuite/TestAddSingleTask_ValidateStats
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestIndependentActivityTaskAssignment_SyncMatch_VersionedWorkflow
go.temporal.io/server/tests/xdc TestFuncClustersTestSuite/TestCronWorkflowCompleteAndFailover
go.temporal.io/server/tests/xdc TestHistoryReplicationSignalsAndUpdatesTestSuite/TestConflictResolutionDoesNotReapplyAcceptedUpdateWithConflictingId
go.temporal.io/server/tests TestVersioningFunctionalSuite/TestDispatchUpgradeWait
go.temporal.io/server/tests/xdc TestFuncClustersTestSuite/TestCronWorkflowStartAndFailover
go.temporal.io/server/tests TestActivityTestSuite/TestUpdateWorkflow_ContinueAsNewAfterUpdateAdmitted
go.temporal.io/server/tests TestActivityTestSuite/TestClientDataConverter_WithChild
go.temporal.io/server/tests TestAcquireShard_EventualSuccess/TestEventuallySucceeds
TestVersioningFunctionalSuite/TestDispatchChildWorkflowOld
TestVersioningFunctionalSuite/TestIndependentActivityTaskAssignment_Spooled_VersionedWorkflow

Also increased TEST_TIMEOUT in the Makefile to 30m to allow XDC tests to span the entire allowed time specified in run-tests.yml.

Why?

Make CI stable again. We'll need to triage reenabling and fixing these tests over time.

How did you test it?

How did you test it?!

@bergundy bergundy requested a review from a team as a code owner October 9, 2024 20:31
@bergundy bergundy merged commit 2b0b647 into temporalio:main Oct 10, 2024
47 checks passed
@bergundy bergundy deleted the flaky-test-skips branch October 10, 2024 00:58
pdoerner
pdoerner reviewed Oct 10, 2024
View reviewed changes
Makefile
TEST_TIMEOUT ?= 25m
# 30 minutes is the upper bound defined for all tests, the longer running ones at the time of writing are XDC tests. The
# 30 minute timeout is also defined in run-tests.yml
TEST_TIMEOUT ?= 30m
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually wanted to leave this timeout lower than the timeout defined in run-tests.yml

If the GHA times out while the tests are still running it will not produce any output logs and we have no way of telling which tests are taking longer than expected.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave some grace time in #6648

bergundy added a commit to bergundy/temporal that referenced this pull request Oct 11, 2024
@bergundy
Disable more flaky tests
5d93b51 
Follow up to temporalio#6634, the following tests are still flaky:

```
TestTaskWriterShutdown
TestVersioningFunctionalSuite/TestDispatchRetry
TestUpdateWorkflowSdkSuite/TestUpdateWorkflow_TimeoutWorkflowAfterUpdateAccepted
```

Also increased XDC timeout to 40 minutes since those tests occasionally
time out.
@bergundy bergundy mentioned this pull request Oct 11, 2024
Open
bergundy added a commit to bergundy/temporal that referenced this pull request Oct 11, 2024
@bergundy
Disable more flaky tests
06dca10 
Follow up to temporalio#6634, the following tests are still flaky:

```
TestTaskWriterShutdown
TestVersioningFunctionalSuite/TestDispatchRetry
TestUpdateWorkflowSdkSuite/TestUpdateWorkflow_TimeoutWorkflowAfterUpdateAccepted
```

Also increased XDC timeout to 40 minutes since those tests occasionally
time out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdoerner pdoerner pdoerner left review comments

@dnr dnr dnr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bergundy @dnr @pdoerner