Repository navigation
test: make the leader-gate tests wait on a signal, not a delay - #1207
Conversation
…e leader-gate tests On .NET 10, BackgroundService.StartAsync returns before ExecuteAsync runs its first iteration, so a fixed delay after StartAsync raced a starved thread pool and the leader did nothing before the test stopped it (#1202). Both tests now wait for the stub lease's second acquisition, which the sequential loop reaches only after the first iteration's work is done. The OAuth purge tests also move onto a host without its own DurableJobWorker, whose Postgres-leased poll swept the same rows and could delete them during the follower run. NoHostWorkerFactory replaces the identical IdempotencyPurgeSweepFactory.
|
Review round 1 (Codex, read-only) at The diagnosis holds. Both leader tests inject a lease that always returns Finding 1 (test-only, P2): a signal timeout leaves the worker running. VERDICT: 0 product defects, 1 test-only finding. |
A timed-out wait still fails the test, but the worker it started would otherwise keep polling into later tests of the same fixture.
|
Fixed in 64ee166: the signal wait sits in Shown with a mutation that never signals (acquisition threshold set to
Mutation and scratch reverted; the three touched classes pass 16 of 16 at the head. |
Closes #1202
Cause: test timing, not a gate race
Both CI failures (run 38028393782 attempt 1 on #1172, run 38031633308 attempt 1 on #1175) are the same shape: the test started the worker with a stub lease answering
Leader, slept a fixed delay, stopped the worker, and then found the first iteration had done nothing.OAuthPurgeTests.Sweep_RunsOnlyOnTheLeader:the leader did not run the OAuth sweepafter a 500 ms sleep.DurableJobWorkerLeaderGateTests.Leader_Polls:Creates > 0false after a 120 ms sleep. The test took 999 ms, so the runner was starved.On .NET 10,
BackgroundService.StartAsyncreturns beforeExecuteAsyncruns its first iteration. A probe asserting the scope count right afterawait StartAsyncread 0 with the loop still pending. The first poll is therefore a thread-pool work item scheduled after the sleep starts, and a starved pool can run it after the sleep ends andStopAsynchas cancelled it. The gate itself is deterministic given the stub, so there is no product change.A second, independent race in the OAuth test: the test host runs the real
DurableJobWorkerunder the Postgres lease, and its own poll runsOAuthPurgeSweepon the same rows. With the test's worker delayed, the follower assertion failed witha follower ran the OAuth sweepbecause the host's worker had deleted the row.IdempotencyRecordPurgeSweepTestsalready removes the host worker for the same reason; that factory is nowNoHostWorkerFactoryand the OAuth purge tests use it too.Fix
SignallingLeaseanswers with a scripted status and completes aTaskCompletionSourceon the second acquisition. The loop is sequential, so the second acquisition proves the first iteration (poll, sweeps or heartbeat stamp) has finished. Each test starts the worker, awaits that signal, then stops the worker. The 30 sWaitAsyncbounds a hung worker; it never paces the test.Proof
Injected-delay mutation:
await Task.Delay(1000, stoppingToken)at the top ofExecuteAsync, so the first iteration arrives late.Leader_Polls(same assertion as CI),Follower_NeverPolls_ButStampsHeartbeat;Sweep_RunsOnlyOnTheLeaderred on the follower assertion from the host worker's sweepSignal-timeout mutation (lease never signals, wait shortened to 2 s): every gate test fails with
TimeoutExceptionon both helpers. A scratch test asserting the worker'sExecuteTaskis completed afterwards fails on the helper in 74ef602 (the worker is still running after the timeout) and passes on 64ee166, where the wait sits intry/finallywith the boundedStopAsyncandDispose.Looped runs of the fixed tests (
Sweep_RunsOnlyOnTheLeader+ the four gate tests, serial, each run its owndotnet teston the Testcontainers cluster):Pre-fix rate under the same pressure: pending, this body will be updated.
Tests run locally
dotnet test tests/Cluckwork.Api.IntegrationTests --filter "FullyQualifiedName~OAuthPurgeTests|FullyQualifiedName~DurableJobWorkerLeaderGateTests|FullyQualifiedName~IdempotencyRecordPurgeSweepTests": 16 passed.Not changed:
DurableJobWorkerResilienceTests.HostedLoop_SurvivesContinuousFailures_AndStopsPromptlysleeps the same way but asserts only that the loop is not faulted, which starvation cannot make fail.