Repository navigation
Serialize ProjectionCoordinatorBase lifecycle calls so overlapping stops cannot race (#1148) - #1149
Merged
Merged
Conversation
…ops cannot race (#1148) StartAsync, PauseAsync and StopAsync read, await across, then dispose and replace the shared _cancellation/_runner fields with no synchronization. Two overlapping StopAsync calls (RunCommand stops the host again when ApplicationStopping fires, so a WebApplicationFactory/Alba teardown has two in flight) could CancelAsync a CancellationTokenSource the other caller had just disposed. Overlapping StartAsync calls could also each build a runner, orphaning one, and the runner read _cancellation lazily, so it could see a cleared field. A SemaphoreSlim now serializes each whole lifecycle transition, so concurrent stops behave like sequential ones: the first tears down, the rest find nothing left. That also keeps the daemon fan-out, disposal and ReleaseAllLocks from running twice at once. The runner captures its token source locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harden the null-threshold HighWater test The Windows CI run on #1149 failed HighWaterAgent_progress_stall_Tests. a_null_threshold_disables_the_probe: the stall probe was off (CeilingCalls == 0), but the separate staleness watchdog restarted the loop because one 10ms polling cycle missed its 200ms threshold on a loaded runner. xunit runs test classes in parallel within an assembly, so the new coordinator overlap tests, which start thousands of thread-pool tasks, were competing with that timing-sensitive test. - The null-threshold test now uses a one-minute staleness threshold, so only the stall probe could restart the loop, which is what it checks. - New src/StressTests project, serialized with [assembly: CollectionBehavior(CollectionPerAssembly)], with its own Nuke target (test-stress, part of test) and a step in both CI jobs. It holds the tests that drive many threads or many rounds of a race: * the two #1148 coordinator overlap tests (from EventTests) * ShardStateTracker publish/wait race and concurrent subscribers (EventTests) * RecentlyUsedAggregateWriteCache concurrent takers (EventTests) * RecentlyUsedCache concurrent store, and store + compact (CoreTests) * InMemoryDocumentStore concurrent units of work (EventStoreTests) CallingAssemblyTests.find_is_safe_to_call_concurrently stays in CoreTests: CallingAssembly.Find walks the stack to the test assembly, so moving it would change what it tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
The Windows job failed Pushed 6e3084c:
Locally, StressTests (8), EventTests (1,829), CoreTests and EventStoreTests all pass on net9.0 and net10.0, and 🤖 Generated with Claude Code |
jeremydmiller
added a commit
to JasperFx/wolverine
that referenced
this pull request
Nov 9, 2026
* Bump JasperFx to 2.101.1 Picks up JasperFx/jasperfx#1149: ProjectionCoordinatorBase lifecycle calls are serialized so overlapping stops cannot race (#1148). JasperFx, JasperFx.Events, JasperFx.Events.SourceGenerator, JasperFx.Events.InMemory and JasperFx.SourceGenerator move together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bump JasperFx to 2.102.0 2.102.0 adds IEventDatabase.FetchFirstEventTimestampsAfterAsync (JasperFx/jasperfx#1151, #1150) as a default interface member that throws NotSupportedException, so stores that have not implemented it yet still compile and load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 10, 2026
Closed
This was referenced Nov 18, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1148.
Problem
StartAsync,PauseAsyncandStopAsyncread the shared_cancellationand_runnerfields, await, then dispose and replace them, with nothing serializing callers. The fixes for #574 and #592 made calls that run one after another safe, but calls that overlap still race:ObjectDisposedException: one caller disposes theCancellationTokenSourcewhile another is about toCancelAsync()it. This happens in practice becauseRunCommandcallshost.StopAsyncagain whenApplicationStoppingfires, so aWebApplicationFactory/Alba teardown has two stops in flight. Marten'sProjectionCoordinator.DisposeAsyncis a third way intoStopAsync.StopAllAsyncandSafeDisposeover the same daemons and callReleaseAllLocksconcurrently._cancellationlazily insideTask.Run, so it could see a field another caller had already cleared (NullReferenceException).Fix
SemaphoreSlim(1, 1)is held across each whole lifecycle transition. The public methods take it and call private unlocked versions, becauseStopAsynccallsPauseAsyncinternally. Overlapping stops now behave like sequential ones: the first tears down and the rest find nothing left.StopAsyncdeliberately doesn't use its token for the wait, so a shutdown always completes its teardown.StartAsyncdoes use its token.Tests
Two new tests in
ProjectionCoordinatorBaseTests:overlapping_stops_and_pauses_never_throw_and_tear_down_once: 32 concurrent Stop/Pause callers over 200 rounds. Nothing may throw, and each daemon is disposed exactly once.overlapping_resumes_and_stops_never_orphan_a_leadership_loop: 16 concurrent Resume/Stop callers over 30 rounds. Nothing may throw, and no loop may still be polling for locks after the final stop.Both fail on
main, on the double dispose and theNullReferenceException. Each has a 10-second timeout because the unserialized code can also deadlock: the first run of these tests onmainhung instead of failing. With the fix, the issue's console repro runs all 10,000 rounds with 2 and with 32 callers without failing, andEventTestspasses in full (1,834 tests) on net9.0 and net10.0.Downstream
JasperFx/marten#5842 tracks the same overlap in Marten's
ExplicitProjectionCoordinator, which doesn't derive from this base. Marten'sProjectionCoordinatorgets this fix through the JasperFx version bump.🤖 Generated with Claude Code