Skip to content

Serialize ProjectionCoordinatorBase lifecycle calls so overlapping stops cannot race (#1148) - #1149

Merged
jeremydmiller merged 2 commits into
mainfrom
fix/1148-serialize-coordinator-lifecycle
Nov 9, 2026
Merged

jeremydmiller merged 2 commits into
mainfrom
fix/1148-serialize-coordinator-lifecycle

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #1148.

Problem

StartAsync, PauseAsync and StopAsync read the shared _cancellation and _runner fields, await, then dispose and replace them, with nothing serializing callers. The fixes for #574 and #592 made calls that run one after another safe, but calls that overlap still race:

  • Overlapping stops throw ObjectDisposedException: one caller disposes the CancellationTokenSource while another is about to CancelAsync() it. This happens in practice because RunCommand calls host.StopAsync again when ApplicationStopping fires, so a WebApplicationFactory/Alba teardown has two stops in flight. Marten's ProjectionCoordinator.DisposeAsync is a third way into StopAsync.
  • Overlapping stops also tear down twice. Both callers run StopAllAsync and SafeDispose over the same daemons and call ReleaseAllLocks concurrently.
  • Overlapping starts each build a runner, and the second overwrites the first, leaving a leadership loop nothing will ever cancel. The runner also read _cancellation lazily inside Task.Run, so it could see a field another caller had already cleared (NullReferenceException).

Fix

  • A SemaphoreSlim(1, 1) is held across each whole lifecycle transition. The public methods take it and call private unlocked versions, because StopAsync calls PauseAsync internally. Overlapping stops now behave like sequential ones: the first tears down and the rest find nothing left.
  • The leadership loop never takes the semaphore, so draining the runner while holding it cannot deadlock.
  • StopAsync deliberately doesn't use its token for the wait, so a shutdown always completes its teardown. StartAsync does use its token.
  • The runner captures its token source in a local instead of reading the field.

Tests

Two new tests in ProjectionCoordinatorBaseTests:

  • overlapping_stops_and_pauses_never_throw_and_tear_down_once: 32 concurrent Stop/Pause callers over 200 rounds. Nothing may throw, and each daemon is disposed exactly once.
  • overlapping_resumes_and_stops_never_orphan_a_leadership_loop: 16 concurrent Resume/Stop callers over 30 rounds. Nothing may throw, and no loop may still be polling for locks after the final stop.

Both fail on main, on the double dispose and the NullReferenceException. Each has a 10-second timeout because the unserialized code can also deadlock: the first run of these tests on main hung instead of failing. With the fix, the issue's console repro runs all 10,000 rounds with 2 and with 32 callers without failing, and EventTests passes in full (1,834 tests) on net9.0 and net10.0.

Downstream

JasperFx/marten#5842 tracks the same overlap in Marten's ExplicitProjectionCoordinator, which doesn't derive from this base. Marten's ProjectionCoordinator gets this fix through the JasperFx version bump.

🤖 Generated with Claude Code

…ops cannot race (#1148)

StartAsync, PauseAsync and StopAsync read, await across, then dispose and
replace the shared _cancellation/_runner fields with no synchronization. Two
overlapping StopAsync calls (RunCommand stops the host again when
ApplicationStopping fires, so a WebApplicationFactory/Alba teardown has two in
flight) could CancelAsync a CancellationTokenSource the other caller had just
disposed. Overlapping StartAsync calls could also each build a runner, orphaning
one, and the runner read _cancellation lazily, so it could see a cleared field.

A SemaphoreSlim now serializes each whole lifecycle transition, so concurrent
stops behave like sequential ones: the first tears down, the rest find nothing
left. That also keeps the daemon fan-out, disposal and ReleaseAllLocks from
running twice at once. The runner captures its token source locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harden the null-threshold HighWater test

The Windows CI run on #1149 failed HighWaterAgent_progress_stall_Tests.
a_null_threshold_disables_the_probe: the stall probe was off (CeilingCalls == 0),
but the separate staleness watchdog restarted the loop because one 10ms polling
cycle missed its 200ms threshold on a loaded runner. xunit runs test classes in
parallel within an assembly, so the new coordinator overlap tests, which start
thousands of thread-pool tasks, were competing with that timing-sensitive test.

- The null-threshold test now uses a one-minute staleness threshold, so only
  the stall probe could restart the loop, which is what it checks.
- New src/StressTests project, serialized with
  [assembly: CollectionBehavior(CollectionPerAssembly)], with its own Nuke
  target (test-stress, part of test) and a step in both CI jobs. It holds the
  tests that drive many threads or many rounds of a race:
    * the two #1148 coordinator overlap tests (from EventTests)
    * ShardStateTracker publish/wait race and concurrent subscribers (EventTests)
    * RecentlyUsedAggregateWriteCache concurrent takers (EventTests)
    * RecentlyUsedCache concurrent store, and store + compact (CoreTests)
    * InMemoryDocumentStore concurrent units of work (EventStoreTests)
  CallingAssemblyTests.find_is_safe_to_call_concurrently stays in CoreTests:
  CallingAssembly.Find walks the stack to the test assembly, so moving it
  would change what it tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller

Copy link
Copy Markdown
Member Author

The Windows job failed HighWaterAgent_progress_stall_Tests.a_null_threshold_disables_the_probe on net9.0. The stall probe really was off (its CeilingCalls == 0 assertion passed). The Restarted came from the separate staleness watchdog: one 10ms polling cycle missed its 200ms threshold on a loaded runner. xunit runs EventTests' classes in parallel, so the new coordinator overlap tests were competing with that test for thread-pool threads.

Pushed 6e3084c:

  • Hardened the test. a_null_threshold_disables_the_probe now uses a one-minute staleness threshold, so the stall probe is the only thing that could restart the loop, which is what the test checks.
  • New src/StressTests project. It runs its tests one at a time ([assembly: CollectionBehavior(CollectionPerAssembly)]), with its own test-stress Nuke target (part of test) and a step in both CI jobs. It holds the tests that drive many threads or many rounds of a race:
  • One concurrent test stays put. CallingAssemblyTests.find_is_safe_to_call_concurrently is still in CoreTests, because CallingAssembly.Find walks the stack to the test assembly and moving it would change what it tests.

Locally, StressTests (8), EventTests (1,829), CoreTests and EventStoreTests all pass on net9.0 and net10.0, and ./build.sh test-stress runs green.

🤖 Generated with Claude Code

@jeremydmiller
jeremydmiller merged commit da37356 into main Nov 9, 2026
3 checks passed
jeremydmiller added a commit to JasperFx/wolverine that referenced this pull request Nov 9, 2026
* Bump JasperFx to 2.101.1

Picks up JasperFx/jasperfx#1149: ProjectionCoordinatorBase lifecycle calls are
serialized so overlapping stops cannot race (#1148). JasperFx,
JasperFx.Events, JasperFx.Events.SourceGenerator, JasperFx.Events.InMemory and
JasperFx.SourceGenerator move together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bump JasperFx to 2.102.0

2.102.0 adds IEventDatabase.FetchFirstEventTimestampsAfterAsync (JasperFx/jasperfx#1151,
#1150) as a default interface member that throws NotSupportedException, so
stores that have not implemented it yet still compile and load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProjectionCoordinatorBase: overlapping StopAsync/PauseAsync calls throw ObjectDisposedException from stopRunnerAsync

1 participant