Skip to content

Durable receiver hardening, part 2: batch member settlement, inbox ownership, paused local queues - #5157

Merged
jeremydmiller merged 8 commits into
mainfrom
durable-hardening-2
Oct 23, 2026
Merged

jeremydmiller merged 8 commits into
mainfrom
durable-hardening-2

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Second hardening pass from the durable receiver compliance rig (src/Testing/DurableComplianceRig, merged in #5129). Every fix below was reproduced by the rig and has a regression test that was red before the fix.

Batch members are settled as themselves (F1, B1, F2, F3, B2, F8)

  • Replay (ApplyItem, probing) reuses the original member envelopes (BatchReplay) instead of new Envelope(item). Each member is marked handled only when its own replay finishes, so a stop mid-replay no longer loses the survivors, and a probed poison member is dead-lettered under its own id instead of disappearing.
  • A failed dead-letter write is retried member by member (BatchDeadLetters) rather than escaping and marking the whole carrier handled.
  • The default buffered batch queue now settles its members' inbox rows when the batch finishes (BatchMemberInbox). A member whose dead-letter write fails is handed back to run again.
  • BatchingPendingCounts looks pipelines up by element type. MoveToErrorQueue rewrites the message type, so every dead-lettered batch used to leak its whole count; the other leak paths (native DLQ branch, handed-back members, drained buffered batches) settle too.

Durable receivers stop releasing rows still being worked on (F15, F16, F12, F19, F4, F5)

  • An unlatched drain (a pause raised inside the handler pipeline, rate limiting) waits in the background for running handlers before it releases.
  • Each runtime keeps a registry of its DurableReceivers. A rebuilt receiver skips the startup in-flight sweep while an older receiver at the same address still has work, and a drained receiver hands back only the rows it abandoned, by id, through the existing IMessageStore.ReassignIncomingAsync, when anything else at its address is busy.
  • Batch members stay in the custody of the receiver they arrived on until their batch settles: the InboxStaleTime heartbeat keeps the members' rows alive (not the row-less carrier), and a drain doesn't release them.
  • _scheduleExecution retries until the store comes back, with the same pauses An envelope whose completion could not be written stays owned by the live node until the node stops (6.41.0) #4664 gave mark-as-handled.
  • MultiTenantedMessageStore implements the existing IMessageInboxHeartbeat, restarting each envelope's clock in its own tenant database.
  • A post racing a drain is handed back by id instead of being dropped in memory.

Durable local queue pauses (F17, L1, F6)

  • EnqueueDirectlyAsync on a paused queue throws, so the caller hands the row back instead of dropping it.
  • A pause raised from inside the pipeline drains in the background (it used to wait on itself until DrainTimeout), and StartAsync waits for that drain.
  • Enqueues while paused are held and re-enqueued on resume instead of NRE-ing.

RabbitMQ requeue (N1)

  • RabbitMqListener.RequeueAsync goes through DeferAsync, which publishes the copy before it acks the original, so a queue with native dead lettering no longer dead-letters the original as well.

Not changing public API

No breaking changes; MultiTenantedMessageStore implements an existing interface, and everything else is internal.

Docs

  • docs/guide/handlers/batching.md: how batch members are settled and held by the listener they arrived on.
  • docs/guide/durability/index.md: the heartbeat covers batch members and a database per tenant.

Left open

  • F9 (a batch's attempt count is not carried to its members), low priority.
  • F18: the synchronous Dispose() still leaves an idle receiver's heartbeat timer running (pre-existing).
  • A one-round-trip window where a brand-new receiver's first receipt can be released by an older receiver's blanket release.

Local verification

Pinned wolverine.slnx build (-f net9.0) clean. CoreTests, EfCoreTests, MartenTests, Wolverine.RabbitMQ.Tests, PostgresqlTests, SqlServerTests, PolecatTests and FisherTests run on the final branch; the rig's disruption, heartbeat and batching pause/stale specs are 82/82 green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

jeremydmiller and others added 8 commits October 22, 2026 15:28
…ing worked on

F15: an unlatched drain (a pause raised from inside the handler pipeline) now
holds its release until the handlers running at that moment finish, on a
continuation, the same way GH-4797 holds the latched drain's.

F16: a rebuilt receiver's ClearInFlightIncomingAsync skips the blanket sweep
while a previous receiver at the same address still holds work, and a drained
receiver whose address is busy hands back only the rows it abandoned, by id
(ReassignIncomingAsync to any node), instead of every row at the address.
DurableReceiverRegistry (per runtime) answers "is another receiver here still
working".

F12: _scheduleExecution retries until the store comes back, as GH-4664 did for
mark-as-handled.

F19: MultiTenantedMessageStore implements IMessageInboxHeartbeat and restarts
each envelope's clock in the database that holds its row.

F4/F5: batch members stay in the custody of the receiver they arrived on until
the carrier's terminal settles them (InboxCustody.SettleBatchMembers /
TrySettleBatchMember). The heartbeat covers them, the drain skips them, and the
batch queue's drain hands back the members of a cancelled batch retry by id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…itself, or NREs

- F17: IListenerCircuit.EnqueueDirectlyAsync on a paused DurableLocalQueue returned
  silently, so a scheduled retry the poller had already claimed stayed Incoming and
  owned by a live node. It now throws, like ListeningAgent, so the caller's
  GH-3680/GH-4822 release hands the rows back to any node.
- L1: PauseAsync latched and then awaited the drain, which waits for the frame that
  raised the pause (AndPauseProcessing / PauseThenRequeue), so a 2s pause lasted
  DrainTimeout + 2s. PauseAsync now latches and drains in the background (the drain
  still waits for running handlers before releasing their rows, GH-4797);
  PauseWithDrainAsync keeps waiting. StartAsync awaits that drain before building
  the new receiver, so the old receiver's release cannot hand back rows the new one
  has taken. A second pause while paused replaces the restarter instead of leaking it.
- F6: ILocalReceiver.Enqueue/EnqueueAsync (the batching processor's carrier hand-off)
  hit a null receiver while paused. What reaches a paused queue is now held and
  enqueued into the new receiver when the queue resumes; the same applies to a
  pause landing mid-StoreAndForward / EnqueueOutgoing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
RabbitMqListener.RequeueAsync (TryRequeueAsync) nacked the original with
requeue:false before republishing. On a queue with native dead lettering (the
default) that dead-lettered the original as well as putting a copy back, and a copy
that failed to publish left the message only in the DLQ. It now goes through
RabbitMqEnvelope.DeferAsync: publish the copy, then ack, which never dead letters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
F1/B1: item isolation (ApplyItemException, IsolateBatchMembers, ProbeIndividuallyAfter)
replays the ORIGINAL member envelopes instead of new Envelope(item). The replayed
members are taken out of the failing batch first, so the batch's terminal no longer
marks them Handled while their replay exists only in memory; each is settled by its
own replay's terminal, and a probed poison member is dead lettered under its own id.

F2: a DeadLetterMoveFailedException inside ApplyItem/ProbeIndividually no longer
escapes into the pipeline's recovery (which completed the batch and marked every
member Handled). A member that could not be dead lettered is handed back, taken out
of the batch, and the survivors are still replayed. Partial whole-batch failures
settle the rest of the batch before reporting the failure.

F3/B2: a batch executing on a BUFFERED local queue now marks its members' inbox
rows handled at its terminal (BatchMemberInbox, routed through the member's own
store, retried until the store returns). A member whose dead letter write fails
there is handed back as a Scheduled, unowned row instead of being left Incoming.

F8: BatchingPendingCounts looks a batch's pipeline up by element type first;
MoveToErrorQueue rewrites MessageType before completing, so every batch dead
lettered or requeued into the error queue leaked its members' count. Batches
dropped by a drained buffered queue and handed-back members are settled too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
… memory

An enqueue that reads a DurableLocalQueue's old receiver just before the pause
swaps it out could post into a block the drain had already completed, which
drops the post without a word and strands the row owned by this node. Posts
now count themselves in; the drain waits (bounded, 1s) for posts that started
before it, so they reach the block and are skipped and abandoned by the latch,
and a post that sees the drain is abandoned directly. Either way the row is
handed back by id.

Tests also pin the same-queue batching topology: a pause of the queue that
batches its own items keeps the members held, and a carrier parked while
paused and run by the resumed receiver still settles them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…InBatch

The batching lane settles members of a batch on the buffered queue in
BatchMemberInbox, which clears InBatch but never calls InboxCustody. A held
member whose InBatch has been cleared is now treated as settled, so the receiver
it arrived on stops holding it there too. Otherwise the address would look busy
for good and its drains could never use the blanket release again. An abandoned
carrier (a latched defer, a cancelled batch retry) clears InBatch on its members
as it settles their pending counts and hands their rows back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit c0f1c78 into main Oct 23, 2026
48 checks passed
This was referenced Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant