Repository navigation
Durable receiver hardening, part 2: batch member settlement, inbox ownership, paused local queues - #5157
Merged
Conversation
…ing worked on F15: an unlatched drain (a pause raised from inside the handler pipeline) now holds its release until the handlers running at that moment finish, on a continuation, the same way GH-4797 holds the latched drain's. F16: a rebuilt receiver's ClearInFlightIncomingAsync skips the blanket sweep while a previous receiver at the same address still holds work, and a drained receiver whose address is busy hands back only the rows it abandoned, by id (ReassignIncomingAsync to any node), instead of every row at the address. DurableReceiverRegistry (per runtime) answers "is another receiver here still working". F12: _scheduleExecution retries until the store comes back, as GH-4664 did for mark-as-handled. F19: MultiTenantedMessageStore implements IMessageInboxHeartbeat and restarts each envelope's clock in the database that holds its row. F4/F5: batch members stay in the custody of the receiver they arrived on until the carrier's terminal settles them (InboxCustody.SettleBatchMembers / TrySettleBatchMember). The heartbeat covers them, the drain skips them, and the batch queue's drain hands back the members of a cancelled batch retry by id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…itself, or NREs - F17: IListenerCircuit.EnqueueDirectlyAsync on a paused DurableLocalQueue returned silently, so a scheduled retry the poller had already claimed stayed Incoming and owned by a live node. It now throws, like ListeningAgent, so the caller's GH-3680/GH-4822 release hands the rows back to any node. - L1: PauseAsync latched and then awaited the drain, which waits for the frame that raised the pause (AndPauseProcessing / PauseThenRequeue), so a 2s pause lasted DrainTimeout + 2s. PauseAsync now latches and drains in the background (the drain still waits for running handlers before releasing their rows, GH-4797); PauseWithDrainAsync keeps waiting. StartAsync awaits that drain before building the new receiver, so the old receiver's release cannot hand back rows the new one has taken. A second pause while paused replaces the restarter instead of leaking it. - F6: ILocalReceiver.Enqueue/EnqueueAsync (the batching processor's carrier hand-off) hit a null receiver while paused. What reaches a paused queue is now held and enqueued into the new receiver when the queue resumes; the same applies to a pause landing mid-StoreAndForward / EnqueueOutgoing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
RabbitMqListener.RequeueAsync (TryRequeueAsync) nacked the original with requeue:false before republishing. On a queue with native dead lettering (the default) that dead-lettered the original as well as putting a copy back, and a copy that failed to publish left the message only in the DLQ. It now goes through RabbitMqEnvelope.DeferAsync: publish the copy, then ack, which never dead letters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
F1/B1: item isolation (ApplyItemException, IsolateBatchMembers, ProbeIndividuallyAfter) replays the ORIGINAL member envelopes instead of new Envelope(item). The replayed members are taken out of the failing batch first, so the batch's terminal no longer marks them Handled while their replay exists only in memory; each is settled by its own replay's terminal, and a probed poison member is dead lettered under its own id. F2: a DeadLetterMoveFailedException inside ApplyItem/ProbeIndividually no longer escapes into the pipeline's recovery (which completed the batch and marked every member Handled). A member that could not be dead lettered is handed back, taken out of the batch, and the survivors are still replayed. Partial whole-batch failures settle the rest of the batch before reporting the failure. F3/B2: a batch executing on a BUFFERED local queue now marks its members' inbox rows handled at its terminal (BatchMemberInbox, routed through the member's own store, retried until the store returns). A member whose dead letter write fails there is handed back as a Scheduled, unowned row instead of being left Incoming. F8: BatchingPendingCounts looks a batch's pipeline up by element type first; MoveToErrorQueue rewrites MessageType before completing, so every batch dead lettered or requeued into the error queue leaked its members' count. Batches dropped by a drained buffered queue and handed-back members are settled too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
… memory An enqueue that reads a DurableLocalQueue's old receiver just before the pause swaps it out could post into a block the drain had already completed, which drops the post without a word and strands the row owned by this node. Posts now count themselves in; the drain waits (bounded, 1s) for posts that started before it, so they reach the block and are skipped and abandoned by the latch, and a post that sees the drain is abandoned directly. Either way the row is handed back by id. Tests also pin the same-queue batching topology: a pause of the queue that batches its own items keeps the members held, and a carrier parked while paused and run by the resumed receiver still settles them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
…InBatch The batching lane settles members of a batch on the buffered queue in BatchMemberInbox, which clears InBatch but never calls InboxCustody. A held member whose InBatch has been cleared is now treated as settled, so the receiver it arrived on stops holding it there too. Otherwise the address would look busy for good and its drains could never use the blanket release again. An abandoned carrier (a latched defer, a cancelled batch retry) clears InBatch on its members as it settles their pending counts and hands their rows back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
This was referenced Oct 23, 2026
This was referenced Nov 1, 2026
This was referenced Nov 9, 2026
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second hardening pass from the durable receiver compliance rig (
src/Testing/DurableComplianceRig, merged in #5129). Every fix below was reproduced by the rig and has a regression test that was red before the fix.Batch members are settled as themselves (F1, B1, F2, F3, B2, F8)
ApplyItem, probing) reuses the original member envelopes (BatchReplay) instead ofnew Envelope(item). Each member is marked handled only when its own replay finishes, so a stop mid-replay no longer loses the survivors, and a probed poison member is dead-lettered under its own id instead of disappearing.BatchDeadLetters) rather than escaping and marking the whole carrier handled.BatchMemberInbox). A member whose dead-letter write fails is handed back to run again.BatchingPendingCountslooks pipelines up by element type.MoveToErrorQueuerewrites the message type, so every dead-lettered batch used to leak its whole count; the other leak paths (native DLQ branch, handed-back members, drained buffered batches) settle too.Durable receivers stop releasing rows still being worked on (F15, F16, F12, F19, F4, F5)
DurableReceivers. A rebuilt receiver skips the startup in-flight sweep while an older receiver at the same address still has work, and a drained receiver hands back only the rows it abandoned, by id, through the existingIMessageStore.ReassignIncomingAsync, when anything else at its address is busy.InboxStaleTimeheartbeat keeps the members' rows alive (not the row-less carrier), and a drain doesn't release them._scheduleExecutionretries until the store comes back, with the same pauses An envelope whose completion could not be written stays owned by the live node until the node stops (6.41.0) #4664 gave mark-as-handled.MultiTenantedMessageStoreimplements the existingIMessageInboxHeartbeat, restarting each envelope's clock in its own tenant database.Durable local queue pauses (F17, L1, F6)
EnqueueDirectlyAsyncon a paused queue throws, so the caller hands the row back instead of dropping it.DrainTimeout), andStartAsyncwaits for that drain.RabbitMQ requeue (N1)
RabbitMqListener.RequeueAsyncgoes throughDeferAsync, which publishes the copy before it acks the original, so a queue with native dead lettering no longer dead-letters the original as well.Not changing public API
No breaking changes;
MultiTenantedMessageStoreimplements an existing interface, and everything else is internal.Docs
docs/guide/handlers/batching.md: how batch members are settled and held by the listener they arrived on.docs/guide/durability/index.md: the heartbeat covers batch members and a database per tenant.Left open
Dispose()still leaves an idle receiver's heartbeat timer running (pre-existing).Local verification
Pinned
wolverine.slnxbuild (-f net9.0) clean. CoreTests, EfCoreTests, MartenTests, Wolverine.RabbitMQ.Tests, PostgresqlTests, SqlServerTests, PolecatTests and FisherTests run on the final branch; the rig's disruption, heartbeat and batching pause/stale specs are 82/82 green.🤖 Generated with Claude Code
https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U