You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
InMemoryOutboxCircuitBreaker: tripped topics go stale on an instance that stops obtaining the sweeper lock #4628
InMemoryOutboxCircuitBreaker (ADR 0028) trips a topic when publishing to it keeps failing, and the sweeper then leaves that topic's outstanding messages out of each sweep until the cooldown has run down.
The cooldown only advances on a sweep that obtained the distributed lock. The lock is taken and released on every sweep, so in a multi-instance deployment the instance that obtains it can change from tick to tick. An instance that stops obtaining the lock keeps its tripped topics indefinitely. When it next obtains the lock it applies those trips to its own sweeps, possibly long after the broker has recovered, so messages for those topics wait in the outbox for no current reason.
The breaker is per instance even with a durable outbox and a distributed lock, so another instance cannot see or clear these trips.
Why this is not a one-line fix
Clearing the trips when the lock is lost would disable the breaker, because the lock is released after every sweep.
Options, both of which change ADR 0028:
Expire tripped topics by time, independent of whether a sweep obtained the lock.
Share tripped state between instances, for example through the outbox or the lock store.
Context
Found while specifying the sweeper health check (#4573, spec 0038). That work reports tripped topics only while an instance is an active sweeper, which hides this staleness on inactive instances but does not expire the trips. It is listed there as out of scope.
Ordering
This follows #4573: draft PR #4627 must merge first, because the health check and the tripped-topics gauge define what an operator can see of this state. After that this needs its own issue discussion and an ADR that amends or supersedes parts of ADR 0028.
Options, both of which change ADR 0028:
Expire tripped topics by time, independent of whether a sweep obtained the lock.
Share tripped state between instances, for example through the outbox or the lock store.
Of these two options, the second may be preferable as it actually allows tripped topics to work as intended, cool down a topic across sweeper runs when it keep failing to post, and thus avoid it blocking other topics.
Activity
iancooper commentedon Oct 18, 2026
Of these two options, the second may be preferable as it actually allows tripped topics to work as intended, cool down a topic across sweeper runs when it keep failing to post, and thus avoid it blocking other topics.