Skip to content

InMemoryOutboxCircuitBreaker: tripped topics go stale on an instance that stops obtaining the sweeper lock #4628

Open

Description

@iancooper

Problem

InMemoryOutboxCircuitBreaker (ADR 0028) trips a topic when publishing to it keeps failing, and the sweeper then leaves that topic's outstanding messages out of each sweep until the cooldown has run down.

The cooldown only advances on a sweep that obtained the distributed lock. The lock is taken and released on every sweep, so in a multi-instance deployment the instance that obtains it can change from tick to tick. An instance that stops obtaining the lock keeps its tripped topics indefinitely. When it next obtains the lock it applies those trips to its own sweeps, possibly long after the broker has recovered, so messages for those topics wait in the outbox for no current reason.

The breaker is per instance even with a durable outbox and a distributed lock, so another instance cannot see or clear these trips.

Why this is not a one-line fix

Clearing the trips when the lock is lost would disable the breaker, because the lock is released after every sweep.

Options, both of which change ADR 0028:

  • Expire tripped topics by time, independent of whether a sweep obtained the lock.
  • Share tripped state between instances, for example through the outbox or the lock store.

Context

Found while specifying the sweeper health check (#4573, spec 0038). That work reports tripped topics only while an instance is an active sweeper, which hides this staleness on inactive instances but does not expire the trips. It is listed there as out of scope.

Ordering

This follows #4573: draft PR #4627 must merge first, because the health check and the tripped-topics gauge define what an operator can see of this state. After that this needs its own issue discussion and an ADR that amends or supersedes parts of ADR 0028.

Activity

  1. self-assigned this
    on Oct 18, 2026
  2. iancooper commented on Oct 18, 2026

    @iancooper
    MemberAuthor

    Options, both of which change ADR 0028:
    Expire tripped topics by time, independent of whether a sweep obtained the lock.
    Share tripped state between instances, for example through the outbox or the lock store.

    Of these two options, the second may be preferable as it actually allows tripped topics to work as intended, cool down a topic across sweeper runs when it keep failing to post, and thus avoid it blocking other topics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions