From 94527b8c2ceb334a4c9a010a119fbea83277f5e1 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 17:36:57 +0100 Subject: [PATCH 01/36] docs: add ADR 0081 for TimedOutboxSweeper health metrics TimedOutboxSweeper records tick lag, sweep duration and sweeps by outcome directly through a new IAmABrighterSweeperMeter role, so a stalled or late sweeper is visible when traces are sampled out. Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- ...081-timed-outbox-sweeper-health-metrics.md | 297 ++++++++++++++++++ docs/adr/index.md | 3 +- 2 files changed, 299 insertions(+), 1 deletion(-) create mode 100644 docs/adr/0081-timed-outbox-sweeper-health-metrics.md diff --git a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md new file mode 100644 index 0000000000..3dc5152a4f --- /dev/null +++ b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md @@ -0,0 +1,297 @@ +--- +id: 0081-timed-outbox-sweeper-health-metrics +title: "TimedOutboxSweeper health metrics" +status: Accepted +author: + - "Ian Cooper" +created: 2026-10-08 +summary: "TimedOutboxSweeper records three instruments directly through a new IAmABrighterSweeperMeter role (no-op by default, registered by AddBrighterInstrumentation): tick lag, sweep duration, and sweeps by outcome (completed, failed, lock_unavailable), so a stalled or late sweeper is visible even when traces are sampled out." +tags: + - "observability" + - "metrics" + - "outbox" + - "otel" +--- + +# 81. TimedOutboxSweeper health metrics + +Date: 2026-10-08 + +## Status + +Accepted + +## Context + +`TimedOutboxSweeper` re-sends outbox messages that were never confirmed as dispatched. When the +sweeper stops running, or runs late, messages wait in the outbox. With the `InMemoryOutbox`, those +messages are lost if the process restarts. Today an operator cannot see that this is happening. +The only signals are log lines, and the sweep's trace span, which production sampling usually drops. + +### Scope + +- **Parent issue**: [#4560](https://github.com/BrighterCommand/Brighter/issues/4560). The bugfix + record is `bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md`. +- **In scope** + - Three instruments that report the sweeper's own health: how late each sweep starts, how long + each sweep takes, and how each sweep ends. + - The role that records them, its default no-op implementation, and how it is registered. +- **Out of scope** + - How many messages a sweep found or cleared. Draft PR #4484 (proposed ADR "Outbox metrics") + counts cleared messages with a `clear_source` of `sweeper`. This ADR does not count messages + again. + - Thread-pool health (queue length, thread count). The .NET `System.Runtime` meter already + publishes those. Brighter documents how to turn it on, and does not re-export it. + - How the sweeper schedules its sweeps. The #4560 bugfix moves sweeps onto a dedicated thread. + This ADR only defines what the sweeper measures on that schedule. + - The outstanding-message check. Issue #4554 owns it. + +### Silent stalls + +| What went wrong | What an operator sees today | +| --- | --- | +| Blocked pool threads delay each sweep by seconds (measured 3–4.5 s on a 1 s interval) | Nothing, unless they compare log timestamps by hand | +| One sweep hangs on a slow broker, and every later tick gives up because the lock is held | A Warning log line on every tick | +| A sweep throws | An Error log line. Before #4560, the process crashed | +| The sweeper never starts, or starts once and stops (for example, `TimerInterval = 0`) | Nothing | + +In each of these cases, the outbox grows and nothing in the metrics shows why. + +### The forces + +- Sweeper health must be visible when tracing is off or unsampled. Metrics derived from spans + (ADR 0022) do not meet that need. +- The sweeper must not depend on OpenTelemetry. `Paramore.Brighter.Outbox.Hosting` must work when + no metrics are configured. +- Attribute cardinality must stay fixed and small. The sweeper runs every few seconds in every + process. +- Names follow ADR 0010. A Brighter-specific instrument uses the `paramore.brighter.*` prefix, as + `paramore.brighter.circuit_breaker.trips` does. +- These metrics must not duplicate the outbox metrics that #4484 proposes. + +## Decision + +**`TimedOutboxSweeper` records three instruments directly, through an `IAmABrighterSweeperMeter` +role: tick lag, sweep duration, and a count of sweeps by outcome.** + +The sweeper owns its schedule, so the sweeper is the only type that knows when a sweep was due and +when it started. The sweeper calls a small meter role at three points in each sweep. When metrics +are not configured, a no-op implementation keeps the cost at almost nothing. + +### The mechanism, end to end + +```mermaid +sequenceDiagram + participant Sweeper as TimedOutboxSweeper (dedicated thread) + participant Meter as IAmABrighterSweeperMeter + participant Lock as IDistributedLock + participant Mediator as IAmAnOutboxProducerMediator + + Sweeper->>Sweeper: wait until the next sweep is due (TimeProvider) + Sweeper->>Meter: RecordTickLag(started - due) + Sweeper->>Lock: ObtainLockAsync + alt lock held by another sweep + Sweeper->>Meter: RecordSweep(lock_unavailable, duration) + else lock obtained + Sweeper->>Mediator: ClearOutstandingFromOutboxAsync + alt sweep completed + Sweeper->>Meter: RecordSweep(completed, duration) + else sweep threw + Sweeper->>Meter: RecordSweep(failed, duration) + end + Sweeper->>Lock: ReleaseLockAsync + end +``` + +Three invariants follow from this sequence: + +- **Every sweep is measured exactly once.** That includes a sweep that gives up because the lock is + held, and a sweep that throws. The outcome counter therefore counts every sweep the sweeper + started. +- **Tick lag is measured before any I/O.** A slow broker therefore shows up in sweep duration, not + in tick lag. High tick lag means the sweeper could not start on time. With a dedicated thread, + that points at CPU throttling rather than the thread pool. +- **The due time comes from the schedule, not from the previous sweep.** If one sweep overruns, the + lag on the next sweep shows it. + +### Where the pieces live + +```mermaid +flowchart LR + subgraph Core["Paramore.Brighter"] + Role["IAmABrighterSweeperMeter (new)"] + Null["NullSweeperMeter (new)"] + Impl["SweeperMeter (new)"] + Conv["BrighterSemanticConventions (+ sweeper names)"] + Null --> Role + Impl --> Role + Impl --> Conv + end + subgraph Hosting["Paramore.Brighter.Outbox.Hosting"] + Sweeper["TimedOutboxSweeper (records)"] + end + subgraph Diag["Paramore.Brighter.Extensions.Diagnostics"] + Reg["AddBrighterInstrumentation (registers SweeperMeter)"] + end + Sweeper --> Role + Reg --> Impl +``` + +`TimedOutboxSweeper` depends only on the role. `AddBrighterInstrumentation` registers the real +meter. When that call is absent, the sweeper uses `NullSweeperMeter`. + +### Key Components + +#### The roles, and what each is responsible for + +| Role | Type | Responsibilities | Responsibility classifier | Collaborators | +| --- | --- | --- | --- | --- | +| Records sweeper health | `IAmABrighterSweeperMeter` (Paramore.Brighter) | Accepts a tick lag; accepts a sweep's outcome and duration | doing | `TimedOutboxSweeper` | +| Publishes sweeper instruments | `SweeperMeter` (Paramore.Brighter) | Owns the three instruments on the `Paramore.Brighter` meter; maps an outcome to its attribute value | knowing, doing | `IMeterFactory`, `BrighterSemanticConventions` | +| Records nothing | `NullSweeperMeter` (Paramore.Brighter) | Satisfies the role at no cost when metrics are not configured | doing | none | +| Measures its own sweeps | `TimedOutboxSweeper` (Outbox.Hosting) | Knows when each sweep was due and when it started; classifies each sweep's outcome | knowing, deciding | `IAmABrighterSweeperMeter`, `TimeProvider` | + +The split that matters: the sweeper **decides** what happened, and the meter only **records** it. +The meter holds no schedule state, so a second sweeper type could reuse it unchanged. + +#### `IAmABrighterSweeperMeter` + +```csharp +public interface IAmABrighterSweeperMeter +{ + void RecordTickLag(TimeSpan lag); + void RecordSweep(SweepOutcome outcome, TimeSpan duration); +} + +public enum SweepOutcome { Completed, Failed, LockUnavailable } +``` + +| Member | Input | Output | Error conditions | +| --- | --- | --- | --- | +| `RecordTickLag` | Time from when the sweep was due to when it started. A negative value is recorded as zero | none | Never throws | +| `RecordSweep` | How the sweep ended, and how long it took | none | Never throws. A metrics failure must not fail a sweep | + +#### The instruments + +All three instruments are on the existing `Paramore.Brighter` meter (`BrighterSemanticConventions.MeterName`). + +| Instrument | Type | Unit | Attributes | +| --- | --- | --- | --- | +| `paramore.brighter.outbox_sweeper.tick.lag` | `Histogram` | `s` | none | +| `paramore.brighter.outbox_sweeper.sweep.duration` | `Histogram` | `s` | `paramore.brighter.outbox_sweeper.outcome` | +| `paramore.brighter.outbox_sweeper.sweeps` | `Counter` | `{sweep}` | `paramore.brighter.outbox_sweeper.outcome` | + +`paramore.brighter.outbox_sweeper.outcome` takes one of three values: `completed`, `failed` or +`lock_unavailable`. + +#### Where each type is touched + +| Assembly | Type | Change | +| --- | --- | --- | +| Paramore.Brighter | `IAmABrighterSweeperMeter`, `SweepOutcome`, `SweeperMeter`, `NullSweeperMeter` | New | +| Paramore.Brighter | `BrighterSemanticConventions` | New constants for the instrument and attribute names | +| Paramore.Brighter.Outbox.Hosting | `TimedOutboxSweeper` | New optional constructor parameter `IAmABrighterSweeperMeter? meter = null`; records at the three points above | +| Paramore.Brighter.Extensions.Diagnostics | `BrighterMetricsBuilderExtensions` | `TryAddSingleton()` | + +The following stay **unchanged**: + +- `IAmABrighterMessagingMeter`, `IAmABrighterDbMeter` and `BrighterMetricsFromTracesProcessor`. +- The sweeper's log lines, which remain for operators who read logs. +- `OutboxProducerMediator`, because the sweeper measures the call, not what happens inside it. +- `TimedOutboxArchiver`. It could adopt the same role later, but this ADR does not change it. + +### Technology Choices + +#### Why record directly instead of deriving from spans? + +The span-derived pattern (ADR 0022) only produces a metric when a span is recorded. Sweeper spans +are high-volume and low-value, so production sampling drops most of them. A stall is visible only +if every sweep is measured. Direct instruments measure every sweep whatever the sampling is. + +#### Why a role and a null object instead of a nullable `Meter`? + +`TimedOutboxSweeper` must not reference OpenTelemetry. A null object also removes a null check from +every recording point. A test can supply the real `SweeperMeter` with an `IMeterFactory` and read +the values with a `MeterListener`. The test then needs no OpenTelemetry pipeline. + +#### Why three outcomes and no more? + +These three outcomes are the ones that need different responses. `failed` means read the logs. +`lock_unavailable` means a sweep is overrunning or hung. `completed` is the baseline. A sweep that +the mediator skipped internally is still `completed` from the sweeper's point of view. That case +can only happen with a second caller of `ClearOutstandingFromOutboxAsync`. + +### Implementation Approach + +1. **Structural:** add the names to `BrighterSemanticConventions`. Add `IAmABrighterSweeperMeter`, + `SweepOutcome` and `NullSweeperMeter`. Add the optional parameter to the `TimedOutboxSweeper` + constructor and default it to `NullSweeperMeter`. No behaviour changes. +2. **Behavioural, test first:** add `SweeperMeter` and record the outcome and duration of each + sweep. The tests use a `MeterListener` over an `IMeterFactory` from `ServiceCollection.AddMetrics()`. +3. **Behavioural, test first:** record tick lag. This step depends on the #4560 scheduling change, + which gives the sweeper a due time to measure against. +4. **Behavioural:** register `SweeperMeter` in `AddBrighterInstrumentation` (`BrighterMetricsBuilderExtensions.cs:38`). +5. **Docs:** add a sweeper section to the observability guide. It covers the three instruments, + suggested alerts, and turning on the .NET `System.Runtime` meter for thread-pool queue length + and thread count. + +## Consequences + +### Positive + +- An operator can alert on a stalled or late sweeper. Two examples: no `completed` sweeps for N + intervals, or tick-lag p99 above the interval. +- The instruments separate a slow broker (high sweep duration) from a starved process (high tick + lag). It took several experiments to tell those apart for #4560. +- Every recording point is cheap when metrics are not configured. + +### Negative + +- Brighter gains a third meter role next to `IAmABrighterMessagingMeter` and `IAmABrighterDbMeter`. + The draft outbox meter in #4484 would make a fourth. Merging the outbox meter and the sweeper + meter later would break the source of anyone who implements the role. +- The constructor of `TimedOutboxSweeper` grows again. Callers who construct the sweeper by hand get + no metrics unless they pass a meter. + +### Risks and Mitigations + +| Risk | Mitigation | +| --- | --- | +| The names conflict with the outbox metrics in #4484 | Both use the `paramore.brighter.` prefix, and this ADR uses the `outbox_sweeper` segment. Neither counts messages twice. The #4484 review should cross-reference this ADR | +| A meter failure breaks sweeping | The role's contract says it never throws. The sweeper records outside the lock and inside its catch-all. A test checks that a throwing meter does not stop sweeps | +| Tick lag is misread when the sweeper is stopped | The sweeper records lag only for sweeps it starts, never for the wait that `StopAsync` cancels | + +## Alternatives Considered + +**1. Derive the metrics from sweeper spans (ADR 0022).** This needs no new role. +**Rejected because** sampling drops the spans, so a stall would be invisible exactly when it +matters. + +**2. Add the instruments to the outbox meter proposed in #4484.** This gives one meter for the outbox. +**Rejected because** #4484 is a draft with open naming questions. Sweeper health and outbox volume +also answer different questions. A later ADR can merge the two meters once #4484 lands. + +**3. Expose only the thread-pool runtime metrics.** This needs no Brighter change. +**Rejected because** those metrics show the cause (a starved pool) but not the effect on Brighter +(a late or stalled sweeper). Delays not caused by the pool, such as a hung sweep or CPU throttling, +would not show at all. + +**4. Do nothing and rely on logs.** **Rejected because** the logs report each event but give no +rate or latency, and most log pipelines sample or drop Information lines. + +## References + +- Issue: [#4560](https://github.com/BrighterCommand/Brighter/issues/4560). Bugfix record + `bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md` +- Related ADRs: + - [ADR 0010: Brighter OpenTelemetry Semantic Conventions](0010-brighter-semantic-conventions.md), + which supplies the naming rules + - [ADR 0022: Generate Metrics from Traces](0022-generate-metrics-from-traces.md), the pattern this + ADR does not use + - [ADR 0039: OpenTelemetry Builder Extension](0039-opentelemetry-builder-extension.md), which + supplies the registration point + - [ADR 0028: Support Circuit Breaking for Topics in Outbox Sweeper](0028-support-circuit-breaking-of-topics.md), + which also changes the sweeper +- Related work: draft PR [#4484](https://github.com/BrighterCommand/Brighter/pull/4484), outbox + metrics; issue [#4554](https://github.com/BrighterCommand/Brighter/issues/4554), the outstanding-message check +- External: [.NET runtime metrics (`System.Runtime` meter)](https://learn.microsoft.com/dotnet/core/diagnostics/built-in-metrics-runtime) diff --git a/docs/adr/index.md b/docs/adr/index.md index a230f4051d..ac942b8d32 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -130,5 +130,6 @@ Identity is the `id` (filename stem), not the leading number — numbers may rep | `0079-azure-service-bus-lock-renewal` | Renew Azure Service Bus locks while messages are pending | Proposed | transports, configuration, message-pump | Bound automatic message and session lock renewal, protect buffered deliveries, and stop renewal when delivery ownership ends. | | `0079-claim-check-cleanup-after-acknowledgement` | Clean up claim-check luggage after final acknowledgement | Proposed | claim-check, message-pump, lifetime, message-mapping | Message pumps unwrap a separate envelope and defer non-retained claim-check deletion until successful dispatch and final acknowledgement. Unwrap pipelines with pending cleanup stay alive so scoped storage dependencies remain usable; other pipelines are released before dispatch. | | `0080-share-resilience-registry-between-handlers-and-producers` | Share resilience registry between handlers and producers | Proposed | di, configuration, resilience, outbox | Resolve one resilience registry per service provider for handlers and outbox producers. An explicit builder registry overrides options, and missing default pipelines are added without replacing custom pipelines. | +| `0081-timed-outbox-sweeper-health-metrics` | TimedOutboxSweeper health metrics | Accepted | observability, metrics, outbox, otel | TimedOutboxSweeper records three instruments directly through a new IAmABrighterSweeperMeter role (no-op by default, registered by AddBrighterInstrumentation): tick lag, sweep duration, and sweeps by outcome (completed, failed, lock_unavailable), so a stalled or late sweeper is visible even when traces are sampled out. | -_117 ADRs indexed._ +_118 ADRs indexed._ From 5e50e0bb3d3e1f555223bccbc8eeb426cc9cc28f Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 17:41:49 +0100 Subject: [PATCH 02/36] test: add regression tests for TimedOutboxSweeper starvation and confirm floods Pins the defects confirmed for #4560. These tests are red until the fix lands: - a sweep still runs while every thread-pool worker is blocked - StopAsync waits for an in-flight sweep - a throwing sweep or CreateScope does not crash the process, and the lock is released - TimerInterval <= 0 is rejected - Kafka and RMQ.Sync raise publish confirmations one at a time, in order - the sweeper records tick lag, sweep duration and sweeps by outcome (ADR 0081) Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TestDoubles/FailsFirstScopeFactory.cs | 19 ++++ .../TestDoubles/GatedDistributedLock.cs | 39 ++++++++ .../TestDoubles/HeldDistributedLock.cs | 16 ++++ .../RecordedSweeperMeasurements.cs | 63 +++++++++++++ .../Sweeping/TestDoubles/StarvedThreadPool.cs | 88 ++++++++++++++++++ .../Sweeping/TestDoubles/SweepableOutbox.cs | 52 +++++++++++ .../TestDoubles/SweeperMeasurement.cs | 4 + .../TestDoubles/ThrowingSweeperMeter.cs | 14 +++ .../ThreadPoolStarvationCollection.cs | 12 +++ ...cord_a_completed_sweep_and_its_duration.cs | 43 +++++++++ ..._late_should_record_how_late_it_started.cs | 45 ++++++++++ ...eep_throws_should_record_a_failed_sweep.cs | 40 +++++++++ ...ows_should_sweep_again_on_the_next_tick.cs | 76 ++++++++++++++++ ...should_release_the_lock_and_sweep_again.cs | 72 +++++++++++++++ ...eep_should_wait_for_the_sweep_to_finish.cs | 70 +++++++++++++++ ..._should_record_a_lock_unavailable_sweep.cs | 36 ++++++++ ...er_throws_should_still_sweep_the_outbox.cs | 35 ++++++++ ..._not_positive_should_reject_the_options.cs | 24 +++++ ...l_blocked_should_still_sweep_the_outbox.cs | 83 +++++++++++++++++ ...se_confirmations_one_at_a_time_in_order.cs | 89 +++++++++++++++++++ ...se_confirmations_one_at_a_time_in_order.cs | 78 ++++++++++++++++ 21 files changed, 998 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/FailsFirstScopeFactory.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/GatedDistributedLock.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/HeldDistributedLock.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/RecordedSweeperMeasurements.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/StarvedThreadPool.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweepableOutbox.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweeperMeasurement.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/ThrowingSweeperMeter.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/ThreadPoolStarvationCollection.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_starts_late_should_record_how_late_it_started.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_record_a_failed_sweep.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_sweep_again_on_the_next_tick.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_meter_throws_should_still_sweep_the_outbox.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_timer_interval_is_not_positive_should_reject_the_options.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs create mode 100644 tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/FailsFirstScopeFactory.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/FailsFirstScopeFactory.cs new file mode 100644 index 0000000000..21e292ef48 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/FailsFirstScopeFactory.cs @@ -0,0 +1,19 @@ +using System; +using System.Threading; +using Microsoft.Extensions.DependencyInjection; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An whose first throws, as a +/// provider being torn down at shutdown would; later calls go to the real factory. +/// +public sealed class FailsFirstScopeFactory(IServiceScopeFactory inner) : IServiceScopeFactory +{ + private int _calls; + + public IServiceScope CreateScope() => + Interlocked.Increment(ref _calls) == 1 + ? throw new ObjectDisposedException(nameof(IServiceProvider)) + : inner.CreateScope(); +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/GatedDistributedLock.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/GatedDistributedLock.cs new file mode 100644 index 0000000000..be70315d5b --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/GatedDistributedLock.cs @@ -0,0 +1,39 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An that holds every caller at the door until is +/// called. The sweeper asks for its lock first thing in a sweep, so this lets a test keep a sweep +/// in flight for as long as it needs. +/// +public sealed class GatedDistributedLock : IDistributedLock, IDisposable +{ + private readonly InMemoryLock _inner = new(); + private readonly ManualResetEventSlim _gate = new(false); + private readonly ManualResetEventSlim _entered = new(false); + + /// Waits until a sweep has reached the lock. + public bool WaitForSweepToStart(TimeSpan timeout) => _entered.Wait(timeout); + + /// Lets every held and future caller through. + public void Open() => _gate.Set(); + + public Task ObtainLockAsync(string resource, CancellationToken cancellationToken) + { + _entered.Set(); + _gate.Wait(cancellationToken); + return _inner.ObtainLockAsync(resource, cancellationToken); + } + + public Task ReleaseLockAsync(string resource, string lockId, CancellationToken cancellationToken) => + _inner.ReleaseLockAsync(resource, lockId, cancellationToken); + + public void Dispose() + { + _gate.Dispose(); + _entered.Dispose(); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/HeldDistributedLock.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/HeldDistributedLock.cs new file mode 100644 index 0000000000..b94f05d55c --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/HeldDistributedLock.cs @@ -0,0 +1,16 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An that another process always holds: every attempt to obtain it fails. +/// +public sealed class HeldDistributedLock : IDistributedLock +{ + public Task ObtainLockAsync(string resource, CancellationToken cancellationToken) => + Task.FromResult(null); + + public Task ReleaseLockAsync(string resource, string lockId, CancellationToken cancellationToken) => + Task.CompletedTask; +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/RecordedSweeperMeasurements.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/RecordedSweeperMeasurements.cs new file mode 100644 index 0000000000..95f35b5055 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/RecordedSweeperMeasurements.cs @@ -0,0 +1,63 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Linq; +using System.Threading; +using Microsoft.Extensions.DependencyInjection; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// Owns an and records every measurement made by meters that factory +/// created, so a test sees only its own sweeper's instruments. +/// +public sealed class RecordedSweeperMeasurements : IDisposable +{ + public const string TickLag = "paramore.brighter.outbox_sweeper.tick.lag"; + public const string SweepDuration = "paramore.brighter.outbox_sweeper.sweep.duration"; + public const string Sweeps = "paramore.brighter.outbox_sweeper.sweeps"; + public const string Outcome = "paramore.brighter.outbox_sweeper.outcome"; + + private readonly ServiceProvider _provider; + private readonly MeterListener _listener = new(); + private readonly ConcurrentQueue _measurements = new(); + + public RecordedSweeperMeasurements() + { + _provider = new ServiceCollection().AddMetrics().BuildServiceProvider(); + MeterFactory = _provider.GetRequiredService(); + + _listener.InstrumentPublished = (instrument, listener) => + { + if (ReferenceEquals(instrument.Meter.Scope, MeterFactory)) + listener.EnableMeasurementEvents(instrument); + }; + _listener.SetMeasurementEventCallback((instrument, value, tags, _) => Record(instrument, value, tags)); + _listener.SetMeasurementEventCallback((instrument, value, tags, _) => Record(instrument, value, tags)); + _listener.Start(); + } + + public IMeterFactory MeterFactory { get; } + + public IReadOnlyList Of(string instrument) => + _measurements.Where(m => m.Instrument == instrument).ToList(); + + public bool WaitFor(Func condition, TimeSpan timeout) => + SpinWait.SpinUntil(() => condition(this), timeout); + + public void Dispose() + { + _listener.Dispose(); + _provider.Dispose(); + } + + private void Record(Instrument instrument, double value, ReadOnlySpan> tags) + { + string? outcome = null; + foreach (var tag in tags) + if (tag.Key == Outcome) outcome = tag.Value as string; + + _measurements.Enqueue(new SweeperMeasurement(instrument.Name, value, outcome)); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/StarvedThreadPool.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/StarvedThreadPool.cs new file mode 100644 index 0000000000..43f48e2d6c --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/StarvedThreadPool.cs @@ -0,0 +1,88 @@ +using System; +using System.Threading; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// Caps the thread pool at its minimum size and parks every worker on a gate, so that no pool thread +/// is free to run anything — not even a callback — until . +/// Process-wide: only use from a test in . +/// +public sealed class StarvedThreadPool : IDisposable +{ + private const int MaxAttempts = 20; + private static readonly TimeSpan s_probeWindow = TimeSpan.FromMilliseconds(300); + private static readonly TimeSpan s_drainTimeout = TimeSpan.FromSeconds(30); + + private readonly ManualResetEventSlim _gate = new(false); + private readonly CountdownEvent _blockersFinished = new(1); + private readonly int _maxWorkers; + private readonly int _maxCompletionPorts; + + public StarvedThreadPool() + { + ThreadPool.GetMaxThreads(out _maxWorkers, out _maxCompletionPorts); + try + { + Assert.True(ThreadPool.SetMaxThreads(Environment.ProcessorCount, _maxCompletionPorts), "Could not cap the thread pool"); + Assert.True(BlockEveryWorker(), "Could not block every thread-pool worker"); + } + catch + { + Dispose(); + throw; + } + } + + /// Opens the gate, restores the pool's limits and waits for every blocked item to finish. + public void Dispose() + { + _gate.Set(); + ThreadPool.SetMaxThreads(_maxWorkers, _maxCompletionPorts); + _blockersFinished.Signal(); + _blockersFinished.Wait(s_drainTimeout); + _blockersFinished.Dispose(); + _gate.Dispose(); + } + + // Threads left over from earlier tests can sit above the cap until they go idle, so keep queueing + // blockers until a probe timer — the same mechanism the sweeper uses — can no longer fire. + private bool BlockEveryWorker() + { + for (var attempt = 0; attempt < MaxAttempts; attempt++) + { + QueueBlockers(Math.Max(ThreadPool.ThreadCount, Environment.ProcessorCount) * 2); + if (!TimerCanFire()) return true; + } + return false; + } + + private void QueueBlockers(int count) + { + for (var i = 0; i < count; i++) + { + _blockersFinished.AddCount(); + ThreadPool.UnsafeQueueUserWorkItem(_ => + { + _gate.Wait(); + _blockersFinished.Signal(); + }, null); + } + } + + private static bool TimerCanFire() + { + using var fired = new ManualResetEventSlim(false); + using var probe = new Timer(_ => SetIfNotDisposed(fired), null, TimeSpan.Zero, System.Threading.Timeout.InfiniteTimeSpan); + var result = fired.Wait(s_probeWindow); + probe.Change(System.Threading.Timeout.Infinite, System.Threading.Timeout.Infinite); + return result; + } + + private static void SetIfNotDisposed(ManualResetEventSlim fired) + { + try { fired.Set(); } + catch (ObjectDisposedException) { } + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweepableOutbox.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweepableOutbox.cs new file mode 100644 index 0000000000..ebb5206098 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweepableOutbox.cs @@ -0,0 +1,52 @@ +using System; +using System.Collections.Generic; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Polly.Registry; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An in-memory outbox wired to an through a real +/// , as a sweeper sees it from DI. +/// +public sealed class SweepableOutbox : IDisposable +{ + private readonly ServiceProvider _provider; + + public SweepableOutbox() + { + var producer = new InMemoryMessageProducer(Bus, new Publication { Topic = RoutingKey }); + var mediator = new OutboxProducerMediator( + new ProducerRegistry(new Dictionary { { RoutingKey, producer } }), + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + Outbox); + + var services = new ServiceCollection(); + services.AddSingleton(mediator); + _provider = services.BuildServiceProvider(); + } + + public RoutingKey RoutingKey { get; } = new("MyCommand"); + public InternalBus Bus { get; } = new(); + public InMemoryOutbox Outbox { get; } = new(TimeProvider.System); + public IServiceScopeFactory ScopeFactory => _provider.GetRequiredService(); + + public Message AddUndispatched() + { + var message = new Message( + new MessageHeader(Id.Random(), RoutingKey, MessageType.MT_COMMAND), + new MessageBody("awaiting a publish confirmation that never came")); + Outbox.Add(message, new RequestContext()); + return message; + } + + public void Dispose() => _provider.Dispose(); +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweeperMeasurement.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweeperMeasurement.cs new file mode 100644 index 0000000000..3b5535d63d --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SweeperMeasurement.cs @@ -0,0 +1,4 @@ +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// One value recorded on a sweeper instrument, with its outcome attribute if it had one. +public sealed record SweeperMeasurement(string Instrument, double Value, string? Outcome); diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/ThrowingSweeperMeter.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/ThrowingSweeperMeter.cs new file mode 100644 index 0000000000..519b08612c --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/ThrowingSweeperMeter.cs @@ -0,0 +1,14 @@ +using System; +using Paramore.Brighter.Observability; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// A sweeper meter whose every recording throws, as a broken metrics exporter might. +/// +public sealed class ThrowingSweeperMeter : IAmABrighterSweeperMeter +{ + public void RecordTickLag(TimeSpan lag) => throw new InvalidOperationException("metrics are broken"); + + public void RecordSweep(SweepOutcome outcome, TimeSpan duration) => throw new InvalidOperationException("metrics are broken"); +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/ThreadPoolStarvationCollection.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/ThreadPoolStarvationCollection.cs new file mode 100644 index 0000000000..1ee9c34d92 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/ThreadPoolStarvationCollection.cs @@ -0,0 +1,12 @@ +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +// The sweeper starvation tests cap the thread pool with ThreadPool.SetMaxThreads and park every +// worker on a gate. Both are process-wide: run alongside another class, they would starve it too +// (and its pool work would unblock ours). Serialising through this collection removes the overlap. +[CollectionDefinition(Name, DisableParallelization = true)] +public class ThreadPoolStarvationCollection +{ + public const string Name = "ThreadPoolStarvation"; +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration.cs new file mode 100644 index 0000000000..69a6e4bd27 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration.cs @@ -0,0 +1,43 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class SweeperCompletedSweepMetricsTests +{ + [Fact] + public async Task When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration() + { + //Arrange + using var outbox = new SweepableOutbox(); + using var measurements = new RecordedSweeperMeasurements(); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }, + meter: new SweeperMeter(measurements.MeterFactory)); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var recorded = measurements.WaitFor( + m => m.Of(RecordedSweeperMeasurements.Sweeps).Any() && m.Of(RecordedSweeperMeasurements.SweepDuration).Any(), + TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(recorded, "The sweeper recorded no sweep"); + var sweep = measurements.Of(RecordedSweeperMeasurements.Sweeps).First(); + Assert.Equal(1, sweep.Value); + Assert.Equal("completed", sweep.Outcome); + var duration = measurements.Of(RecordedSweeperMeasurements.SweepDuration).First(); + Assert.Equal("completed", duration.Outcome); + Assert.True(duration.Value >= 0); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_starts_late_should_record_how_late_it_started.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_starts_late_should_record_how_late_it_started.cs new file mode 100644 index 0000000000..fc6f6a897c --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_starts_late_should_record_how_late_it_started.cs @@ -0,0 +1,45 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Time.Testing; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class SweeperTickLagMetricsTests +{ + [Fact] + public async Task When_a_sweep_starts_late_should_record_how_late_it_started() + { + //Arrange + var timeProvider = new FakeTimeProvider(); + using var outbox = new SweepableOutbox(); + using var measurements = new RecordedSweeperMeasurements(); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 5, MinimumMessageAge = TimeSpan.Zero }, + timeProvider: timeProvider, + meter: new SweeperMeter(measurements.MeterFactory)); + + await sweeper.StartAsync(CancellationToken.None); + var firstSweep = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.TickLag).Count == 1, TimeSpan.FromSeconds(3)); + + //Act + timeProvider.Advance(TimeSpan.FromSeconds(7)); // the next sweep was due at 5 s; the clock is now at 7 s + var secondSweep = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.TickLag).Count == 2, TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(firstSweep, "The sweeper did not record the lag of its first sweep"); + Assert.True(secondSweep, "The sweeper did not record the lag of its second sweep"); + var lags = measurements.Of(RecordedSweeperMeasurements.TickLag); + Assert.Equal(0, lags[0].Value); + Assert.Equal(2, lags[1].Value); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_record_a_failed_sweep.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_record_a_failed_sweep.cs new file mode 100644 index 0000000000..64b172ce68 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_record_a_failed_sweep.cs @@ -0,0 +1,40 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class SweeperFailedSweepMetricsTests +{ + [Fact] + public async Task When_a_sweep_throws_should_record_a_failed_sweep() + { + //Arrange + var services = new ServiceCollection(); + services.AddTransient(_ => throw new InvalidOperationException("every sweep fails")); + await using var provider = services.BuildServiceProvider(); + + using var measurements = new RecordedSweeperMeasurements(); + var sweeper = new TimedOutboxSweeper( + provider.GetRequiredService(), + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }, + meter: new SweeperMeter(measurements.MeterFactory)); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var recorded = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.Sweeps).Any(), TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(recorded, "The sweeper recorded no sweep"); + Assert.Equal("failed", measurements.Of(RecordedSweeperMeasurements.Sweeps).First().Outcome); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_sweep_again_on_the_next_tick.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_sweep_again_on_the_next_tick.cs new file mode 100644 index 0000000000..3d91f58bac --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_a_sweep_throws_should_sweep_again_on_the_next_tick.cs @@ -0,0 +1,76 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperFailedSweepTests +{ + private static readonly TimeSpan s_waitForSweep = TimeSpan.FromSeconds(3); + + [Fact] + public async Task When_a_sweep_throws_should_sweep_again_on_the_next_tick() + { + //Arrange + var routingKey = new RoutingKey("MyCommand"); + var bus = new InternalBus(); + var outbox = new InMemoryOutbox(TimeProvider.System); + var mediator = CreateMediator(bus, routingKey, outbox); + + var undispatched = new Message( + new MessageHeader(Id.Random(), routingKey, MessageType.MT_COMMAND), + new MessageBody("awaiting a publish confirmation that never came")); + outbox.Add(undispatched, new RequestContext()); + + var sweepsAttempted = 0; + var services = new ServiceCollection(); + services.AddTransient(_ => + Interlocked.Increment(ref sweepsAttempted) == 1 + ? throw new InvalidOperationException("the first sweep fails") + : mediator); + await using var provider = services.BuildServiceProvider(); + + var sweeper = new TimedOutboxSweeper( + provider.GetRequiredService(), + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var swept = SpinWait.SpinUntil(() => bus.Stream(routingKey).Any(), s_waitForSweep); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(swept, $"The sweeper did not recover from a failed sweep within {s_waitForSweep}"); + } + + private static OutboxProducerMediator CreateMediator( + InternalBus bus, RoutingKey routingKey, InMemoryOutbox outbox) + { + var producer = new InMemoryMessageProducer(bus, new Publication { Topic = routingKey }); + var producerRegistry = new ProducerRegistry(new Dictionary + { + { routingKey, producer } + }); + + return new OutboxProducerMediator( + producerRegistry, + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + outbox); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again.cs new file mode 100644 index 0000000000..3f651e12d9 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again.cs @@ -0,0 +1,72 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperScopeFailureTests +{ + private static readonly TimeSpan s_waitForSweep = TimeSpan.FromSeconds(3); + + [Fact] + public async Task When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again() + { + //Arrange + var routingKey = new RoutingKey("MyCommand"); + var bus = new InternalBus(); + var outbox = new InMemoryOutbox(TimeProvider.System); + + var undispatched = new Message( + new MessageHeader(Id.Random(), routingKey, MessageType.MT_COMMAND), + new MessageBody("awaiting a publish confirmation that never came")); + outbox.Add(undispatched, new RequestContext()); + + var services = new ServiceCollection(); + services.AddSingleton(CreateMediator(bus, routingKey, outbox)); + await using var provider = services.BuildServiceProvider(); + + var sweeper = new TimedOutboxSweeper( + new FailsFirstScopeFactory(provider.GetRequiredService()), + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var swept = SpinWait.SpinUntil(() => bus.Stream(routingKey).Any(), s_waitForSweep); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(swept, $"The sweeper did not sweep again within {s_waitForSweep} after failing to create its scope"); + } + + private static OutboxProducerMediator CreateMediator( + InternalBus bus, RoutingKey routingKey, InMemoryOutbox outbox) + { + var producer = new InMemoryMessageProducer(bus, new Publication { Topic = routingKey }); + var producerRegistry = new ProducerRegistry(new Dictionary + { + { routingKey, producer } + }); + + return new OutboxProducerMediator( + producerRegistry, + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + outbox); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs new file mode 100644 index 0000000000..ee117a1782 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs @@ -0,0 +1,70 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperStopTests +{ + private static readonly TimeSpan s_graceForStop = TimeSpan.FromMilliseconds(500); + + [Fact] + public async Task When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish() + { + //Arrange + var routingKey = new RoutingKey("MyCommand"); + var services = new ServiceCollection(); + services.AddSingleton(CreateMediator(routingKey)); + // Not disposed: today the sweep carries on after StopAsync returns, and a disposed provider + // makes it throw out of the sweeper and crash the test host instead of failing the assertion. + var provider = services.BuildServiceProvider(); + + using var sweepInFlight = new GatedDistributedLock(); + var sweeper = new TimedOutboxSweeper( + provider.GetRequiredService(), + sweepInFlight, + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + await sweeper.StartAsync(CancellationToken.None); + Assert.True(sweepInFlight.WaitForSweepToStart(TimeSpan.FromSeconds(5)), "No sweep started"); + + //Act + var stopping = Task.Run(() => sweeper.StopAsync(CancellationToken.None)); + var stoppedMidSweep = await Task.WhenAny(stopping, Task.Delay(s_graceForStop)) == stopping; + + sweepInFlight.Open(); + await stopping.WaitAsync(TimeSpan.FromSeconds(5)); + sweeper.Dispose(); + + //Assert + Assert.False(stoppedMidSweep, "StopAsync returned while a sweep was still running"); + } + + private static OutboxProducerMediator CreateMediator(RoutingKey routingKey) + { + var producer = new InMemoryMessageProducer(new InternalBus(), new Publication { Topic = routingKey }); + var producerRegistry = new ProducerRegistry(new Dictionary + { + { routingKey, producer } + }); + + return new OutboxProducerMediator( + producerRegistry, + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + new InMemoryOutbox(TimeProvider.System)); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep.cs new file mode 100644 index 0000000000..eb997250c4 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep.cs @@ -0,0 +1,36 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class SweeperLockUnavailableMetricsTests +{ + [Fact] + public async Task When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep() + { + //Arrange + using var outbox = new SweepableOutbox(); + using var measurements = new RecordedSweeperMeasurements(); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + new HeldDistributedLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }, + meter: new SweeperMeter(measurements.MeterFactory)); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var recorded = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.Sweeps).Any(), TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(recorded, "The sweeper recorded no sweep"); + Assert.Equal("lock_unavailable", measurements.Of(RecordedSweeperMeasurements.Sweeps).First().Outcome); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_meter_throws_should_still_sweep_the_outbox.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_meter_throws_should_still_sweep_the_outbox.cs new file mode 100644 index 0000000000..67e2ebcf3e --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_meter_throws_should_still_sweep_the_outbox.cs @@ -0,0 +1,35 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class SweeperThrowingMeterTests +{ + [Fact] + public async Task When_the_sweeper_meter_throws_should_still_sweep_the_outbox() + { + //Arrange + using var outbox = new SweepableOutbox(); + var undispatched = outbox.AddUndispatched(); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }, + meter: new ThrowingSweeperMeter()); + + //Act + await sweeper.StartAsync(CancellationToken.None); + var swept = SpinWait.SpinUntil( + () => outbox.Bus.Stream(outbox.RoutingKey).Any(m => m.Id == undispatched.Id), TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(swept, "A throwing meter stopped the sweeper from dispatching the outstanding message"); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_timer_interval_is_not_positive_should_reject_the_options.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_timer_interval_is_not_positive_should_reject_the_options.cs new file mode 100644 index 0000000000..6deffe19fc --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_timer_interval_is_not_positive_should_reject_the_options.cs @@ -0,0 +1,24 @@ +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperOptionsValidationTests +{ + [Theory] + [InlineData(0)] + [InlineData(-1)] + public void When_the_timer_interval_is_not_positive_should_reject_the_options(int timerInterval) + { + //Arrange + var scopeFactory = new ServiceCollection().BuildServiceProvider().GetRequiredService(); + var options = new TimedOutboxSweeperOptions { TimerInterval = timerInterval }; + + //Act + var exception = Record.Exception(() => new TimedOutboxSweeper(scopeFactory, new InMemoryLock(), options)); + + //Assert + Assert.IsType(exception); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs new file mode 100644 index 0000000000..78bfcffc49 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs @@ -0,0 +1,83 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +[Collection(ThreadPoolStarvationCollection.Name)] +public class TimedOutboxSweeperStarvationTests +{ + private static readonly TimeSpan s_waitForSweep = TimeSpan.FromSeconds(3); + + [Fact] + public void When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox() + { + //Arrange + var routingKey = new RoutingKey("MyCommand"); + var bus = new InternalBus(); + var outbox = new InMemoryOutbox(TimeProvider.System); + var mediator = CreateMediator(bus, routingKey, outbox); + + var undispatched = new Message( + new MessageHeader(Id.Random(), routingKey, MessageType.MT_COMMAND), + new MessageBody("awaiting a publish confirmation that never came")); + outbox.Add(undispatched, new RequestContext()); + + var services = new ServiceCollection(); + services.AddSingleton(mediator); + // Not disposed: a starved timer tick can still fire after StopAsync, and a disposed provider + // makes that tick throw out of the sweeper and crash the test host (a separate defect, #4560). + var provider = services.BuildServiceProvider(); + + var sweeper = new TimedOutboxSweeper( + provider.GetRequiredService(), + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + var starvedPool = new StarvedThreadPool(); + try + { + //Act + sweeper.StartAsync(CancellationToken.None).GetAwaiter().GetResult(); + var swept = SpinWait.SpinUntil(() => bus.Stream(routingKey).Any(), s_waitForSweep); + + //Assert + Assert.True(swept, $"The sweeper did not dispatch the outstanding message within {s_waitForSweep} while every thread-pool worker was blocked"); + } + finally + { + starvedPool.Dispose(); + sweeper.StopAsync(CancellationToken.None).GetAwaiter().GetResult(); + sweeper.Dispose(); + } + } + + private static OutboxProducerMediator CreateMediator( + InternalBus bus, RoutingKey routingKey, InMemoryOutbox outbox) + { + var producer = new InMemoryMessageProducer(bus, new Publication { Topic = routingKey }); + var producerRegistry = new ProducerRegistry(new Dictionary + { + { routingKey, producer } + }); + + return new OutboxProducerMediator( + producerRegistry, + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + outbox); + } +} diff --git a/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs b/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs new file mode 100644 index 0000000000..155f1dd091 --- /dev/null +++ b/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs @@ -0,0 +1,89 @@ +using System; +using System.Collections.Concurrent; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.MessagingGateway.Kafka; +using Xunit; + +namespace Paramore.Brighter.Kafka.Tests.MessagingGateway.Reactor; + +[Trait("Category", "Kafka")] +[Collection("Kafka")] +public class KafkaConfirmationOrderingTests : IDisposable +{ + private const int MessageCount = 50; + private readonly string _topic = Guid.NewGuid().ToString(); + private readonly IAmAProducerRegistry _producerRegistry; + + public KafkaConfirmationOrderingTests() + { + _producerRegistry = new KafkaProducerRegistryFactory( + new KafkaMessagingGatewayConfiguration + { + Name = "Kafka Confirmation Ordering Test", BootStrapServers = new[] { "localhost:9092" } + }, + [ + new KafkaPublication + { + Topic = new RoutingKey(_topic), + NumPartitions = 1, + ReplicationFactor = 1, + MessageTimeoutMs = 2000, + RequestTimeoutMs = 2000, + MakeChannels = OnMissingChannel.Create + } + ]).Create(); + } + + [Fact] + public async Task When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order() + { + //Arrange + await Task.Delay(500); //Let the topic propagate in the broker + + var routingKey = new RoutingKey(_topic); + var producer = _producerRegistry.LookupBy(routingKey); + var confirmations = (ISupportPublishConfirmationAsync)producer; + + var inFlight = 0; + var mostAtOnce = 0; + var confirmedInOrder = new ConcurrentQueue(); + using var allConfirmed = new CountdownEvent(MessageCount); + confirmations.OnMessagePublishedAsync += async result => + { + var now = Interlocked.Increment(ref inFlight); + InterlockedMax(ref mostAtOnce, now); + await Task.Delay(10); // a handler that takes a little time, like marking an outbox row dispatched + confirmedInOrder.Enqueue(result.MessageId); + Interlocked.Decrement(ref inFlight); + allConfirmed.Signal(); + }; + + var sent = Enumerable.Range(0, MessageCount) + .Select(_ => new Message( + new MessageHeader(Id.Random(), routingKey, MessageType.MT_EVENT), + new MessageBody("confirm me"))) + .ToArray(); + + //Act + foreach (var message in sent) + ((IAmAMessageProducerSync)producer).Send(message); + + var completed = allConfirmed.Wait(TimeSpan.FromSeconds(30)); + + //Assert + Assert.True(completed, "Not every message was confirmed"); + Assert.Equal(1, mostAtOnce); + Assert.Equal(sent.Select(m => m.Id), confirmedInOrder); + } + + private static void InterlockedMax(ref int target, int value) + { + int current; + while ((current = Volatile.Read(ref target)) < value + && Interlocked.CompareExchange(ref target, value, current) != current) { } + } + + public void Dispose() => _producerRegistry.Dispose(); +} diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs new file mode 100644 index 0000000000..b734ac568e --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs @@ -0,0 +1,78 @@ +using System; +using System.Collections.Concurrent; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using Xunit; + +namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; + +[Trait("Category", "RMQ")] +[Collection("RMQ")] +public class RmqConfirmationOrderingTests : IDisposable +{ + private const int MessageCount = 50; + private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); + private readonly RmqMessageProducer _messageProducer; + + public RmqConfirmationOrderingTests() + { + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672/%2f")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + _messageProducer = new RmqMessageProducer(rmqConnection); + + //we need a queue to avoid a discard + new QueueFactory(rmqConnection, new ChannelName(Guid.NewGuid().ToString()), new RoutingKeys(_routingKey)) + .Create(TimeSpan.FromMilliseconds(1000)); + } + + [Fact] + public void When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order() + { + //Arrange + var inFlight = 0; + var mostAtOnce = 0; + var confirmedInOrder = new ConcurrentQueue(); + using var allConfirmed = new CountdownEvent(MessageCount); + ((ISupportPublishConfirmationAsync)_messageProducer).OnMessagePublishedAsync += async result => + { + var now = Interlocked.Increment(ref inFlight); + InterlockedMax(ref mostAtOnce, now); + await Task.Delay(10); // a handler that takes a little time, like marking an outbox row dispatched + confirmedInOrder.Enqueue(result.MessageId); + Interlocked.Decrement(ref inFlight); + allConfirmed.Signal(); + }; + + var sent = Enumerable.Range(0, MessageCount) + .Select(_ => new Message( + new MessageHeader(Id.Random(), _routingKey, MessageType.MT_COMMAND), + new MessageBody("confirm me"))) + .ToArray(); + + //Act + foreach (var message in sent) + _messageProducer.Send(message); + + var completed = allConfirmed.Wait(TimeSpan.FromSeconds(30)); + + //Assert + Assert.True(completed, "Not every message was confirmed"); + Assert.Equal(1, mostAtOnce); + Assert.Equal(sent.Select(m => m.Id), confirmedInOrder); + } + + private static void InterlockedMax(ref int target, int value) + { + int current; + while ((current = Volatile.Read(ref target)) < value + && Interlocked.CompareExchange(ref target, value, current) != current) { } + } + + public void Dispose() => _messageProducer.Dispose(); +} From 84341b0bb45207dff31c1e9adba8d9fd19d90e3a Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 17:44:02 +0100 Subject: [PATCH 03/36] refactor: add sweeper meter role and optional TimeProvider to TimedOutboxSweeper Structural only, per ADR 0081: IAmABrighterSweeperMeter, SweepOutcome, NullSweeperMeter and the instrument names. TimedOutboxSweeper gains optional timeProvider and meter constructor parameters, not yet used. Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TimedOutboxSweeper.cs | 11 +++- .../BrighterSemanticConventions.cs | 4 ++ .../Observability/IAmABrighterSweeperMeter.cs | 52 +++++++++++++++++++ .../Observability/NullSweeperMeter.cs | 43 +++++++++++++++ .../Observability/SweepOutcome.cs | 41 +++++++++++++++ 5 files changed, 150 insertions(+), 1 deletion(-) create mode 100644 src/Paramore.Brighter/Observability/IAmABrighterSweeperMeter.cs create mode 100644 src/Paramore.Brighter/Observability/NullSweeperMeter.cs create mode 100644 src/Paramore.Brighter/Observability/SweepOutcome.cs diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index b35b1a43d7..68b8826cf4 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -31,6 +31,7 @@ THE SOFTWARE. */ using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using Paramore.Brighter.Logging; +using Paramore.Brighter.Observability; using ILogger = Microsoft.Extensions.Logging.ILogger; namespace Paramore.Brighter.Outbox.Hosting @@ -44,6 +45,8 @@ public partial class TimedOutboxSweeper : IHostedService, IDisposable private readonly IServiceScopeFactory _serviceScopeFactory; private readonly IDistributedLock _distributedLock; private readonly TimedOutboxSweeperOptions _options; + private readonly TimeProvider _timeProvider; + private readonly IAmABrighterSweeperMeter _meter; private static readonly ILogger s_logger = ApplicationLogging.CreateLogger(); private Timer? _timer; private const string LockingResourceName = "OutboxSweeper"; @@ -54,15 +57,21 @@ public partial class TimedOutboxSweeper : IHostedService, IDisposable /// Needed to create a scope within which to create a /// Used to ensure that only one instance of the is running /// The that can be used to configure how this runs, such as interval or age + /// The clock that schedules sweeps; defaults to + /// Records the sweeper's health (see ADR 0081); defaults to , which records nothing public TimedOutboxSweeper( IServiceScopeFactory serviceScopeFactory, IDistributedLock distributedLock, - TimedOutboxSweeperOptions options + TimedOutboxSweeperOptions options, + TimeProvider? timeProvider = null, + IAmABrighterSweeperMeter? meter = null ) { _serviceScopeFactory = serviceScopeFactory; _distributedLock = distributedLock; _options = options; + _timeProvider = timeProvider ?? TimeProvider.System; + _meter = meter ?? NullSweeperMeter.Instance; } /// diff --git a/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs b/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs index d9c352010e..0ea9a73594 100644 --- a/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs +++ b/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs @@ -102,6 +102,10 @@ public static class BrighterSemanticConventions public const string NetworkPeerPort = "network.peer.port"; public const string Operation = "paramore.brighter.operation"; public const string OutboxSharedTransaction = "paramore.brighter.outbox.shared_transaction"; + public const string OutboxSweeperOutcome = "paramore.brighter.outbox_sweeper.outcome"; + public const string OutboxSweeperSweepDuration = "paramore.brighter.outbox_sweeper.sweep.duration"; + public const string OutboxSweeperSweeps = "paramore.brighter.outbox_sweeper.sweeps"; + public const string OutboxSweeperTickLag = "paramore.brighter.outbox_sweeper.tick.lag"; public const string OutboxType = "paramore.brighter.outbox.type"; public const string ReplyTo = "paramore.brighter.replyto"; public const string RequestId = "paramore.brighter.request.id"; diff --git a/src/Paramore.Brighter/Observability/IAmABrighterSweeperMeter.cs b/src/Paramore.Brighter/Observability/IAmABrighterSweeperMeter.cs new file mode 100644 index 0000000000..7dac26c2f5 --- /dev/null +++ b/src/Paramore.Brighter/Observability/IAmABrighterSweeperMeter.cs @@ -0,0 +1,52 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System; + +namespace Paramore.Brighter.Observability; + +/// +/// Records the health of an outbox sweeper: how late each sweep started, and how each sweep ended. +/// See ADR 0081. +/// +/// +/// The sweeper decides what happened in a sweep; an implementation only records it. Implementations +/// must never throw: a metrics failure must not fail a sweep. +/// +public interface IAmABrighterSweeperMeter +{ + /// + /// Records how long after its due time a sweep started. + /// + /// The time from when the sweep was due to when it started. A negative value is recorded as zero. + void RecordTickLag(TimeSpan lag); + + /// + /// Records that a sweep ended, how it ended, and how long it took. + /// + /// How the sweep ended. + /// How long the sweep took, including any time spent waiting for the lock. + void RecordSweep(SweepOutcome outcome, TimeSpan duration); +} diff --git a/src/Paramore.Brighter/Observability/NullSweeperMeter.cs b/src/Paramore.Brighter/Observability/NullSweeperMeter.cs new file mode 100644 index 0000000000..8dbfbb25be --- /dev/null +++ b/src/Paramore.Brighter/Observability/NullSweeperMeter.cs @@ -0,0 +1,43 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System; + +namespace Paramore.Brighter.Observability; + +/// +/// An that records nothing. Used when metrics are not configured. +/// +public sealed class NullSweeperMeter : IAmABrighterSweeperMeter +{ + /// The shared instance. + public static NullSweeperMeter Instance { get; } = new(); + + /// + public void RecordTickLag(TimeSpan lag) { } + + /// + public void RecordSweep(SweepOutcome outcome, TimeSpan duration) { } +} diff --git a/src/Paramore.Brighter/Observability/SweepOutcome.cs b/src/Paramore.Brighter/Observability/SweepOutcome.cs new file mode 100644 index 0000000000..8af664e79e --- /dev/null +++ b/src/Paramore.Brighter/Observability/SweepOutcome.cs @@ -0,0 +1,41 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +namespace Paramore.Brighter.Observability; + +/// +/// How an outbox sweep ended. +/// +public enum SweepOutcome +{ + /// The sweep ran to the end. + Completed, + + /// The sweep threw an exception. + Failed, + + /// The sweep did not run because another sweep held the lock. + LockUnavailable +} From 85b219f1400b2e19b49dc03c5b2c43e7989490fb Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 17:50:26 +0100 Subject: [PATCH 04/36] fix: run TimedOutboxSweeper on a dedicated thread and record its health The sweeper ran each sweep as an async void callback on a thread-pool Timer, so it stopped sweeping while every pool worker was blocked. Under load that left unconfirmed messages in the InMemoryOutbox. - Sweeps run on a dedicated thread, scheduled by an optional TimeProvider - StopAsync waits for a sweep in flight - A failed sweep is logged and retried on the next tick instead of crashing the process; the lock is released on every path, including CreateScope - TimerInterval below one second is rejected with a ConfigurationException - SweeperMeter records tick lag, sweep duration and sweeps by outcome (ADR 0081), registered by AddBrighterInstrumentation Fixes #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../BrighterMetricsBuilderExtensions.cs | 1 + .../TimedOutboxSweeper.cs | 197 ++++++++++++++---- .../Observability/SweeperMeter.cs | 89 ++++++++ ...eep_should_wait_for_the_sweep_to_finish.cs | 4 +- ...l_blocked_should_still_sweep_the_outbox.cs | 4 +- 5 files changed, 252 insertions(+), 43 deletions(-) create mode 100644 src/Paramore.Brighter/Observability/SweeperMeter.cs diff --git a/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs b/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs index 8f4a13993a..868d33fa84 100644 --- a/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs +++ b/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs @@ -37,6 +37,7 @@ public static MeterProviderBuilder AddBrighterInstrumentation(this MeterProvider { services.TryAddSingleton(); services.TryAddSingleton(); + services.TryAddSingleton(); builder.AddMeter(BrighterSemanticConventions.MeterName); }); diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index 68b8826cf4..6f26c4a519 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -40,6 +40,10 @@ namespace Paramore.Brighter.Outbox.Hosting /// Runs a sweeper that will find outstanding messages in the Outbox and produce them via a broker /// Uses a time to run at pre-defined intervals /// + /// + /// Sweeps run on a dedicated thread, not on the thread pool. A thread-pool timer cannot fire while every + /// pool worker is blocked, and that is exactly when unsent messages pile up in the Outbox (#4560). + /// public partial class TimedOutboxSweeper : IHostedService, IDisposable { private readonly IServiceScopeFactory _serviceScopeFactory; @@ -47,8 +51,13 @@ public partial class TimedOutboxSweeper : IHostedService, IDisposable private readonly TimedOutboxSweeperOptions _options; private readonly TimeProvider _timeProvider; private readonly IAmABrighterSweeperMeter _meter; + private readonly TimeSpan _interval; + private readonly CancellationTokenSource _stopping = new(); + private readonly AutoResetEvent _due = new(false); + private readonly TaskCompletionSource _stopped = new(TaskCreationOptions.RunContinuationsAsynchronously); private static readonly ILogger s_logger = ApplicationLogging.CreateLogger(); - private Timer? _timer; + private Thread? _sweepThread; + private bool _disposed; private const string LockingResourceName = "OutboxSweeper"; /// @@ -59,6 +68,7 @@ public partial class TimedOutboxSweeper : IHostedService, IDisposable /// The that can be used to configure how this runs, such as interval or age /// The clock that schedules sweeps; defaults to /// Records the sweeper's health (see ADR 0081); defaults to , which records nothing + /// Thrown when is less than one second public TimedOutboxSweeper( IServiceScopeFactory serviceScopeFactory, IDistributedLock distributedLock, @@ -67,87 +77,195 @@ public TimedOutboxSweeper( IAmABrighterSweeperMeter? meter = null ) { + if (options.TimerInterval < 1) + throw new ConfigurationException( + $"{nameof(TimedOutboxSweeperOptions)}.{nameof(TimedOutboxSweeperOptions.TimerInterval)} must be at least 1 second, but was {options.TimerInterval}"); + _serviceScopeFactory = serviceScopeFactory; _distributedLock = distributedLock; _options = options; _timeProvider = timeProvider ?? TimeProvider.System; _meter = meter ?? NullSweeperMeter.Instance; + _interval = TimeSpan.FromSeconds(options.TimerInterval); } /// /// Starts an instance of the at the configured interval. See /// - /// Cancels execution of the + /// Not used /// A completed task to allow other background services to be run public Task StartAsync(CancellationToken cancellationToken) { Log.OutboxSweeperServiceIsStarting(s_logger); - _timer = new Timer(Sweep, null, TimeSpan.Zero, TimeSpan.FromSeconds(_options.TimerInterval)); + _sweepThread = new Thread(SweepUntilStopped) { IsBackground = true, Name = "Brighter Outbox Sweeper" }; + _sweepThread.Start(); return Task.CompletedTask; } /// - /// Stops the + /// Stops the , waiting for any sweep in flight to finish /// - /// Not used - /// A completed task to allow other background services to be stopped + /// Abandons the wait for a sweep in flight + /// A task that completes when the sweeper has stopped public Task StopAsync(CancellationToken cancellationToken) { Log.OutboxSweeperServiceIsStopping(s_logger); - _timer?.Change(Timeout.Infinite, 0); + if (_sweepThread is null) + return Task.CompletedTask; - return Task.CompletedTask; + _stopping.Cancel(); + return WaitForStop(cancellationToken); } /// - /// Cleans up the used by the + /// Stops the sweeper's thread and releases the resources it uses /// public void Dispose() { - _timer?.Dispose(); + if (_disposed) return; + _disposed = true; + + _stopping.Cancel(); + + // A sweep still in flight goes on using these; let it finish rather than pull them away. + if (_sweepThread is not null && !_sweepThread.Join(TimeSpan.Zero)) + return; + + _stopping.Dispose(); + _due.Dispose(); } - private async void Sweep(object? state) + private async Task WaitForStop(CancellationToken cancellationToken) { - var lockId = await _distributedLock.ObtainLockAsync(LockingResourceName, CancellationToken.None); - if (lockId != null) + if (!cancellationToken.CanBeCanceled) { - Log.OutboxSweeperLookingForUnsentMessages(s_logger); + await _stopped.Task; + return; + } - var scope = _serviceScopeFactory.CreateScope(); - try - { - IAmAnOutboxProducerMediator outboxProducerMediator = scope.ServiceProvider.GetRequiredService(); - - var outBoxSweeper = new OutboxSweeper( - timeSinceSent: _options.MinimumMessageAge, - outboxProducerMediator: outboxProducerMediator, - new InMemoryRequestContextFactory(), - _options.BatchSize, - _options.UseBulk, - _options.Args); - - await outBoxSweeper.SweepAsync(); - } - finally + var abandoned = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + using (cancellationToken.Register(() => abandoned.TrySetResult(true))) + await Task.WhenAny(_stopped.Task, abandoned.Task); + } + + private void SweepUntilStopped() + { + try + { + var due = _timeProvider.GetUtcNow(); + while (WaitUntil(due)) { - //on a timer thread, so blocking is OK - await _distributedLock.ReleaseLockAsync(LockingResourceName, lockId, CancellationToken.None); - - scope.Dispose(); + var started = _timeProvider.GetUtcNow(); + Record(() => _meter.RecordTickLag(started - due)); + + Sweep(); + + due = NextDue(due, started); } } - else + finally { - Log.OutboxSweeperIsStillRunningAbandoningAttempt(s_logger); + _stopped.TrySetResult(true); + } + } + + // Waits on this thread, never on the pool: the real-time timeout wakes the sweeper even when no pool + // thread is free to run the TimeProvider's timer callback. The callback lets a test clock wake it too. + private bool WaitUntil(DateTimeOffset due) + { + while (!_stopping.IsCancellationRequested) + { + var wait = due - _timeProvider.GetUtcNow(); + if (wait <= TimeSpan.Zero) + return true; + + using var timer = _timeProvider.CreateTimer(_ => SignalDue(), null, wait, Timeout.InfiniteTimeSpan); + WaitHandle.WaitAny(new[] { _due, _stopping.Token.WaitHandle }, wait); } + return false; + } + + private void SignalDue() + { + try { _due.Set(); } + catch (ObjectDisposedException) { } + } + + // Keeps to the schedule, so a late or overrunning sweep shows as lag on the next one. A sweep that started + // a whole interval or more late re-anchors the schedule, so the sweeper never sweeps back to back to catch up. + private DateTimeOffset NextDue(DateTimeOffset due, DateTimeOffset started) + { + var anchor = started - due >= _interval ? started : due; + return anchor + _interval; + } + + private void Sweep() + { + var started = _timeProvider.GetTimestamp(); + var outcome = SweepOnce(); + var duration = _timeProvider.GetElapsedTime(started); + Record(() => _meter.RecordSweep(outcome, duration)); + Log.OutboxSweeperSleeping(s_logger); } + private SweepOutcome SweepOnce() + { + try + { + // On the sweeper's own thread, so blocking on the sweep is safe: there is no synchronization context to deadlock on. + return SweepAsync().GetAwaiter().GetResult(); + } + catch (Exception e) + { + Log.OutboxSweepFailed(s_logger, e); + return SweepOutcome.Failed; + } + } + + private async Task SweepAsync() + { + var lockId = await _distributedLock.ObtainLockAsync(LockingResourceName, CancellationToken.None); + if (lockId == null) + { + Log.OutboxSweeperIsStillRunningAbandoningAttempt(s_logger); + return SweepOutcome.LockUnavailable; + } + + try + { + Log.OutboxSweeperLookingForUnsentMessages(s_logger); + + using var scope = _serviceScopeFactory.CreateScope(); + IAmAnOutboxProducerMediator outboxProducerMediator = scope.ServiceProvider.GetRequiredService(); + + var outBoxSweeper = new OutboxSweeper( + timeSinceSent: _options.MinimumMessageAge, + outboxProducerMediator: outboxProducerMediator, + new InMemoryRequestContextFactory(), + _options.BatchSize, + _options.UseBulk, + _options.Args); + + await outBoxSweeper.SweepAsync(); + return SweepOutcome.Completed; + } + finally + { + await _distributedLock.ReleaseLockAsync(LockingResourceName, lockId, CancellationToken.None); + } + } + + private static void Record(Action record) + { + try { record(); } + catch (Exception e) { Log.SweeperMeterFailed(s_logger, e); } + } + private static partial class Log { [LoggerMessage(LogLevel.Information, "Outbox Sweeper Service is starting.")] @@ -164,7 +282,12 @@ private static partial class Log [LoggerMessage(LogLevel.Information, "Outbox Sweeper sleeping")] public static partial void OutboxSweeperSleeping(ILogger logger); + + [LoggerMessage(LogLevel.Error, "Outbox Sweeper failed to sweep the outbox; it will try again on the next sweep")] + public static partial void OutboxSweepFailed(ILogger logger, Exception exception); + + [LoggerMessage(LogLevel.Warning, "Outbox Sweeper could not record a metric")] + public static partial void SweeperMeterFailed(ILogger logger, Exception exception); } } } - diff --git a/src/Paramore.Brighter/Observability/SweeperMeter.cs b/src/Paramore.Brighter/Observability/SweeperMeter.cs new file mode 100644 index 0000000000..631c1f9296 --- /dev/null +++ b/src/Paramore.Brighter/Observability/SweeperMeter.cs @@ -0,0 +1,89 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System; +using System.Collections.Generic; +using System.Diagnostics.Metrics; + +namespace Paramore.Brighter.Observability; + +/// +/// Publishes an outbox sweeper's health on the meter: +/// how late each sweep started, how long each sweep took, and how many sweeps ended each way. See ADR 0081. +/// +/// +/// The instruments are recorded directly, not derived from spans, so they report every sweep even when +/// traces are sampled out. +/// +public sealed class SweeperMeter : IAmABrighterSweeperMeter +{ + private readonly Histogram _tickLag; + private readonly Histogram _sweepDuration; + private readonly Counter _sweeps; + + /// + /// Creates the sweeper's instruments on the Brighter meter. + /// + /// Creates the meter. + public SweeperMeter(IMeterFactory meterFactory) + { + var meter = meterFactory.Create(BrighterSemanticConventions.MeterName); + + _tickLag = meter.CreateHistogram( + name: BrighterSemanticConventions.OutboxSweeperTickLag, + unit: "s", + description: "How long after its due time an outbox sweep started."); + + _sweepDuration = meter.CreateHistogram( + name: BrighterSemanticConventions.OutboxSweeperSweepDuration, + unit: "s", + description: "How long an outbox sweep took."); + + _sweeps = meter.CreateCounter( + name: BrighterSemanticConventions.OutboxSweeperSweeps, + unit: "{sweep}", + description: "Number of outbox sweeps, by how each ended."); + } + + /// + public void RecordTickLag(TimeSpan lag) => + _tickLag.Record(Math.Max(0, lag.TotalSeconds)); + + /// + public void RecordSweep(SweepOutcome outcome, TimeSpan duration) + { + var outcomeTag = new KeyValuePair(BrighterSemanticConventions.OutboxSweeperOutcome, ToTagValue(outcome)); + _sweepDuration.Record(duration.TotalSeconds, outcomeTag); + _sweeps.Add(1, outcomeTag); + } + + private static string ToTagValue(SweepOutcome outcome) => outcome switch + { + SweepOutcome.Completed => "completed", + SweepOutcome.Failed => "failed", + SweepOutcome.LockUnavailable => "lock_unavailable", + _ => throw new ArgumentOutOfRangeException(nameof(outcome), outcome, null) + }; +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs index ee117a1782..f0b436e27e 100644 --- a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs @@ -24,9 +24,7 @@ public async Task When_stopping_during_a_sweep_should_wait_for_the_sweep_to_fini var routingKey = new RoutingKey("MyCommand"); var services = new ServiceCollection(); services.AddSingleton(CreateMediator(routingKey)); - // Not disposed: today the sweep carries on after StopAsync returns, and a disposed provider - // makes it throw out of the sweeper and crash the test host instead of failing the assertion. - var provider = services.BuildServiceProvider(); + await using var provider = services.BuildServiceProvider(); using var sweepInFlight = new GatedDistributedLock(); var sweeper = new TimedOutboxSweeper( diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs index 78bfcffc49..ad00be8f36 100644 --- a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs @@ -34,9 +34,7 @@ public void When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outb var services = new ServiceCollection(); services.AddSingleton(mediator); - // Not disposed: a starved timer tick can still fire after StopAsync, and a disposed provider - // makes that tick throw out of the sweeper and crash the test host (a separate defect, #4560). - var provider = services.BuildServiceProvider(); + using var provider = services.BuildServiceProvider(); var sweeper = new TimedOutboxSweeper( provider.GetRequiredService(), From 185282398b44385af3d3c60668071fbf489b6f8e Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 17:53:15 +0100 Subject: [PATCH 05/36] fix: raise Kafka and RMQ.Sync publish confirmations one at a time, in order Both producers queued one thread-pool work item per delivery report or ack, so under load confirmations ran concurrently (50 of 50 at once on Kafka) and the queue grew with the post rate, delaying MarkDispatched and causing the sweeper to re-send delivered messages. A new SerialCallbackQueue in Paramore.Brighter.Tasks runs the callbacks on a single worker, in delivery order, and keeps the in-flight tracking that Dispose waits on. Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../KafkaMessageProducer.cs | 26 ++--- .../RmqMessageProducer.cs | 19 ++-- .../Tasks/SerialCallbackQueue.cs | 102 ++++++++++++++++++ 3 files changed, 120 insertions(+), 27 deletions(-) create mode 100644 src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs diff --git a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs index 8f5c715f9b..273116076b 100644 --- a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs @@ -76,10 +76,11 @@ event Func ISupportPublishConfirmationAsync.OnM private bool _hasFatalProducerError; private readonly InstrumentationOptions _instrumentation; private event Func? _onMessagePublishedAsync; - // Confirmation raises run on worker tasks (never on Confluent's poll thread); the tracker - // lets Dispose wait for those callbacks — including the awaited Outbox mark-dispatched — - // after Flush() has drained the delivery reports themselves. - private readonly InFlightCallbackTracker _confirmationCallbacks = new(); + // Confirmation raises run one at a time, in delivery order, on a single worker (never on Confluent's + // poll thread, and never one thread-pool item per report); the queue lets Dispose wait for those + // callbacks — including the awaited Outbox mark-dispatched — after Flush() has drained the delivery + // reports themselves. + private readonly SerialCallbackQueue _confirmationCallbacks = new(); public KafkaMessageProducer( KafkaMessagingGatewayConfiguration configuration, @@ -398,6 +399,7 @@ private void Dispose(bool disposing) // awaited Outbox mark-dispatched) may still be running, so wait for those too. Flush(); WaitForConfirmationCallbacks(); + _confirmationCallbacks.Complete(); _producer?.Dispose(); } } @@ -447,13 +449,11 @@ private void PublishResults(PersistenceStatus status, Headers headers, RoutingKe private void RaisePublishConfirmation(PublishConfirmationResult result) { - // Raise on a worker thread so we never block Confluent's delivery-report handler. Wrap the - // invoke so a faulting subscriber is logged rather than left as an unobserved Task exception - // (which can escalate via TaskScheduler.UnobservedTaskException). Brighter's own mediator - // callback is already self-contained; this guards any other subscriber and the broker thread. - // The in-flight tracker must be released on every path or dispose would block on its timeout. - _confirmationCallbacks.Begin(); - Task.Run(async () => + // Raise on the confirmation queue's worker so we never block Confluent's delivery-report handler, + // and confirmations reach subscribers one at a time, in delivery order. Wrap the invoke so a + // faulting subscriber is logged. Brighter's own mediator callback is already self-contained; + // this guards any other subscriber and the queue's worker. + _confirmationCallbacks.Enqueue(async () => { try { @@ -464,10 +464,6 @@ private void RaisePublishConfirmation(PublishConfirmationResult result) { Log.PublishConfirmationRaiseFault(s_logger, ex); } - finally - { - _confirmationCallbacks.End(); - } }); } diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index d006967ed9..fb4cec74f6 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -60,10 +60,10 @@ public partial class RmqMessageProducer : RmqMessageGateway, IAmAMessageProducer private readonly int _waitForConfirmsTimeOutInMilliseconds; private event Func? _onMessagePublishedAsync; // The ack/nack handlers run on the client's connection loop, which must never block on a - // subscriber, so awaited callbacks run on worker tasks. The tracker lets Dispose wait for - // them — including the awaited Outbox mark-dispatched — after WaitForConfirms has drained - // the broker acks themselves. - private readonly InFlightCallbackTracker _confirmationCallbacks = new(); + // subscriber, so awaited callbacks run one at a time, in ack order, on the queue's single worker + // (never one thread-pool item per ack). The queue lets Dispose wait for them — including the + // awaited Outbox mark-dispatched — after WaitForConfirms has drained the broker acks themselves. + private readonly SerialCallbackQueue _confirmationCallbacks = new(); /// /// Action taken when a message is published, following receipt of a confirmation from the broker @@ -251,6 +251,7 @@ protected override void Dispose(bool disposing) // WaitForConfirms drains the broker acks; the callbacks those acks spawned (including // the awaited Outbox mark-dispatched) run on worker tasks, so wait for them too. WaitForConfirmationCallbacks(); + _confirmationCallbacks.Complete(); } } finally @@ -289,10 +290,8 @@ private void RaisePublishConfirmation(PublishConfirmationResult result) return; // Awaited callbacks must not block the connection loop (WaitForConfirms depends on it to - // process acks), so they run on a worker task; Dispose waits on the in-flight tracker, - // which must be released on every path or that wait would hang until its timeout. - _confirmationCallbacks.Begin(); - Task.Run(async () => + // process acks), so they run on the confirmation queue's worker, one at a time, in ack order. + _confirmationCallbacks.Enqueue(async () => { try { @@ -302,10 +301,6 @@ private void RaisePublishConfirmation(PublishConfirmationResult result) { Log.ConfirmationCallbackFault(s_logger, result.MessageId.Value, ex); } - finally - { - _confirmationCallbacks.End(); - } }); } diff --git a/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs b/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs new file mode 100644 index 0000000000..8fb9ffcb54 --- /dev/null +++ b/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs @@ -0,0 +1,102 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Tasks +{ + /// + /// Runs callbacks one at a time, in the order they were queued, on a single worker. A producer uses it to + /// raise publish confirmations off the broker's thread without queuing one thread-pool work item per + /// confirmation (#4560). + /// + /// + /// A callback must handle its own exceptions; one that escapes is swallowed so that the worker keeps + /// draining. lets a producer's dispose wait for the queued callbacks to finish. + /// + public sealed class SerialCallbackQueue + { + private readonly System.Threading.Channels.Channel> _callbacks = + System.Threading.Channels.Channel.CreateUnbounded>(new UnboundedChannelOptions { SingleReader = true }); + private readonly InFlightCallbackTracker _inFlight = new(); + private int _workerStarted; + + /// + /// Queues a callback to run after every callback queued before it. + /// + /// The callback; it must not throw. + public void Enqueue(Func callback) + { + _inFlight.Begin(); + if (!_callbacks.Writer.TryWrite(callback)) + { + _inFlight.End(); + return; + } + + if (Interlocked.CompareExchange(ref _workerStarted, 1, 0) == 0) + _ = Task.Run(DrainAsync); + } + + /// + /// Waits for every queued callback to finish. + /// + /// How long to wait. + /// How many callbacks had not finished when the wait ended. + /// if every callback finished in time. + public bool TryWait(TimeSpan timeout, out int stillInFlight) => _inFlight.TryWait(timeout, out stillInFlight); + + /// + /// Stops accepting callbacks; the worker ends once it has run those already queued. + /// + public void Complete() => _callbacks.Writer.TryComplete(); + + private async Task DrainAsync() + { + var reader = _callbacks.Reader; + while (await reader.WaitToReadAsync().ConfigureAwait(false)) + { + while (reader.TryRead(out var callback)) + { + try + { + await callback().ConfigureAwait(false); + } + catch + { + // The callback owns its fault handling; swallow anything it let escape so later callbacks still run. + } + finally + { + _inFlight.End(); + } + } + } + } + } +} From 0e84c1094d2bab0dd3dcaaa26dbd7ac6a09e8272 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 18:42:08 +0100 Subject: [PATCH 06/36] fix: drain publish confirmations on a dedicated thread SerialCallbackQueue started its drain worker with Task.Run, so confirmations still waited for a thread-pool thread and stalled when the pool was starved. It now drains on one background thread per queue, running each callback in a BrighterAsyncContext so its continuations stay on that thread. A confirmation that arrives after Complete is dropped instead of throwing on the broker's thread. Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../Tasks/SerialCallbackQueue.cs | 72 +++++++++++-------- ...arved_should_still_run_queued_callbacks.cs | 41 +++++++++++ 2 files changed, 83 insertions(+), 30 deletions(-) create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs diff --git a/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs b/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs index 8fb9ffcb54..d5d538ac18 100644 --- a/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs +++ b/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs @@ -24,27 +24,29 @@ THE SOFTWARE. */ #endregion using System; +using System.Collections.Concurrent; using System.Threading; -using System.Threading.Channels; using System.Threading.Tasks; namespace Paramore.Brighter.Tasks { /// - /// Runs callbacks one at a time, in the order they were queued, on a single worker. A producer uses it to - /// raise publish confirmations off the broker's thread without queuing one thread-pool work item per - /// confirmation (#4560). + /// Runs callbacks one at a time, in the order they were queued, on a single dedicated thread. A producer uses + /// it to raise publish confirmations off the broker's thread without queuing any work on the thread pool, so a + /// busy producer does not add to the pool's queue and confirmations keep flowing when the pool is starved (#4560). /// /// - /// A callback must handle its own exceptions; one that escapes is swallowed so that the worker keeps - /// draining. lets a producer's dispose wait for the queued callbacks to finish. + /// Each callback runs inside a , so its continuations come back to the + /// dedicated thread; only a continuation that opts out of the context (ConfigureAwait(false)) can land + /// on the pool. A callback must handle its own exceptions; one that escapes is swallowed so that the thread + /// keeps draining. lets a producer's dispose wait for the queued callbacks to finish. + /// The thread starts with the first callback and ends after once the queue is empty. /// public sealed class SerialCallbackQueue { - private readonly System.Threading.Channels.Channel> _callbacks = - System.Threading.Channels.Channel.CreateUnbounded>(new UnboundedChannelOptions { SingleReader = true }); + private readonly BlockingCollection> _callbacks = new(); private readonly InFlightCallbackTracker _inFlight = new(); - private int _workerStarted; + private int _threadStarted; /// /// Queues a callback to run after every callback queued before it. @@ -53,14 +55,14 @@ public sealed class SerialCallbackQueue public void Enqueue(Func callback) { _inFlight.Begin(); - if (!_callbacks.Writer.TryWrite(callback)) + if (!TryAdd(callback)) { _inFlight.End(); return; } - if (Interlocked.CompareExchange(ref _workerStarted, 1, 0) == 0) - _ = Task.Run(DrainAsync); + if (Interlocked.CompareExchange(ref _threadStarted, 1, 0) == 0) + new Thread(Drain) { IsBackground = true, Name = "Brighter Confirmation Callbacks" }.Start(); } /// @@ -72,29 +74,39 @@ public void Enqueue(Func callback) public bool TryWait(TimeSpan timeout, out int stillInFlight) => _inFlight.TryWait(timeout, out stillInFlight); /// - /// Stops accepting callbacks; the worker ends once it has run those already queued. + /// Stops accepting callbacks; the thread ends once it has run those already queued. /// - public void Complete() => _callbacks.Writer.TryComplete(); + public void Complete() => _callbacks.CompleteAdding(); - private async Task DrainAsync() + // A confirmation that arrives after Complete (a late broker ack during dispose) is dropped rather than + // thrown back onto the broker's thread; BlockingCollection throws once adding is complete. + private bool TryAdd(Func callback) { - var reader = _callbacks.Reader; - while (await reader.WaitToReadAsync().ConfigureAwait(false)) + try { - while (reader.TryRead(out var callback)) + return _callbacks.TryAdd(callback); + } + catch (InvalidOperationException) + { + return false; + } + } + + private void Drain() + { + foreach (var callback in _callbacks.GetConsumingEnumerable()) + { + try + { + BrighterAsyncContext.Run(callback); + } + catch + { + // The callback owns its fault handling; swallow anything it let escape so later callbacks still run. + } + finally { - try - { - await callback().ConfigureAwait(false); - } - catch - { - // The callback owns its fault handling; swallow anything it let escape so later callbacks still run. - } - finally - { - _inFlight.End(); - } + _inFlight.End(); } } } diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs new file mode 100644 index 0000000000..927167b409 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs @@ -0,0 +1,41 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Tasks; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Tasks; + +[Collection(ThreadPoolStarvationCollection.Name)] +public class SerialCallbackQueueStarvationTests +{ + [Fact] + public void When_the_thread_pool_is_starved_should_still_run_queued_callbacks() + { + //Arrange + var queue = new SerialCallbackQueue(); + using var confirmed = new ManualResetEventSlim(false); + + var starvedPool = new StarvedThreadPool(); + try + { + //Act + queue.Enqueue(() => + { + confirmed.Set(); // a publish confirmation reaching its subscriber + return Task.CompletedTask; + }); + var ran = confirmed.Wait(TimeSpan.FromSeconds(3)); + + //Assert + Assert.True(ran, "The queued callback did not run while every thread-pool worker was blocked"); + } + finally + { + starvedPool.Dispose(); + queue.Complete(); + } + } +} From ea92447c9ad56c8978260b7d9ba786de92bdc610 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 18:42:39 +0100 Subject: [PATCH 07/36] docs: release note for sweeper and publish-confirmation threading (#4560) Covers the dedicated sweeper and confirmation threads, the new sweeper metrics, the TimerInterval and subscriber-ordering behaviour changes, and the cost of one thread per confirming producer on small pods. Refs #4560 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- release_notes.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/release_notes.md b/release_notes.md index 7056a7c45b..35968abb17 100644 --- a/release_notes.md +++ b/release_notes.md @@ -2,6 +2,49 @@ ## Master +### Outbox sweeper and publish confirmations no longer depend on the thread pool (#4560) + +`TimedOutboxSweeper` used to run each sweep from a thread-pool `Timer`. When every pool thread was +busy or blocked, sweeps ran late or not at all, so messages awaiting a publish confirmation stayed in +the `InMemoryOutbox`. Kafka and RabbitMQ (`RMQ.Sync`) producers also queued one thread-pool work item +per publish confirmation, which added to that pressure under load. + +- **The sweeper runs on its own thread.** Sweeps now start on time even when the thread pool is + starved. +- **`StopAsync` waits for a sweep in flight** before it returns. +- **A failed sweep no longer crashes the process.** The sweeper logs the error and tries again on the + next sweep. It also releases its lock on every path, so one failure no longer blocks every later + sweep. +- **Kafka and `RMQ.Sync` raise publish confirmations on one thread per producer**, one at a time, in + delivery order. +- **New sweeper metrics.** `AddBrighterInstrumentation` registers + `paramore.brighter.outbox_sweeper.tick.lag`, `.sweep.duration` and `.sweeps` (by outcome). See + [ADR 0081](docs/adr/0081-timed-outbox-sweeper-health-metrics.md). + +#### Behaviour change: `TimerInterval` below 1 is rejected + +`TimedOutboxSweeperOptions.TimerInterval` must be at least 1 second. A value of 0 or less now throws +a `ConfigurationException` when the sweeper is created. Before, 0 swept once and then never again, +and a negative value threw when the sweeper started. + +#### Behaviour change: publish-confirmation subscribers run one at a time + +On Kafka and `RMQ.Sync`, subscribers to `OnMessagePublished` and to the awaited confirmation event +used to run concurrently. They now run one at a time, in the order the broker confirmed the +messages. A slow subscriber therefore delays the confirmations behind it. Brighter's own subscriber +marks the message dispatched in the outbox. + +#### Cost: one thread per confirming producer + +Each Kafka or `RMQ.Sync` producer now starts one background thread, with its first confirmation, and +keeps it until the producer is disposed. Both transports create one producer per publication, so an +app with twenty publications runs twenty confirmation threads, plus one sweeper thread. + +An idle thread costs little. Under load, though, those threads compete with the rest of the app for +CPU. That can matter on a pod limited to 1–2 CPUs. If you run many publications on a small pod, watch +CPU throttling (`container_cpu_cfs_throttled_periods_total` on Kubernetes) and the new sweeper +tick-lag metric. + ### AWS SQS, GCP Pub/Sub and RocketMQ: `requeueCount` now runs down (#4341, spec 0037) On these transports the broker re-serves its own stored copy of a requeued message, and Brighter never From 9d559582e3a18e028140e1e44232019357611034 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Thu, 8 Oct 2026 20:23:53 +0100 Subject: [PATCH 08/36] docs: bugfix record for TimedOutboxSweeper starvation (#4560) Triage, confirm evidence, regression tests, fix and verify for #4560. diagnostic/ keeps the starve and pileup experiments as an end-to-end tool; the samples/ harness was dropped, since the starvation and confirm-ordering regression tests already prove the fix. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../.confirm-approved | 0 .../.issue-number | 1 + .../bugfix.md | 708 ++++++++++++++++++ .../diagnostic/README.md | 49 ++ .../diagnostic/pileup/Program.cs | 165 ++++ .../diagnostic/pileup/pileup.csproj | 15 + .../diagnostic/starve/Program.cs | 80 ++ .../diagnostic/starve/starve.csproj | 11 + 8 files changed, 1029 insertions(+) create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/.confirm-approved create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/.issue-number create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/README.md create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/Program.cs create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/pileup.csproj create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/Program.cs create mode 100644 bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/starve.csproj diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/.confirm-approved b/bugfixes/0052-timed-outbox-sweeper-starvation/.confirm-approved new file mode 100644 index 0000000000..e69de29bb2 diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/.issue-number b/bugfixes/0052-timed-outbox-sweeper-starvation/.issue-number new file mode 100644 index 0000000000..cbd82cad48 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/.issue-number @@ -0,0 +1 @@ +4560 diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md b/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md new file mode 100644 index 0000000000..12eca28a77 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md @@ -0,0 +1,708 @@ +# Bugfix: TimedOutboxSweeper ticks are starved or delayed under load (InMemoryOutbox + Kafka async confirms, EKS) + +**Linked Issue**: #4560 +**Related**: #4554 (same outstanding-check defect as Confirmed Root Cause item 3, claimed by +@guilhermeappel; no PR yet), PR #4552 (same contributor; adds `GetAwaiter().GetResult()` inside the +same `Task.Run` check for async-only outboxes, i.e. more pool blocking on that path), draft PR #4484 +(outbox metrics; overlaps part of the telemetry gap) +**Status**: Verified + +## Symptom + +**Setup reported by the maintainer.** For speed, some teams combine three things: +- `CommandProcessor.Post` / `PostAsync` +- `InMemoryOutbox` +- a producer whose publisher confirms arrive out of band, most commonly `KafkaMessageProducer` + +`Post` writes to the in-memory outbox and dispatches. A message is only marked dispatched when the +broker's delivery report (the confirm) arrives. Anything never confirmed must be re-sent by +`TimedOutboxSweeper`, a hosted service in the same process. + +**Observed.** Under load on Kubernetes (specifically EKS), the sweeper's background work runs rarely +or not at all. Unconfirmed messages pile up in the `InMemoryOutbox` and are not re-sent in good +time. Because the outbox is in memory, a pod restart loses them for good. + +**Expected.** One sweep starts roughly every `TimerInterval` seconds (default 5 s, +`TimedOutboxSweeperOptions.cs:38`). Each sweep re-dispatches up to `BatchSize` (default 100, `:47`) +undispatched messages older than `MinimumMessageAge` (default 5 s, `:42`). This should keep +happening whatever the Post rate. + +**Reproduction.** There is no reliable reproduction; maintainers have not reproduced it. The reports +share these conditions: +- high Post rate +- Kafka with delivery reports +- container CPU limits, so `Environment.ProcessorCount` is small and CFS throttling is possible +- the sweeper in the same process as the publishers + +## Suspected Location + +All references below were checked against the working tree at `2890f8610`. + +**Sweeper host (timer and tick)** +- `src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs:77`: + `new Timer(Sweep, null, TimeSpan.Zero, TimeSpan.FromSeconds(_options.TimerInterval))`. This is a + `System.Threading.Timer`, so its callbacks are queued to the shared thread pool. The time source is + hard-wired: there is no `TimeProvider` or `ITimer` injection. +- `TimedOutboxSweeper.cs:104`: `private async void Sweep(object? state)`. Because it is async void, + there is no handle to await, and exceptions escape to the thread pool. +- `TimedOutboxSweeper.cs:106`: `ObtainLockAsync` is the first thing each tick does, which makes it + the best seam for timestamping when a tick actually runs. +- `TimedOutboxSweeper.cs:136`: the "Outbox Sweeper is still running - abandoning attempt" log (the + H2 signal). +- `TimedOutboxSweeper.cs:114`: resolves `IAmAnOutboxProducerMediator` from a new scope. +- `TimedOutboxSweeper.cs:124`: awaits `OutboxSweeper.SweepAsync()`. No cancellation token or timeout + is passed. +- `TimedOutboxSweeper.cs:47`: static logger from `ApplicationLogging.CreateLogger` + (`src/Paramore.Brighter/Logging/ApplicationLogging.cs:7-8`, a settable static `LoggerFactory`). +- `src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeperOptions.cs:38,42,47`: defaults + `TimerInterval = 5` (int seconds), `MinimumMessageAge = 5s`, `BatchSize = 100`. +- `src/Paramore.Brighter.Outbox.Hosting/HostedServiceCollectionExtensions.cs:41-50`: + `UseOutboxSweeper` registration via `AddHostedService()`. +- `src/Paramore.Brighter/OutboxSweeper.cs:71-80`: `SweepAsync` calls + `ClearOutstandingFromOutboxAsync` at `:79`, with no cancellation token. + +**Sweeper lock** +- `src/Paramore.Brighter/InMemoryLock.cs:44-49`: `SemaphoreSlim(1,1)` with + `WaitAsync(TimeSpan.Zero)`, so it never waits; it returns null if the lock is held. `:58-63` + releases it. +- `src/Paramore.Brighter.Extensions.DependencyInjection/ServiceCollectionExtensions.cs:410-411` and + `:541-545`: `InMemoryLock` is the default singleton `IDistributedLock`. + +**Mediator: singleton, shared by Post and the sweeper** +- `ServiceCollectionExtensions.cs:419-423` and `:557-574`: `IAmAnOutboxProducerMediator` is + registered `ServiceLifetime.Singleton`. The scoped resolution at `TimedOutboxSweeper.cs:114` + therefore returns the same instance that `Post` uses. +- `src/Paramore.Brighter/OutboxProducerMediator.cs:190`: `ConfigureCallbacks` is called once, in the + constructor. + - At `:819-834`, the `switch` tests `IAmAMessageProducerAsync` first (`:826-827`). + `KafkaMessageProducer` implements both sync and async, so the **async** callback path is the one + wired up. + - `:843-858` `ConfigureAsyncPublisherCallbackMaybe` → `:860-906` `HandleAsyncPublishConfirmation`, + which runs `_asyncOutbox.MarkDispatchedAsync` inside `ExecuteWithResiliencePipelineAsync` + (`:879-884`). + - For Kafka, the sync `ConfigurePublisherCallbackMaybe` / `_outBox.MarkDispatched` delegate + (`:952-1009`, `:977`) is not the subscriber. +- `OutboxProducerMediator.cs:715-777` `BackgroundDispatchUsingAsync`: + - `:724` calls `_outboxCircuitBreaker?.CoolDown()`. + - `:726` does `_backgroundClearSemaphore.WaitAsync(TimeSpan.Zero, …)`; the semaphore is a + per-instance field at `:70`. + - `:737-738` queries outstanding messages, excluding `TrippedTopics`. + - `:753` calls `DispatchAsync`. + - `:762` rethrows; `:775` logs `SkippingDispatchOfMessages`. +- `OutboxProducerMediator.cs:495-506`: `ClearOutstandingFromOutboxAsync` is the only public entry + into `BackgroundDispatchUsingAsync`. Its only caller in `src/` is `OutboxSweeper.cs:79`. +- `OutboxProducerMediator.cs:1175-1234` `DispatchAsync`: a sequential `foreach`. Each message awaits + `producerAsync.SendAsync` (`:1203`) through the resilience pipeline. For a confirming producer it + does not mark dispatched (`:1211`); on failure it trips the topic (`:1223`). + +**Outstanding-count check: runs after every Post and blocks pool threads** +- `OutboxProducerMediator.cs:411` (`ClearOutbox`), `:480` (`ClearOutboxAsync`), `:770`, `:1400` and + `:1432` all call `CheckOutstandingMessages`. +- `:795-814` `CheckOutstandingMessages`: gated on + `now - _lastOutStandingMessageCheckAt >= _maxOutStandingCheckInterval` (default 1 s, `:184`; + initialised at `:162`). It then does `Task.Run(() => OutstandingMessagesCheck(...))` at `:811`. +- `:1332-1378` `OutstandingMessagesCheck`: `s_checkOutstandingSemaphoreToken.Wait()` at `:1334` is a + **blocking** wait on a pool thread. The semaphore is **static**: `:74`, shared by every mediator in + the process. `_lastOutStandingMessageCheckAt` is only updated at `:1336`, *after* the queued item + has run and taken the semaphore. Until then, every `Post` that passes the gate at `:803` queues + *another* `Task.Run`. +- `:1344-1349`: `GetOutstandingMessageCount` on the sync outbox. + +**Post entry points** +- `src/Paramore.Brighter/CommandProcessor.cs:714-721`: `Post` → `ClearOutbox` → mediator + `ClearOutbox` → `Dispatch` (`OutboxProducerMediator.cs:1035-1090`) → `producerSync.Send` + (`:1060-1062`). For confirming producers it does not mark dispatched. +- `src/Paramore.Brighter/CommandProcessor.cs:776-787`: `PostAsync` → `ClearOutboxAsync` → + `DispatchAsync`. + +**Kafka producer** +- `src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs:230-233,258-323`: sync + `Send` → `SendWithDelay` → `_publisher.PublishMessage(...)` at `:295`. + - `src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessagePublisher.cs:40-45` calls + `producer.Produce(topic, msg, deliveryReport)`. That is fire-and-enqueue: no `Flush`, no wait. + - The delivery-report callback is invoked on Confluent's own poll/delivery thread, not a pool + thread. This is Confluent.Kafka behaviour and is not in this repo, so it is **unverified here**. + - `Flush` is only called from `Dispose` (`KafkaMessageProducer.cs:393-403`, `:399`). + - The only sync-over-async in the send path is the scheduler branch (`:274`, + `BrighterAsyncContext.Run`), which only applies when there is a delay. +- `KafkaMessageProducer.cs:245-248,334-391`: `SendAsync` → `PublishMessageAsync` + (`KafkaMessagePublisher.cs:47-70`), which **awaits `producer.ProduceAsync`** (`:53`). It then calls + the delivery-report action inline on the continuation, so each awaited send waits for the broker + ack. +- `KafkaMessageProducer.cs:405-446` `PublishResults`, the Brighter work done on the delivery-report + thread: it parses the `MESSAGE_ID` header, may log, then calls `RaisePublishConfirmation`. +- `KafkaMessageProducer.cs:448-472` `RaisePublishConfirmation`: + - `_confirmationCallbacks.Begin()` (`:455`) takes a short lock + (`src/Paramore.Brighter/Tasks/InFlightCallbackTracker.cs:55-57`). + - It then does `Task.Run(async () => { OnMessagePublished?.Invoke; await + _onMessagePublishedAsync.InvokeAllAsync(result); })` at `:456-471`, one pool work item per + delivery report. + - On the sync `Produce` path, the `Task.Run` is issued from a non-pool thread (Confluent's), so it + goes to the pool **global** queue. On the `ProduceAsync` path it is issued from a pool thread, so + it goes to that thread's **local** queue. This is .NET runtime behaviour, **unverified here**. +- `src/Paramore.Brighter.MessagingGateway.Kafka/KafkaPublication.cs:75,81,88,94,130`: defaults + `LingerMs = 5`, `MessageSendMaxRetries = 3`, `MessageTimeoutMs = 5000`, + `MaxInFlightRequestsPerConnection = 1`, `RequestTimeoutMs = 500`. + +**InMemoryOutbox** +- `src/Paramore.Brighter/InMemoryOutbox.cs:548-585` `OutstandingMessages`: + `Requests.Values.OrderBy(TimeStamp)` over **all** retained entries, including dispatched ones + (`:571-572`), *before* `.Where(undispatched && old enough && topic not tripped)`, then + `.Take(pageSize)` (`:573-578`). +- `InMemoryOutbox.cs:617-644` `GetOutstandingMessageCount`: the same full `OrderBy` (`:631-632`). It + runs about once a second per mediator via the outstanding check. +- `InMemoryOutbox.cs:462-478` and `:598-614`: the async methods are synchronous work wrapped in a + `TaskCompletionSource` with `RunContinuationsAsynchronously`, which schedules continuations on the + pool. +- `InMemoryOutbox.cs:514-536` `MarkDispatched`: a dictionary lookup that sets `TimeFlushed`. +- `src/Paramore.Brighter/InMemoryBox.cs:59,68,76`: `EntryLimit = 2048`, `EntryTimeToLive = 5 min`, + `ExpirationScanInterval = 10 min`. Only *dispatched* entries are expired or compacted + (`InMemoryOutbox.cs:76-80`, `:658-665`). + +**Other pool blocker in the same host** +- `src/Paramore.Brighter.Outbox.Hosting/TimedOutboxArchiver.cs:73`: the timer callback does + `Archive(cancellationToken).GetAwaiter().GetResult()`, which blocks a pool thread for each archive + run if the archiver is also registered. + +**Existing tests** +- There are **no `TimedOutboxSweeper` tests**: no `.cs` file under `tests/` references it. +- `OutboxSweeper` is covered by: + - `tests/Paramore.Brighter.InMemory.Tests/Sweeper/When_sweeping_the_outbox.cs`, + `When_sweeping_the_outbox_with_circuit_breaker.cs` and + `When_clearing_outbox_with_missing_messages.cs`. This project does **not** reference + `Paramore.Brighter.Outbox.Hosting`; + `tests/Paramore.Brighter.InMemory.Tests/Paramore.Brighter.InMemory.Tests.csproj:28-29` lists only + `Paramore.Brighter` and `Base.Test`. + - `tests/Paramore.Brighter.Core.Tests/CommandProcessors/Clear/When_Implicit_Clearing_The_PostBox_On_The_Command_Processor*.cs`, + `When_Bulk_Clearing_The_PostBox_On_The_Command_Processor_Async.cs` and + `When_independent_mediators_sweep_concurrently_should_clear_both_outboxes.cs`. Core.Tests + **does** reference Outbox.Hosting + (`tests/Paramore.Brighter.Core.Tests/Paramore.Brighter.Core.Tests.csproj:12`) and has + `TimedOutboxArchiver` tests under `tests/Paramore.Brighter.Core.Tests/Archiving/`, which are the + closest pattern to copy. +- Precedent for pool starvation in tests: + `tests/Paramore.Brighter.InMemory.Tests/ThreadPoolSaturatingCollection.cs:29-38` + (`DisableParallelization = true`). Its comment cites #4475, where saturating `Task.Run` items + starved `InMemoryBox` background work "for tens of seconds" on small CI runners. +- In-process stand-in for Kafka's out-of-band confirm: + `src/Paramore.Brighter/InMemoryMessageProducer.cs:112` (`UseAsyncPublishConfirmation { get; init; + }`). With it, `:237-248` sends become fire-and-forget onto a background worker, and `:264-281` + confirms asynchronously, so a test does not need a broker. + +## Root-Cause Hypothesis + +> All hypotheses below are **UNVERIFIED — to be proven or refuted in /bugfix:confirm**. + +### H1 (leading): thread-pool starvation delays the sweeper's timer tick + +**The chain** +1. `TimedOutboxSweeper.cs:77` schedules sweeps with a `System.Threading.Timer`. The runtime timer + thread does not run `Sweep`; it queues the callback to the shared pool's global FIFO queue (.NET + runtime behaviour, unverified here). +2. At a high Post rate with Kafka, every delivery report adds one `Task.Run` work item + (`KafkaMessageProducer.cs:456`). That item runs `HandleAsyncPublishConfirmation` + (`OutboxProducerMediator.cs:860-906`): span creation, a `RequestContext` copy, the resilience + pipeline, then `MarkDispatchedAsync`, whose continuation is posted back to the pool + (`InMemoryOutbox.cs:471`). So each confirm costs at least two pool work items. +3. Two amplifiers turn a busy pool into a **blocked** pool: + - **(a) Outstanding-check pile-up.** `CheckOutstandingMessages` + (`OutboxProducerMediator.cs:795-814`) runs on every `ClearOutbox` / `ClearOutboxAsync` (`:411`, + `:480`), i.e. on every Post. The timestamp that gates it is only refreshed by the queued work + item itself (`:1336`). When the pool is slow, every Post in the gap passes the gate at `:803` + and queues another `Task.Run` (`:811`). Each of those, once running, **blocks a pool thread** on + the static `s_checkOutstandingSemaphoreToken.Wait()` (`:1334`) while one at a time runs an + O(n log n) `OrderBy` over the whole outbox (`InMemoryOutbox.cs:631-632`). That is positive + feedback: starvation → more queued checks → more blocked threads → more starvation. + - **(b) Other blocking.** Synchronous `Post` pipelines on request threads, and + `TimedOutboxArchiver.cs:73` `.GetAwaiter().GetResult()` if the archiver is used. +4. Under a Kubernetes CPU limit, `Environment.ProcessorCount` (and hence the pool's minimum worker + count) is small. The pool then adds threads only gradually, through hill-climbing and starvation + detection, so blocked threads are not quickly replaced (runtime behaviour, unverified here). CFS + throttling also freezes every thread in the cgroup for the rest of each 100 ms period once its + quota is used up. +5. Result: the sweeper's tick sits behind a long queue, and its own awaits + (`TimedOutboxSweeper.cs:106,124`; every `SendAsync` in `DispatchAsync` at + `OutboxProducerMediator.cs:1203`) are posted back to the same starved pool. + +**Feedback loop through re-sends.** Starvation also delays `MarkDispatched` itself. If +confirm-to-mark latency exceeds `MinimumMessageAge` (5 s), a sweep that does run picks up messages +that were in fact delivered and re-sends them. That creates more delivery reports and more +`Task.Run` items. The defaults (`TimerInterval` 5 s = `MinimumMessageAge` 5 s) leave no headroom. + +**Pool-queue ordering (unverified, .NET runtime behaviour).** The timer callback goes to the global +queue. Pool workers drain their **local** queues first, before taking from the global queue. Work +spawned from pool threads (the `ProduceAsync` continuation path; `Task.Run` at `:811` when called +from a pool thread) therefore jumps ahead of the sweeper tick. + +#### Falsifiable predictions for H1 + +**P1-a: tick latency under saturation.** In an isolated process: +- Constrain the pool: `DOTNET_PROCESSOR_COUNT=1` or `2`, plus `ThreadPool.SetMinThreads(1–2, …)` + and, where allowed, `SetMaxThreads`. Note that `SetMaxThreads` returns false below the processor + count, so the env var or runtimeconfig is what actually lowers the ceiling. +- Before calling `StartAsync`, put N `Task.Run` items on the global queue that each block for X ms + (e.g. N = 200, X = 50 ms, simulating confirm callbacks that hit a sync outbox or the semaphore at + `:1334`). +- Prediction: the first `ObtainLockAsync` call on a **recording `IDistributedLock`** comes later than + `StartAsync` by roughly ≥ N·X / workers, i.e. much more than `TimerInterval` (the first tick is due + at `TimeSpan.Zero`, `:77`). +- Prediction: under *sustained* saturation, gaps between ticks are much larger than `TimerInterval` + and irregular. +- Control: a loop on a **dedicated thread** (`new Thread` or `TaskCreationOptions.LongRunning` with a + synchronous wait) under the same load keeps its ticks within ~`TimerInterval` + ε. +- **Refuted if** the `System.Threading.Timer` tick latency stays below `TimerInterval` under the same + saturation. + +**P1-b: outstanding-check amplifier.** With `InMemoryMessageProducer { UseAsyncPublishConfirmation = +true }` and a constrained pool, drive M Posts per second for T seconds. +- Prediction: the number of `OutstandingMessagesCheck` executions (or `RunningOutstandingMessageCheck` + log lines at `:809`) is much greater than T / `maxOutStandingCheckInterval`. +- Prediction: the peak number of pool threads blocked at `:1334` is greater than 1. +- **Refuted if** the count stays ≤ ~T / 1 s and there is never more than one waiter. + +**P1-c: re-sends from late confirms.** Under the same load, count messages the sweeper sends that +already had a success confirmation queued. Prediction: greater than 0 once p99 +confirm-to-`MarkDispatched` latency exceeds `MinimumMessageAge`, and 0 without saturation. + +**P1-d: production signals that separate H1 from H2.** +- Gaps between consecutive "Outbox Sweeper looking for unsent messages" logs are much larger than + `TimerInterval`, with **no** "still running - abandoning attempt" warnings in between. +- `dotnet-counters` `threadpool-queue-length` keeps rising and `threadpool-thread-count` creeps up + slowly. +- Kubernetes `container_cpu_cfs_throttled_periods_total` is non-zero. + +#### Testability obstacles for a deterministic regression test +- `TimedOutboxSweeper` constructs its `Timer` directly (`:77`). There is no `TimeProvider` / + `ITimer` seam, so the test cannot fake time and must use real wall-clock bounds. Assertions must + be "delay > k·TimerInterval" with generous margins, not exact values. +- `Sweep` is private `async void` (`:104`), so there is no task to await and no completion event. + - Usable seams: the injected `IDistributedLock` (its `ObtainLockAsync` is the first call per tick, + `:106`) and the injected `IServiceScopeFactory` (`:111`, `:114`, which can return a fake mediator + that records `ClearOutstandingFromOutboxAsync`). + - `TimerInterval` is an `int` in seconds (`TimedOutboxSweeperOptions.cs:38`), so the smallest + period is 1 s and tests will take seconds. +- The logger is static (`TimedOutboxSweeper.cs:47` via the settable static + `ApplicationLogging.LoggerFactory`). Capturing logs means swapping a process-wide static: serialise + such tests (compare `tests/Paramore.Brighter.Extensions.Tests/LoggerCaptureCollection.cs`) or rely + on the lock/scope seams instead. +- Pool limits are process-wide: + - `ThreadPool.SetMinThreads` / `SetMaxThreads` and `DOTNET_PROCESSOR_COUNT` affect the whole test + host, and `DOTNET_PROCESSOR_COUNT` must be set before the runtime starts. + - A trustworthy test needs a **separate child process** (no `RemoteExecutor`-style helper exists + in `tests/`; a grep found none), or a dedicated test project whose runtimeconfig / env sets the + limits. + - At minimum it needs an xUnit collection with `DisableParallelization = true`, following + `ThreadPoolSaturatingCollection.cs`. Even then, other test assemblies running in parallel under + `dotnet test` share neither the process nor its pool, but they do compete for CPU. +- `InMemory.Tests` does not reference `Paramore.Brighter.Outbox.Hosting`. A test of + `TimedOutboxSweeper` belongs in Core.Tests (which already references it), in a new isolated + project, or InMemory.Tests needs a new project reference. +- `s_checkOutstandingSemaphoreToken` is `static` (`OutboxProducerMediator.cs:74`). Mediators in other + tests in the same process share it, so P1-b must run in isolation. + +### H2: a sweep hangs inside `DispatchAsync` and holds the lock +`DispatchAsync` sends one message at a time, awaiting each `SendAsync` +(`OutboxProducerMediator.cs:1187-1203`). For Kafka that awaits `ProduceAsync` until the broker acks +or fails (`KafkaMessagePublisher.cs:53`), which can take up to `MessageTimeoutMs` (5000 ms default, +`KafkaPublication.cs:88`), with `MessageSendMaxRetries = 3` inside librdkafka, plus any Brighter +resilience-pipeline retries. With `BatchSize = 100`, a degraded broker can make one sweep take +minutes. No cancellation or timeout is passed (`OutboxSweeper.cs:79`, `TimedOutboxSweeper.cs:124`). +`InMemoryLock` stays held (`InMemoryLock.cs:48`) until `ReleaseLockAsync` at +`TimedOutboxSweeper.cs:129`. +- **Distinguishing signal:** every tick in that window logs "Outbox Sweeper is still running - + abandoning attempt" (`:136`) at the expected 5 s cadence, i.e. ticks *are* being scheduled. + Per-message `SendAsync` durations approach `MessageTimeoutMs`. +- **Falsified if** no "abandoning attempt" warnings appear during the stall. + +### H3: `_backgroundClearSemaphore` contention → `SkippingDispatchOfMessages` +The semaphore is per mediator instance (`OutboxProducerMediator.cs:70`). The mediator is a singleton +(`ServiceCollectionExtensions.cs:419-423`, `:557-574`). The only path into +`BackgroundDispatchUsingAsync` is `ClearOutstandingFromOutboxAsync` (`:495-506`), and in `src/` its +only caller is `OutboxSweeper.cs:79`. `Post` / `PostAsync` take the explicit-id `ClearOutbox` / +`ClearOutboxAsync` path (`CommandProcessor.cs:720`, `:786`), which does **not** touch this +semaphore. Within one `TimedOutboxSweeper`, the `InMemoryLock` already serialises sweeps, so +contention needs a second caller: a second sweeper registration, or user code calling +`ClearOutstandingFromOutboxAsync`. +- **Low prior.** +- **Distinguishing signal:** the `SkippingDispatchOfMessages` log (`:775`) shows up alongside + "looking for unsent messages" logs. + +### H4: sweeps run but are slow or filtered +- `OutstandingMessages` sorts **all** retained entries, including dispatched ones, before filtering + (`InMemoryOutbox.cs:571-578`). Dispatched entries are kept for 5 min (`InMemoryBox.cs:68`), and the + expiry scan runs every 10 min (`:76`). So at high throughput, every sweep (and every 1 s + outstanding check, `:631-632`) sorts a large collection, which adds CPU pressure and feeds H1. +- Topics whose circuit breaker has tripped are excluded (`:576`, fed from + `OutboxProducerMediator.cs:738`). Failed confirms trip the wire topic (`:893`) and failed sends do + too (`:1223`). Cool-down happens only once per sweep (`:724`), so if confirms keep failing, a topic + can stay excluded. +- **Distinguishing signal:** `FoundMessagesToClear` (`:745`) reports 0 or far fewer messages than are + outstanding, while ticks are on schedule; or sweep duration grows with `EntryCount`. + +### H5: unhandled exception from `async void Sweep` (side observation) +`BackgroundDispatchUsingAsync` rethrows (`OutboxProducerMediator.cs:762`) after logging, for example +when `OutstandingMessagesAsync` or producer lookup fails. `OutboxSweeper.SweepAsync` does not catch. +`TimedOutboxSweeper.Sweep` is `async void` with only `try/finally` (`:104-133`), so the exception +escapes to the thread pool and, by default .NET behaviour, kills the process. On Kubernetes that +appears as pod restarts, and every unsent message in the in-memory outbox is lost. +`TimedOutboxArchiver` does catch everything (`TimedOutboxArchiver.cs:129-132`); the sweeper does not. +- **Distinguishing signal:** `ErrorWhileDispatchingFromOutbox` followed by an unhandled-exception + crash or container restart count above 0. + +### H6: configuration check +`TimeSpan.FromSeconds(_options.TimerInterval)` is used as the timer **period** +(`TimedOutboxSweeper.cs:77`). Per `System.Threading.Timer` semantics, a period of zero or `Infinite` +turns off repeating, so `TimerInterval = 0` would sweep exactly once ("never scheduled" afterwards). +- **Distinguishing signal:** exactly one "looking for unsent messages" log after start-up. Check the + affected teams' `UseOutboxSweeper` options. + +### Checked and ruled out as causes on their own +- The sweeper's scoped resolution of `IAmAnOutboxProducerMediator` returns the same singleton `Post` + uses, so there is no split-brain outbox (`ServiceCollectionExtensions.cs:419-423`). +- The Kafka sync `Send` path does not `Flush` or block on a pool thread. `Produce` enqueues to + librdkafka (`KafkaMessagePublisher.cs:44`), and `Flush` is only on dispose + (`KafkaMessageProducer.cs:399`). +- On Confluent's delivery thread, Brighter does only header parsing and a short lock before handing + off with `Task.Run` (`KafkaMessageProducer.cs:405-471`). + +## Maintainer Guidance (recorded after triage) + +These loosen the testability obstacles listed under H1: + +- **`TimedOutboxSweeper` tests may be added.** There are none today. +- **A time seam is acceptable.** Add an optional `TimeProvider` parameter that defaults to the real + timer. Brighter guarantees **source** compatibility only, not binary, so an optional constructor + parameter is fine. +- **`Sweep` may be made public for testing.** That gives a test something to call and await + directly, instead of inferring ticks from a recording `IDistributedLock`. +- **A harness under `samples/` is acceptable.** It is preferred to a test that tries to control + process-wide thread-pool limits. Use it to reproduce starvation under constrained CPU + (`DOTNET_PROCESSOR_COUNT`, `docker --cpus`) with Post + InMemoryOutbox + async confirms + (`InMemoryMessageProducer { UseAsyncPublishConfirmation = true }` and/or Kafka). +- **OpenTelemetry blind spots are in scope.** If the sweeper's health cannot be seen through + Brighter's telemetry, fixing that is part of this defect. Gaps found so far (to be confirmed): + - **No tick-lag signal.** Nothing records the delay between when a tick was due and when `Sweep` + actually ran, which is the H1 signal. + - **Lock-abandon and mediator-skip are logs only.** The H2 "still running - abandoning attempt" and + H3 `SkippingDispatchOfMessages` events have no metric. + - **Sweep duration and messages found per sweep are visible only in sampled traces.** The + "Implicit Clear Outbox" span is created via the obsolete `ApplicationTelemetry` + (`OutboxSweeper.cs`). It shares the `Paramore.Brighter` source name, so it is exported, but it + is not a metric. + - **No gauge for the outbox's outstanding count.** `_outStandingCount` is computed in + `OutboxProducerMediator` but not exported. + - **No confirm-to-dispatched latency.** The time from `Post` to `MarkDispatched` is the P1-c + signal. + - **Thread-pool health is a runtime metric, not Brighter's.** Queue length and thread count come + from the .NET `System.Runtime` meter. The sample and docs should turn it on rather than Brighter + re-exporting it. + +## Confirmed Root Cause + +**Verdict: H1 PARTIALLY CONFIRMED.** The mechanism is proven, but the driver named in triage is +refuted as the cause on its own. A separate, more severe defect was proven along the way. + +1. **Proven: the sweeper's tick is only as reliable as the shared thread pool.** + `TimedOutboxSweeper` runs `async void Sweep` from a `System.Threading.Timer` + (`TimedOutboxSweeper.cs:77`, `:104`). When pool workers are **blocked or monopolised**, ticks are + delayed 3–4.5× the interval. A dedicated thread under the same load ticks exactly on time + (Experiments A and C). The comment at `:128`, "on a timer thread, so blocking is OK", is wrong: + the callback runs on a pool thread. + +2. **Refuted as a sole cause: the per-confirm `Task.Run` flood.** On .NET 9 a timer firing is + **not** queued behind ordinary pool work. 100k+ pending short `Task.Run` items did not delay a + single tick (Experiment A, flood mode). The likely reason is that the runtime queues timer + firings at high priority; that is runtime behaviour, observed rather than read in source. Under + fair load, where callers yield as real request traffic does, the Kafka-style per-confirm + `Task.Run` kept up and the sweeper ticked at 1.00 s (Experiment D). Ticks are delayed only when + workers are **busy or blocked**, not when the queue is merely long. + +3. **Proven, separate and severe: the outstanding-message check runs on every Post under the + default DI wiring.** It blocks pool threads and collapses throughput. + - **The interval defaults to zero under DI.** `ProducersConfiguration.MaxOutStandingCheckInterval` + defaults to `TimeSpan.Zero` (`ProducersConfiguration.cs:207`), and DI passes it straight + through (`ServiceCollectionExtensions.cs:849`). The mediator's 1 s fallback, + `?? TimeSpan.FromMilliseconds(1000)` at `OutboxProducerMediator.cs:184`, applies only to + `null`, so it never kicks in. + - **So every Post queues a check.** With a Zero interval, the gate at `:803` never holds, and + every `ClearOutbox` / `ClearOutboxAsync` (`:411`, `:480`) calls `Task.Run(OutstandingMessagesCheck)` + (`:811`). + - **Each check blocks a pool thread and sorts the whole outbox.** It parks the thread on a + **static** semaphore with a synchronous `Wait()` (`:1334`), then runs an `OrderBy` over every + retained outbox entry (`InMemoryOutbox.cs:631-632`). + - **The count is computed even when there is no limit.** It runs when `MaxOutStandingMessages` + is -1, the default (`ProducersConfiguration.cs:200`), even though the result is then never used + (`:788`). The docs at `ProducersConfiguration.cs:205` say the opposite. + - **Even with a 1 s interval, nothing stops the checks piling up.** The timestamp is refreshed + only *inside* the queued item, after `Wait()` (`:1336`), and there is no in-flight guard. + Under overload this queued 235k–365k checks in 12 s (Experiment C). + - **Measured cost under fair load:** about 1,270 checks/s, and Post throughput fell **~30×** + (11.7k vs 354k posts in 9 s, Experiment D). With a DB outbox the same path is a blocking COUNT + query on every Post. + +4. **Proven consequence under overload: delivered messages are re-sent.** Late confirms leave + delivered messages looking undispatched past `MinimumMessageAge`, so the sweeper re-sends them. + Experiment C saw 100 re-sends per sweep. + +**Not established:** what blocks or monopolises pool threads in the reporters' EKS pods. Candidates: +- the outstanding check (3) +- `KafkaMessageConsumer.ReceiveAsync`, which runs a blocking `Consume(timeout)` inside `Task.Run` + (`KafkaMessageConsumer.cs:577-591`), pinning a pool thread for every receive when consumers share + the process +- `TimedOutboxArchiver.cs:73` `GetAwaiter().GetResult()` +- `InMemoryScheduler.cs:312,321` `BrighterAsyncContext.Run` +- sync-over-async in application code +- CPU (CFS) throttling + +A `dotnet-dump` of an affected pod (`threadpool`, `clrstack -all`) would settle this. + +## Evidence + +- [x] **Code trace** (Plan sub-agent, opus; all triage line references re-verified at `2890f8610`): + 1. `TimedOutboxSweeper.cs:77` → pool `Timer` → `async void Sweep` (`:104`). There is no seam and + no telemetry. + 2. DI: `ProducersConfiguration.cs:200,207` → `ServiceCollectionExtensions.cs:848-849` → + `OutboxProducerMediator.cs:184`, so the interval stays Zero. + 3. `Post`: `CommandProcessor.cs:720` → `OutboxProducerMediator.cs:411`. `PostAsync`: + `CommandProcessor.cs:786` → `:480`. Both reach `CheckOutstandingMessages` (`:795-814`) → + `Task.Run` (`:811`) with no dedup. + 4. `:1334` blocks on the static `SemaphoreSlim` (`:74`); `:1336` refreshes the timestamp only + after that; `:1340-1360` counts even at limit -1. + 5. Kafka confirms are wired to the **async** callback (`:826`, `:845`; + `KafkaMessageProducer.cs:51`). That costs one `Task.Run` per report (`:456`) and **never + blocks**: `MarkDispatchedAsync` completes synchronously (`InMemoryOutbox.cs:471-477`). + **Correction to triage:** one pool item per confirm, not two. + 6. **Correction to triage:** the sweep's own lock and outstanding-query awaits complete + synchronously. Only each `SendAsync` (`:1203`) hops to the pool. + +- [x] **Executable experiments.** These are scratch console apps in the session scratchpad that + reference the repo projects; they are not committed. All ran on net9.0 with + `DOTNET_PROCESSOR_COUNT` constrained, measuring real `TimedOutboxSweeper` ticks through a + recording `IDistributedLock` at `TimerInterval = 1`. + + **Experiment A** (`starve/`): which kinds of pool load delay the timer? + + | Load | procs | Real sweeper (pool `Timer`) | Dedicated thread | `PeriodicTimer` async loop | + |---|---|---|---|---| + | none | 1 / 2 / 10 | 10 ticks, max gap 1.00 s | 10, 1.00 s | 10, 1.00 s | + | 200 blocked pool items | 1 | **7 ticks, first tick at 1.01 s, max gap 3.03 s** | 10, 1.00 s | **4, 2.00 s** | + | 200 blocked pool items | 2 | **6 ticks, max gap 3.01 s** | 10, 1.02 s | **5, 3.02 s** | + | flood: 20k/s `Task.Run` from a non-pool thread, 100k pending | 1 / 2 | 10, 1.00 s | 10, 1.00 s | 10, 1.00 s | + + **Experiments C and D** (`pileup/`): real `CommandProcessor` + `OutboxProducerMediator` + + `InMemoryOutbox` + real `TimedOutboxSweeper` with `InMemoryLock`, and a fake producer that mimics + Kafka (a non-pool "delivery thread" doing one `Task.Run` per confirm). 64 concurrent posters, + 12 s, procs = 2. + - **C, saturating load** (posters never yield): + - The sweeper's max gap was 3.5–4.5 s in every configuration, including with the check disabled. + - Outstanding-check backlog: 235k at the 1 s interval, 365k at the Zero (DI) interval. + - About 100 delivered messages were re-sent per sweep. + - **D, fair load** (posters `await Task.Yield()`): + - Check disabled: 350k posts, sweeper max gap 1.00 s, 0 re-sends. + - 1 s interval: 354k posts, 12 checks, max gap 1.01 s. + - Zero (DI default): **11.7k posts (~30× fewer)**, 11.7k checks, max gap 1.00 s. + +- [ ] **Red repro for the durable test.** This is deterministic and needs no pool limits. + - **Setup:** build the mediator as DI does (`maxOutStandingMessages: -1`, + `maxOutStandingCheckInterval: TimeSpan.Zero`), with an outbox decorator whose + `GetOutstandingMessageCount` counts calls and blocks on a gate. Post K times, then open the gate. + - **Today:** K checks are queued and run, and K−1 pool threads park at `:1334`. + - **Assertion:** with limit -1, zero checks run. With a limit set, at most one check is in flight + or queued at any time. + - The sweeper-tick test (a tick runs while N pool items are blocked) needs the new seam: either a + dedicated-thread loop or the `TimeProvider` option. + +## Suggested-Fix Assessment + +- **Outstanding check (not in the original fix list; should come first): CONFIRMED needed.** + - Skip it when `MaxOutStandingMessages == -1`. + - ~~Treat a Zero interval as the 1 s default~~ — **rejected on #4554**: the interval is also the + minimum age passed to `GetOutstandingMessageCount`, so the maintainer agreed to keep Zero. + Skipping at -1 removes the per-Post check for default users anyway. Fix the docs. + - Add an `Interlocked` in-flight flag and set the timestamp before queuing. + - Use a non-blocking acquire instead of `Wait()`. + - Make the semaphore per instance. +- **(i) Dedicated sweep thread + optional `TimeProvider` + public `Sweep`: CONFIRMED for the + tick, PARTIAL for the sweep body.** + - Experiment A shows a dedicated thread is immune and a `PeriodicTimer` async loop is **not**. A + plain `BackgroundService` with `await PeriodicTimer` therefore does not fix this. + - After the first real `SendAsync`, the sweep body still continues on the pool. Bound it with a + cancellation token or timeout. + - Replace `async void` with a catch-all, as `TimedOutboxArchiver.cs:129-132` does. +- **(ii) Stop one `Task.Run` per Kafka delivery report: PARTIAL.** It is not what starves the tick, + but under overload it produces an unbounded queue (~1 pending item per Post in Experiment C), + delayed `MarkDispatched`, and re-sends of delivered messages. Per your note, it is worth fixing on + its own merits: a single-reader `Channel` drained by a long-running worker (the pattern in + `InMemoryMessageProducer.cs:264-281`), or bounded parallelism. +- **(iii) OpenTelemetry: CONFIRMED gap.** Nothing today measures: + - tick lag + - sweep duration and messages found + - abandon and skip counts (`:136`, `:775`) + - outstanding-check queued and in-flight counts + - the outstanding gauge + - confirm-to-dispatched latency + +## Scope Notes + +- **Outstanding-check defect affects every outbox, not just InMemory.** With a DB outbox it is a + blocking COUNT query on every Post. Docs disagree: interface `:89` vs `ProducersConfiguration.cs:205`. +- **H2 refuted as "never completes"; it is "slow but bounded".** + - Each `ProduceAsync` is bounded by `MessageTimeoutMs` (5 s), and Polly makes 3 attempts. + - The worst case is about 8 minutes per 100-message sweep, during which every tick logs "abandoning". + - No cancellation token is passed anywhere. +- **H5 confirmed (conditional).** Two failure modes: + - Exceptions rethrown at `OutboxProducerMediator.cs:762` escape `async void Sweep` and crash the + process; one example is a `ConfigurationException` from `LookupBy` at `:1194`. + - **New:** `CreateScope()` at `TimedOutboxSweeper.cs:111` is **outside** the `try`. If it throws + (for example during shutdown), `InMemoryLock` is never released and every later tick logs + "abandoning attempt" forever. +- **H6 confirmed as possible.** `TimerInterval` is not validated: 0 sweeps once, and a negative value + makes `StartAsync` throw. +- **Same timer/blocking pattern elsewhere:** + - `TimedOutboxArchiver.cs:73` (`GetAwaiter().GetResult()` in the timer) and `:120` + - `HeartbeatHostedService.cs:25` + - `InMemoryScheduler.cs:312,321` + - `TimeoutPolicyHandler.cs:120-122,152` + - `KafkaMessageConsumer.cs:577-591` (blocking `Consume` inside `Task.Run`), `:231`, `:421`, `:934` +- **Per-confirm `Task.Run` parity:** `RMQ.Sync/RmqMessageProducer.cs:282-307` has the same pattern. + `RMQ.Async` does not. +- **Test stand-in fidelity:** `InMemoryMessageProducer { UseAsyncPublishConfirmation = true }` runs + confirms serially. It also adds every raise task to `_raiseTasks` (`:50`, `:286`), which grows + without bound in long runs, so it is a poor stand-in for Kafka in the sample harness. +- **Minor:** + - `KafkaMessageProducer.cs:297` catches `ProduceException`, but the producer is + ``, so that catch never matches. + - `InMemoryBox.cs:72` comment says 5 min; the value at `:76` is 10 min. +- **Agreed scope (#4560):** the sweeper (dedicated tick thread, optional `TimeProvider`, public + awaitable `Sweep`, catch-all, lock released on every path, `TimerInterval` validation, cancellation), + a bounded-channel confirm drain for Kafka and RMQ.Sync, sweeper OpenTelemetry metrics, and a + `samples/` harness. **The harness was later dropped** (maintainer, 2026-10-08): the starvation + regression tests and the live-broker confirm-ordering tests already prove the fix, and + `diagnostic/` stays as the end-to-end tool. **Out of scope:** the outstanding check (owned by #4554; evidence posted + there) and #4552 (review note posted). +- **Superseded recommendation: split the work.** Fix the outstanding check and make the sweeper robust (dedicated + tick thread, catch-all, lock leak, interval validation, telemetry) in this bugfix. Track the Kafka + and RMQ.Sync per-confirm `Task.Run` and the other timer/blocking sites as follow-up issues, or + widen this fix if you prefer. + +## Regression Test + +1. `tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_should_still_sweep_the_outbox.cs` + (approved). The pool is capped at `ProcessorCount` and every worker is parked on a gate. The + assertion is that an undispatched message reaches the `InternalBus` within 3 s. + - **RED:** failed 3/3 with "did not dispatch the outstanding message within 00:00:03". With + starvation removed it passes in 90 ms. + - **Collection:** runs in `ThreadPoolStarvationCollection` (`DisableParallelization`). + - **Second defect, seen live while RED:** the starved tick fired *after* `StopAsync` and after + the provider had been disposed. `CreateScope()` (`TimedOutboxSweeper.cs:111`, outside the + `try`) threw `ObjectDisposedException`, the exception escaped `async void Sweep`, and it crashed + the test host. The test leaves the provider undisposed until the fix lands. + - **Refactored** onto the `Sweeping/TestDoubles/StarvedThreadPool` helper. It proves starvation + with a probe `Timer` that can't fire, and releases the pool if it fails. +2. `.../Sweeping/When_stopping_during_a_sweep_should_wait_for_the_sweep_to_finish.cs`. + - A `GatedDistributedLock` holds a sweep in flight. + - **RED (5/5):** "StopAsync returned while a sweep was still running". + - **Revised from the first idea, "don't sweep after stop", which was not a defect:** a timer firing + that has not started yet is cancelled by `Change(Infinite)`. The real defect is that `StopAsync` + does not wait for a sweep that is already running. +3. `.../Sweeping/When_a_sweep_throws_should_sweep_again_on_the_next_tick.cs`. + - **RED:** the exception escapes `async void Sweep` and **crashes the test host**. +4. `.../Sweeping/When_creating_the_sweep_scope_throws_should_release_the_lock_and_sweep_again.cs` + (uses `TestDoubles/FailsFirstScopeFactory`). + - **RED:** crashes the test host. The lock leak would keep it red once the crash is fixed. +5. `.../Sweeping/When_the_timer_interval_is_not_positive_should_reject_the_options.cs` (Theory: 0, -1). + - **RED:** no `ConfigurationException` is thrown. +6. `tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs` + (Kafka category, live broker). + - **Decision:** one worker drains confirms in order. + - **RED (3/3):** the most handlers running at once is **50 of 50** (expected 1). +7. `tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs` + (RMQ category, live broker). + - **RED (3/3):** the most handlers running at once is 15–23 (expected 1). +8. **Sweeper telemetry, per ADR 0081 (Accepted, commit `94527b8c2`).** All these files are in + `tests/Paramore.Brighter.Core.Tests/Sweeping/`: + - `When_a_sweep_completes_should_record_a_completed_sweep_and_its_duration.cs` + - `When_the_sweep_lock_is_held_should_record_a_lock_unavailable_sweep.cs` + - `When_a_sweep_throws_should_record_a_failed_sweep.cs` + - `When_a_sweep_starts_late_should_record_how_late_it_started.cs`. This uses a `FakeTimeProvider`: + the first lag is 0 s. Advancing 7 s past a sweep due at 5 s makes the second lag 2 s. + - `When_the_sweeper_meter_throws_should_still_sweep_the_outbox.cs` + - **Test doubles:** `RecordedSweeperMeasurements` (a `MeterListener` scoped to its own + `IMeterFactory`), `SweeperMeasurement`, `HeldDistributedLock`, `ThrowingSweeperMeter` and + `SweepableOutbox`. + - **RED:** compilation fails because `IAmABrighterSweeperMeter`, `SweepOutcome`, `SweeperMeter` + and the `meter:` and `timeProvider:` constructor parameters do not exist yet. + +## Fix + +Four commits on `bugfix/4560-timed-outbox-sweeper-starvation`: + +1. `94527b8c2` **docs:** ADR 0081, sweeper health metrics. +2. `84341b0bb` **refactor (structural only):** + - adds `IAmABrighterSweeperMeter`, `SweepOutcome` and `NullSweeperMeter`; + - adds the instrument names to `BrighterSemanticConventions`; + - adds optional `timeProvider` and `meter` parameters to the `TimedOutboxSweeper` constructor. +3. `85b219f14` **fix: `src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs`** + - **Sweeps run on a dedicated thread.** The thread waits with a real-time timeout, so a starved + pool cannot stop it, and with a `TimeProvider` timer, so a test clock can wake it. + - **The schedule keeps to its due times.** Lag on the next sweep shows any overrun. The sweeper + re-anchors only when a sweep starts a whole interval or more late, so it never sweeps back to + back. + - **`StopAsync` waits for an in-flight sweep.** + - **A failed sweep is caught and logged**, and `SweepOutcome.Failed` is recorded. + - **The lock is released on every path**, because `CreateScope` now runs inside the `try`. + - **`TimerInterval` below 1 throws `ConfigurationException`.** + - **New `SweeperMeter`**, registered in `AddBrighterInstrumentation`. + - Both tests that had kept their provider undisposed now dispose it again. +4. `185282398` **fix:** a new `src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs`, used by + `KafkaMessageProducer` and RMQ.Sync `RmqMessageProducer`. It replaces one `Task.Run` per + confirmation with a single in-order worker, and `Dispose` completes the queue. + +5. `0e84c1094` **fix:** `SerialCallbackQueue` drains on one dedicated background thread, created with the + first callback, instead of a worker started with `Task.Run`. Each callback runs in a + `BrighterAsyncContext`, and a confirmation that arrives after `Complete` is dropped instead of + throwing on the broker's thread. + - **Test:** `tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs`. + - **RED (3/3)** before the change, green afterwards. +6. **docs:** a release note in `release_notes.md` covering the behaviour changes and the cost of one + thread per confirming producer. + +**Targeted results:** + +| Tests | Result | +| --- | --- | +| Core.Tests `Sweeping` | 11/11 green, 6 runs on net9.0 and 2 on net10.0 | +| Full Core.Tests, net9.0 | 1636 passed, 7 skipped | +| Kafka confirm-ordering test | green 3/3 | +| RMQ.Sync confirm-ordering test | green 3/3 | +| Existing Kafka confirmation and dispose tests | 12/12 | +| Existing RMQ.Sync confirmation and dispose tests | 30/30 | + +## Verify + +Full suites run on 2026-10-08 against `ea92447c9` (all branch commits), on net9.0 and net10.0. The +Kafka suite ran against the local `kafka` container and RMQ.Sync against the shared RabbitMQ. + +| Suite | net9.0 | net10.0 | +| --- | --- | --- | +| Core.Tests | 1637 passed, 0 failed, 7 skipped | 1637 passed, 0 failed, 7 skipped | +| Kafka.Tests | 239 passed, 2 failed | 239 passed, 2 failed | +| RMQ.Sync.Tests | 192 passed, 10 failed, 1 skipped | 192 passed, 10 failed, 1 skipped | + +Every regression test listed above passed. The failures are the same on both frameworks, and all of +them come from the local environment rather than this change: + +- **Kafka (2):** `KafkaMessageProducerHeaderBytesSendTests` (Reactor and Proactor) fail with + `Connection refused (localhost:8081)`. They need a Confluent Schema Registry, and none is running. +- **RMQ.Sync, mTLS (9):** the `RmqMutualTls*` acceptance and observability tests fail with + `FileNotFoundException: Client certificate not found`. They need `tests/generate-test-certs.sh` and + a broker set up for TLS. +- **RMQ.Sync, `DispatchBuilderTests.When_Building_A_Dispatcher` (1):** fails with + `PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'mary'`. A durable `mary` queue + already exists on the shared broker, which belongs to another worktree, and the RMQ.Async + dispatcher tests use the same queue name. This branch does not touch `ChannelFactory` or the + dispatch tests. + +RMQ.Async.Tests was not run: RMQ.Async's `RmqMessageProducer` is unchanged. + +**Dropped from scope:** the `samples/` harness (see Scope Notes). diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/README.md b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/README.md new file mode 100644 index 0000000000..93d12029eb --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/README.md @@ -0,0 +1,49 @@ +# Diagnostics for #4560 + +These scratch console apps produced the evidence in `../bugfix.md`. Use them as the starting point for the +`samples/` harness. Both target net9.0 and reference the repo's projects by relative path. + +## Builds against a fixed commit + +Both apps were written against the code at `2890f8610`, before the fix. + +- **Both still build against the fixed code.** The fix left the `TimedOutboxSweeper` constructor + source compatible. +- **The results change.** Against the fixed code, `blocked` mode with `DOTNET_PROCESSOR_COUNT=1` + gives the real sweeper 10 ticks with a 1.01 s maximum gap. Before the fix it gave 7 ticks and a + 3.03 s gap. The `PeriodicTimer` async loop still slips (2.01 s). That makes the apps useful as + before/after demonstrations. +- **`TimerInterval` must be at least 1.** Values below 1 now throw `ConfigurationException`. + +## `starve/`: when does the sweeper's tick slip? + +`starve/` runs the real `TimedOutboxSweeper` with `TimerInterval = 1`. A recording `IDistributedLock` +times each tick. Three loads are compared: +- none; +- 200 blocked pool items; +- a flood of 20k `Task.Run` per second from a non-pool thread. + +Each load also runs a dedicated-thread loop and a `PeriodicTimer` async loop for comparison. + +```bash +DOTNET_PROCESSOR_COUNT=1 dotnet run -c Release -- blocked 10 # modes: none | blocked | flood +``` + +## `pileup/`: the full pipeline under load + +`pileup/` runs the real `CommandProcessor`, `OutboxProducerMediator`, `InMemoryOutbox` and +`TimedOutboxSweeper`. A fake Kafka-like producer delivers on a non-pool thread and calls `Task.Run` for +each confirm. + +```bash +# args: sync|async workers seconds +DOTNET_PROCESSOR_COUNT=2 dotnet run -c Release -- async 64 12 +``` + +| Variable | Effect | +| --- | --- | +| `DI_DEFAULT=1` | Outstanding-check interval of zero, as DI sets it | +| `NO_CHECK=1` | Effectively turns the outstanding check off (1 h interval) | + +- **Posters yield.** The posting loops `await Task.Yield()`, which gives fair load. Remove the yield + to saturate the pool. diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/Program.cs b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/Program.cs new file mode 100644 index 0000000000..013422ac58 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/Program.cs @@ -0,0 +1,165 @@ +using Paramore.Brighter.Extensions; +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Net.Mime; +using System.Text.Json; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Paramore.Brighter; +using Paramore.Brighter.JsonConverters; +using Paramore.Brighter.Logging; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; + +// args: sync|async workers seconds +var mode = args.Length > 0 ? args[0] : "async"; +var workers = args.Length > 1 ? int.Parse(args[1]) : 64; +var seconds = args.Length > 2 ? int.Parse(args[2]) : 15; + +var sw = Stopwatch.StartNew(); +var counter = new Counter(sw); +ApplicationLogging.LoggerFactory = LoggerFactory.Create(b => b.SetMinimumLevel(LogLevel.Debug).AddProvider(counter)); + +var routingKey = new RoutingKey("MyCommand"); +var producer = new FakeKafkaProducer(new Publication { Topic = routingKey, RequestType = typeof(MyCommand) }); +var mapperRegistry = new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => new MyCommandMessageMapper()), new SimpleMessageMapperFactoryAsync(_ => new MyCommandMessageMapper())); +mapperRegistry.Register(); mapperRegistry.RegisterAsync(); +var producerRegistry = new ProducerRegistry(new Dictionary { { routingKey, producer } }); +var resilience = new ResiliencePipelineRegistry().AddBrighterDefault(); +var outbox = new InMemoryOutbox(TimeProvider.System) { EntryLimit = 1_000_000 }; +var mediator = new OutboxProducerMediator( + producerRegistry, resilience, mapperRegistry, + new EmptyMessageTransformerFactory(), new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), new FindPublicationByPublicationTopicOrRequestType(), outbox, + maxOutStandingCheckInterval: Environment.GetEnvironmentVariable("NO_CHECK") == "1" ? TimeSpan.FromHours(1) : Environment.GetEnvironmentVariable("DI_DEFAULT") == "1" ? TimeSpan.Zero : null); +Console.WriteLine($"checkInterval={(Environment.GetEnvironmentVariable("DI_DEFAULT") == "1" ? "Zero (DI default)" : "1s (ctor default)")}"); +var cp = new CommandProcessor(new InMemoryRequestContextFactory(), new DefaultPolicy(), resilience, mediator, + requestSchedulerFactory: new InMemorySchedulerFactory()); + +var services = new ServiceCollection(); +services.AddSingleton(mediator); +var sp = services.BuildServiceProvider(); +var ticks = new ConcurrentQueue(); +var sweeper = new TimedOutboxSweeper(sp.GetRequiredService(), new RecordingLock(ticks, sw, new InMemoryLock()), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.FromSeconds(1) }); + +ThreadPool.GetMinThreads(out var minW, out _); +Console.WriteLine($"mode={mode} workers={workers} procs={Environment.ProcessorCount} minWorkers={minW}"); + +await sweeper.StartAsync(CancellationToken.None); + +var stop = new CancellationTokenSource(); +long posts = 0; int reported = 0; +var loops = Enumerable.Range(0, workers).Select(_ => Task.Run(async () => +{ + while (!stop.IsCancellationRequested) + { + try { + if (mode == "sync") cp.Post(new MyCommand()); + else await cp.PostAsync(new MyCommand()); + } catch (Exception e) { if (Interlocked.Exchange(ref reported, 1) == 0) Console.WriteLine("POST FAILED: " + e); throw; } + Interlocked.Increment(ref posts); + await Task.Yield(); + } +})).ToArray(); + +// sampler on dedicated thread +var samples = new List(); +var sampler = new Thread(() => +{ + for (int s = 1; s <= seconds; s++) + { + Thread.Sleep(1000); + var outstanding = outbox.Requests_Outstanding(); + samples.Add($"t={s,2}s posts={Interlocked.Read(ref posts),8} undispatched={outstanding,7} entries={outbox.EntryCount,8} pool={ThreadPool.ThreadCount,3} queue={ThreadPool.PendingWorkItemCount,8} checksQueued={counter.Queued,6} checksStarted={counter.Started,6} backlog={counter.Queued - counter.Started,6} sweepTicks={ticks.Count,3} sweeps={counter.Sweeps,3} resends={producer.Resends,6}"); + } +}) { IsBackground = true }; +sampler.Start(); +sampler.Join(); +stop.Cancel(); +foreach (var l in samples) Console.WriteLine(l); +var a = ticks.ToArray(); +var gaps = a.Zip(a.Skip(1), (x, y) => y - x).DefaultIfEmpty(double.NaN).Max(); +Console.WriteLine($"sweeper ticks={a.Length} maxGap={gaps:F2}s sweepsCompleted={counter.Sweeps} maxSweep={counter.MaxSweep:F2}s maxCheckBacklog={counter.MaxBacklog}"); +Environment.Exit(0); + +static class OutboxExt +{ + public static int Requests_Outstanding(this InMemoryOutbox o) => + o.GetOutstandingMessageCount(TimeSpan.Zero, null); +} + +class Counter(Stopwatch sw) : ILoggerProvider, ILogger +{ + public long Queued, Started, Sweeps; public long MaxBacklog; public double MaxSweep; + double _sweepStart; + public ILogger CreateLogger(string categoryName) => this; + public void Dispose() { } + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + var fmt = (state as IReadOnlyList>)?.LastOrDefault(kv => kv.Key == "{OriginalFormat}").Value as string ?? ""; + if (fmt.StartsWith("Running outstanding message check")) { var q = Interlocked.Increment(ref Queued); var b = q - Interlocked.Read(ref Started); if (b > MaxBacklog) MaxBacklog = b; } + else if (fmt.StartsWith("Begin count of outstanding")) Interlocked.Increment(ref Started); + else if (fmt.StartsWith("Outbox Sweeper looking")) _sweepStart = sw.Elapsed.TotalSeconds; + else if (fmt.StartsWith("Outbox Sweeper sleeping") && _sweepStart > 0) { Interlocked.Increment(ref Sweeps); var d = sw.Elapsed.TotalSeconds - _sweepStart; if (d > MaxSweep) MaxSweep = d; _sweepStart = 0; } + else if (logLevel >= LogLevel.Warning && !fmt.StartsWith("Outbox Sweeper is still running")) Console.WriteLine($"[{logLevel}] {formatter(state, exception)} {exception?.GetType().Name}"); + } +} + +class RecordingLock(ConcurrentQueue q, Stopwatch sw, IDistributedLock inner) : IDistributedLock +{ + public Task ObtainLockAsync(string r, CancellationToken ct) { q.Enqueue(sw.Elapsed.TotalSeconds); return inner.ObtainLockAsync(r, ct); } + public Task ReleaseLockAsync(string r, string id, CancellationToken ct) => inner.ReleaseLockAsync(r, id, ct); +} + +// Mimics KafkaMessageProducer: Produce enqueues to librdkafka; delivery reports arrive on a non-pool +// thread; Brighter raises each confirmation with Task.Run (KafkaMessageProducer.cs:456). +class FakeKafkaProducer : IAmAMessageProducerSync, IAmAMessageProducerAsync, ISupportPublishConfirmation, ISupportPublishConfirmationAsync +{ + private readonly BlockingCollection _inFlight = new(); + private readonly ConcurrentDictionary _seen = new(); + public long Resends; + private Func? _async; + public FakeKafkaProducer(Publication p) + { + Publication = p; + new Thread(() => + { + foreach (var id in _inFlight.GetConsumingEnumerable()) + { + var r = new PublishConfirmationResult(true, id, Publication.Topic!, null); + Task.Run(async () => { OnMessagePublished?.Invoke(r); if (_async != null) await _async(r); }); + } + }) { IsBackground = true, Name = "librdkafka-delivery" }.Start(); + } + public bool UseAsyncPublishConfirmation => true; + public Publication Publication { get; } + public Activity? Span { get; set; } + public IAmAMessageScheduler? Scheduler { get; set; } + public event Action? OnMessagePublished; + public event Func OnMessagePublishedAsync { add => _async += value; remove => _async -= value; } + public void Send(Message m) { if (!_seen.TryAdd(m.Id.Value, 0)) Interlocked.Increment(ref Resends); _inFlight.Add(m.Id); } + public void SendWithDelay(Message m, TimeSpan? d) => Send(m); + public Task SendAsync(Message m, CancellationToken ct = default) { Send(m); return Task.CompletedTask; } + public Task SendWithDelayAsync(Message m, TimeSpan? d, CancellationToken ct = default) => SendAsync(m, ct); + public void Dispose() { } + public ValueTask DisposeAsync() => ValueTask.CompletedTask; +} + +public class MyCommand() : Command(Id.Random()) { public string? Value { get; set; } = "hello"; } + +class MyCommandMessageMapper : IAmAMessageMapper, IAmAMessageMapperAsync +{ + public IRequestContext? Context { get; set; } + public Message MapToMessage(MyCommand request, Publication publication) => + new(new MessageHeader(request.Id, publication.Topic!, MessageType.MT_COMMAND, timeStamp: DateTimeOffset.UtcNow, + contentType: new ContentType(MediaTypeNames.Application.Json)), + new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options))); + public MyCommand MapToRequest(Message message) => throw new NotImplementedException(); + public Task MapToMessageAsync(MyCommand r, Publication p, CancellationToken ct = default) => Task.FromResult(MapToMessage(r, p)); + public Task MapToRequestAsync(Message m, CancellationToken ct = default) => throw new NotImplementedException(); +} diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/pileup.csproj b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/pileup.csproj new file mode 100644 index 0000000000..afcfc11be3 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/pileup/pileup.csproj @@ -0,0 +1,15 @@ + + + Exe + net9.0 + enable + enable + + + + + + + + + diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/Program.cs b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/Program.cs new file mode 100644 index 0000000000..3fb2cdd321 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/Program.cs @@ -0,0 +1,80 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using Paramore.Brighter; +using Paramore.Brighter.Outbox.Hosting; + +// args: mode(none|blocked|flood) seconds +var mode = args.Length > 0 ? args[0] : "none"; +var seconds = args.Length > 1 ? int.Parse(args[1]) : 10; +var sw = Stopwatch.StartNew(); +ThreadPool.GetMinThreads(out var minW, out _); +Console.WriteLine($"mode={mode} procs={Environment.ProcessorCount} minWorkers={minW}"); + +var timerTicks = new ConcurrentQueue(); +var threadTicks = new ConcurrentQueue(); +var asyncLoopTicks = new ConcurrentQueue(); + +var stop = new CancellationTokenSource(); +var gate = new ManualResetEventSlim(false); + +// load +Thread? flooder = null; +if (mode == "blocked") +{ + for (int i = 0; i < 200; i++) Task.Run(() => gate.Wait()); +} +else if (mode == "flood") +{ + // simulate Confluent's delivery thread: a non-pool thread doing Task.Run per delivery report + flooder = new Thread(() => + { + var perMs = 20; // 20k reports/sec + while (!stop.IsCancellationRequested) + { + for (int i = 0; i < perMs; i++) + Task.Run(() => { var s = Stopwatch.StartNew(); while (s.Elapsed.TotalMilliseconds < 0.2) { } }); + Thread.Sleep(1); + } + }) { IsBackground = true }; + flooder.Start(); +} + +// 1) the real TimedOutboxSweeper, ticks recorded by the lock +var sweeper = new TimedOutboxSweeper(null!, new RecordingLock(timerTicks, sw), new TimedOutboxSweeperOptions { TimerInterval = 1 }); +await sweeper.StartAsync(CancellationToken.None); + +// 2) control: dedicated thread +var t = new Thread(() => { while (!stop.IsCancellationRequested) { threadTicks.Enqueue(sw.Elapsed.TotalSeconds); Thread.Sleep(1000); } }) { IsBackground = true }; +t.Start(); + +// 3) BackgroundService-style async loop with PeriodicTimer on a LongRunning task +_ = Task.Factory.StartNew(async () => +{ + using var pt = new PeriodicTimer(TimeSpan.FromSeconds(1)); + asyncLoopTicks.Enqueue(sw.Elapsed.TotalSeconds); + while (await pt.WaitForNextTickAsync()) asyncLoopTicks.Enqueue(sw.Elapsed.TotalSeconds); +}, TaskCreationOptions.LongRunning); + +Thread.Sleep(seconds * 1000); +stop.Cancel(); +gate.Set(); +ThreadPool.GetAvailableThreads(out var avail, out _); +Console.WriteLine($"threadCount={ThreadPool.ThreadCount} pending={ThreadPool.PendingWorkItemCount}"); +Report("TimedOutboxSweeper", timerTicks, seconds); +Report("dedicated thread", threadTicks, seconds); +Report("PeriodicTimer async loop", asyncLoopTicks, seconds); +Environment.Exit(0); + +static void Report(string name, ConcurrentQueue q, int seconds) +{ + var a = q.Where(x => x <= seconds).ToArray(); + var first = a.Length > 0 ? a[0] : double.NaN; + var gaps = a.Zip(a.Skip(1), (x, y) => y - x).DefaultIfEmpty(double.NaN).Max(); + Console.WriteLine($" {name,-38} ticks={a.Length,3} first={first,6:F2}s maxGap={gaps,6:F2}s"); +} + +class RecordingLock(ConcurrentQueue q, Stopwatch sw) : IDistributedLock +{ + public Task ObtainLockAsync(string resource, CancellationToken ct) { q.Enqueue(sw.Elapsed.TotalSeconds); return Task.FromResult(null); } + public Task ReleaseLockAsync(string resource, string lockId, CancellationToken ct) => Task.CompletedTask; +} diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/starve.csproj b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/starve.csproj new file mode 100644 index 0000000000..2eda85ab11 --- /dev/null +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/diagnostic/starve/starve.csproj @@ -0,0 +1,11 @@ + + + Exe + net9.0 + enable + enable + + + + + From eece13a938b7aab3f021367b036896d559ba8af7 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 01:36:24 +0100 Subject: [PATCH 09/36] test: sweeper must sweep a batch when sends complete off a starved pool Review of #4562 (findings #1/#2): the existing starvation test uses InMemoryMessageProducer, which completes inline, so it only proves the tick fires. OffPoolCompletingProducer completes each send from a dedicated delivery thread through a RunContinuationsAsynchronously TCS and awaits it plainly, as KafkaMessagePublisher awaits Confluent's ProduceAsync (Confluent.Kafka 2.15.1 uses RunContinuationsAsynchronously). RED (3/3): "dispatched 1 of 5 outstanding messages within 00:00:03". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TestDoubles/OffPoolCompletingProducer.cs | 60 +++++++++++++ ...off_the_pool_should_still_sweep_a_batch.cs | 86 +++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/OffPoolCompletingProducer.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/OffPoolCompletingProducer.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/OffPoolCompletingProducer.cs new file mode 100644 index 0000000000..5a89d05dd2 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/OffPoolCompletingProducer.cs @@ -0,0 +1,60 @@ +using System; +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An async producer that completes each send the way Confluent's ProduceAsync does: a dedicated +/// delivery thread, not a pool thread, writes the message to the bus and then completes a +/// created with . +/// So the continuation after the producer's own await is never run inline by the delivery thread. +/// +public sealed class OffPoolCompletingProducer : IAmAMessageProducerAsync +{ + private readonly InternalBus _bus; + private readonly BlockingCollection<(Message Message, TaskCompletionSource Delivered)> _deliveries = new(); + private readonly Thread _deliveryThread; + + public OffPoolCompletingProducer(InternalBus bus, Publication publication) + { + _bus = bus; + Publication = publication; + _deliveryThread = new Thread(Deliver) { IsBackground = true, Name = "Off-pool delivery" }; + _deliveryThread.Start(); + } + + public Publication Publication { get; } + public Activity? Span { get; set; } + public IAmAMessageScheduler? Scheduler { get; set; } + + // Awaits the delivery inside the producer, as KafkaMessagePublisher awaits ProduceAsync: a plain await, so + // its continuation follows the caller's context. The mediator's own awaits use ConfigureAwait(false). + public async Task SendAsync(Message message, CancellationToken cancellationToken = default) + { + var delivered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _deliveries.Add((message, delivered), cancellationToken); + await delivered.Task; + } + + public Task SendWithDelayAsync(Message message, TimeSpan? delay, CancellationToken cancellationToken = default) + => SendAsync(message, cancellationToken); + + public ValueTask DisposeAsync() + { + _deliveries.CompleteAdding(); + _deliveryThread.Join(TimeSpan.FromSeconds(5)); + return ValueTask.CompletedTask; + } + + private void Deliver() + { + foreach (var (message, delivered) in _deliveries.GetConsumingEnumerable()) + { + _bus.Enqueue(message); + delivered.SetResult(); + } + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch.cs new file mode 100644 index 0000000000..fac2acda0b --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch.cs @@ -0,0 +1,86 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Transactions; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Polly.Registry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +[Collection(ThreadPoolStarvationCollection.Name)] +public class TimedOutboxSweeperAsyncSendStarvationTests +{ + private const int OutstandingMessages = 5; + private static readonly TimeSpan s_waitForSweep = TimeSpan.FromSeconds(3); + + [Fact] + public void When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch() + { + //Arrange + var routingKey = new RoutingKey("MyCommand"); + var bus = new InternalBus(); + var outbox = new InMemoryOutbox(TimeProvider.System); + var producer = new OffPoolCompletingProducer(bus, new Publication { Topic = routingKey }); + var mediator = CreateMediator(producer, routingKey, outbox); + + // Several messages, so the sweep must resume after each awaited send to reach the next one + for (var i = 0; i < OutstandingMessages; i++) + { + outbox.Add(new Message( + new MessageHeader(Id.Random(), routingKey, MessageType.MT_COMMAND), + new MessageBody("awaiting a publish confirmation that never came")), new RequestContext()); + } + + var services = new ServiceCollection(); + services.AddSingleton(mediator); + using var provider = services.BuildServiceProvider(); + + var sweeper = new TimedOutboxSweeper( + provider.GetRequiredService(), + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + var starvedPool = new StarvedThreadPool(); + try + { + //Act + sweeper.StartAsync(CancellationToken.None).GetAwaiter().GetResult(); + var swept = SpinWait.SpinUntil(() => bus.Stream(routingKey).Count() == OutstandingMessages, s_waitForSweep); + + //Assert + Assert.True(swept, $"The sweeper dispatched {bus.Stream(routingKey).Count()} of {OutstandingMessages} outstanding messages within {s_waitForSweep} while every thread-pool worker was blocked"); + } + finally + { + starvedPool.Dispose(); + sweeper.StopAsync(CancellationToken.None).GetAwaiter().GetResult(); + sweeper.Dispose(); + producer.DisposeAsync().AsTask().GetAwaiter().GetResult(); + } + } + + private static OutboxProducerMediator CreateMediator( + IAmAMessageProducer producer, RoutingKey routingKey, InMemoryOutbox outbox) + { + var producerRegistry = new ProducerRegistry(new Dictionary + { + { routingKey, producer } + }); + + return new OutboxProducerMediator( + producerRegistry, + new ResiliencePipelineRegistry().AddBrighterDefault(), + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + outbox); + } +} From 80512a4967d274ed4d7b33982ba00b6d1a04e2e1 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 01:36:24 +0100 Subject: [PATCH 10/36] fix: keep a sweep on the sweeper thread across awaited sends Root cause: SweepOnce blocked on SweepAsync with no context, so the continuation after each awaited Kafka send needed a pool thread; under starvation a sweep dispatched one message and stalled. Running the sweep in a BrighterAsyncContext was not enough on its own: a ConfigureAwait(false) continuation is not inlined on a thread with a SynchronizationContext, so two hops on the sweep path also sent it back to the pool. - TimedOutboxSweeper runs the sweep with BrighterAsyncContext.Run. - BackgroundDispatchUsingAsync (only caller: OutboxSweeper) passes continueOnCapturedContext: true to DispatchAsync/BulkDispatchAsync. - ExecuteWithResiliencePipelineAsync passes the caller's continueOnCapturedContext to Polly through a pooled ResilienceContext; that overload previously always used Polly's default of false. Still a known limit: a producer or outbox that awaits with ConfigureAwait(false) internally resumes on the pool. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TimedOutboxSweeper.cs | 6 ++++-- .../OutboxProducerMediator.cs | 21 +++++++++++++++---- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index 6f26c4a519..ee49c867ac 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -32,6 +32,7 @@ THE SOFTWARE. */ using Microsoft.Extensions.Logging; using Paramore.Brighter.Logging; using Paramore.Brighter.Observability; +using Paramore.Brighter.Tasks; using ILogger = Microsoft.Extensions.Logging.ILogger; namespace Paramore.Brighter.Outbox.Hosting @@ -217,8 +218,9 @@ private SweepOutcome SweepOnce() { try { - // On the sweeper's own thread, so blocking on the sweep is safe: there is no synchronization context to deadlock on. - return SweepAsync().GetAwaiter().GetResult(); + // Runs the sweep in a context on the sweeper's own thread, so its awaits resume here rather than on the pool: + // a producer such as Kafka completes a send from its own thread with asynchronous continuations. + return BrighterAsyncContext.Run(SweepAsync); } catch (Exception e) { diff --git a/src/Paramore.Brighter/OutboxProducerMediator.cs b/src/Paramore.Brighter/OutboxProducerMediator.cs index 82d639a077..7c2d6da036 100644 --- a/src/Paramore.Brighter/OutboxProducerMediator.cs +++ b/src/Paramore.Brighter/OutboxProducerMediator.cs @@ -33,6 +33,7 @@ THE SOFTWARE. */ using Paramore.Brighter.Logging; using Paramore.Brighter.Observability; using Paramore.Brighter.Scheduler.Events; +using Polly; using Polly.Registry; // ReSharper disable StaticMemberInGenericType @@ -744,13 +745,15 @@ CancellationToken cancellationToken Log.FoundMessagesToClear(s_logger, messages.Length, amountToClear); + // Continue on the caller's context: the TimedOutboxSweeper runs this in a context on its own thread, so + // the sweep stays off a starved thread pool. A caller without a context is unaffected. if (useBulk) { - await BulkDispatchAsync(messages, requestContext, false, cancellationToken); + await BulkDispatchAsync(messages, requestContext, true, cancellationToken); } else { - await DispatchAsync(messages, requestContext, false, cancellationToken); + await DispatchAsync(messages, requestContext, true, cancellationToken); } Log.MessagesHaveBeenCleared(s_logger); @@ -1420,8 +1423,18 @@ await resiliencePipeline } else { - await resiliencePipeline.ExecuteAsync(async ct => await send(ct), cancellationToken) - .ConfigureAwait(continueOnCapturedContext); + // Polly awaits with its context's ContinueOnCapturedContext, so pass the caller's choice through. + var resilienceContext = ResilienceContextPool.Shared.Get(continueOnCapturedContext, cancellationToken); + try + { + await resiliencePipeline + .ExecuteAsync(async context => await send(context.CancellationToken), resilienceContext) + .ConfigureAwait(continueOnCapturedContext); + } + finally + { + ResilienceContextPool.Shared.Return(resilienceContext); + } } return true; From e493fefc8bc5f434ed141e9071e95fe31a6401d2 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 01:36:46 +0100 Subject: [PATCH 11/36] docs: amend ADR 0081 for a batched confirmation drain and queue depth Review of #4562 found that raising confirmations one at a time caps a producer with a database outbox at one mark-dispatched round trip at a time. The queue becomes BatchedCallbackQueue: one drain thread per producer, batches of up to 32 started in order inside one BrighterAsyncContext. A new ObservableUpDownCounter, paramore.brighter.publish_confirmation.queue.depth, reads live queues from an internal registry. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- ...081-timed-outbox-sweeper-health-metrics.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md index 3dc5152a4f..044992124a 100644 --- a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md +++ b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md @@ -21,6 +21,9 @@ Date: 2026-10-08 Accepted +Amended 2026-10-09: publish confirmations drain in bounded batches on one thread per producer, and a +queue-depth instrument makes a confirmation backlog visible (see Decision § Amendment 2026-10-09). + ## Context `TimedOutboxSweeper` re-sends outbox messages that were never confirmed as dispatched. When the @@ -235,6 +238,97 @@ can only happen with a second caller of `ClearOutstandingFromOutboxAsync`. suggested alerts, and turning on the .NET `System.Runtime` meter for thread-pool queue length and thread count. +### Amendment 2026-10-09: batched confirmation drain and queue depth + +Review of #4562 found a cost in the #4560 confirmation fix. The fix replaced one `Task.Run` per +confirmation with a `SerialCallbackQueue`, which raises each producer's confirmations one at a time, +in order, on a dedicated thread. With a database outbox each confirmation is a round trip, so +running them one at a time caps how fast a producer can mark messages dispatched. At 2 ms a round +trip that is about 500 messages per second. Above that rate the unbounded queue grows, messages +pass `MinimumMessageAge` while still unmarked, and the sweeper sends them again. Nothing needs the +ordering: marking messages dispatched commutes. Ordering came along with the fix that stopped the +`Task.Run` flood, and was never a requirement. + +#### Batched drain + +The queue becomes `BatchedCallbackQueue` (`Paramore.Brighter.Tasks`). It keeps the parts of the +#4560 fix that matter: one dedicated thread per producer, created with the first callback, and +never the thread pool. + +- **The drain thread takes a batch.** It waits for one callback, then takes whatever else is already + queued, up to `maxBatchSize` (default **32**) in total. +- **It starts every callback in the batch, in queue order,** inside one `BrighterAsyncContext.Run`, + and waits for all of them before taking the next batch. So their I/O overlaps, while their + continuations still come back to the drain thread. +- **The batch size bounds concurrency.** A batch of at most 32 stays below the default connection + pool size of SqlClient and Npgsql (100), which leaves room for the application. The size is a + constructor parameter, not a public option: no producer exposes it in this amendment. +- **Callbacks still run in a `BrighterAsyncContext`.** Only a continuation that opts out with + `ConfigureAwait(false)` can land on the pool. This is unchanged. +- **Unchanged:** `TryWait`, `Complete`, dropping a confirmation added after `Complete`, and + swallowing a callback's escaped exception. +- **Changed:** callbacks may now complete out of order, and up to `maxBatchSize` may be in progress + at once. + +#### Queue depth + +| Instrument | Type | Unit | Attributes | +| --- | --- | --- | --- | +| `paramore.brighter.publish_confirmation.queue.depth` | `ObservableUpDownCounter` | `{confirmation}` | `messaging.system`, `messaging.destination.name` | + +- **What it counts:** for each live queue, the confirmations queued or running. +- **Its attributes:** they come from the producer that owns the queue. Kafka passes `kafka`, RMQ.Sync + passes `rabbitmq`, and both pass the publication's topic. +- **Why an up-down counter:** OpenTelemetry recommends an up-down counter for a queue size, because + depths across topics add up meaningfully. +- **How it reaches the queues.** Producers are built by registry factories outside DI, so a meter + cannot be injected into them. Instead: + 1. Each queue registers itself, with its attributes, in an internal static + `BatchedCallbackQueueRegistry` that holds weak references. + 2. It unregisters once it has completed and drained. + 3. A new `PublishConfirmationMeter` creates the observable instrument through `IMeterFactory` + and reads the registry when the instrument is collected. +- **How the meter is created:** `AddBrighterInstrumentation` registers it, and creates it when the + meter provider is built (`ConfigureBuilder`), because nothing else depends on it. +- **Without `AddBrighterInstrumentation`:** nothing observes the registry. The cost is one + registration per producer. + +#### Where each type is touched (amendment) + +| Assembly | Type | Change | +| --- | --- | --- | +| Paramore.Brighter | `SerialCallbackQueue` | Renamed `BatchedCallbackQueue`; batched drain; new constructor parameters for its attributes and `maxBatchSize` | +| Paramore.Brighter | `BatchedCallbackQueueRegistry` | New, internal | +| Paramore.Brighter | `PublishConfirmationMeter` | New | +| Paramore.Brighter | `BrighterSemanticConventions` | New constant for the instrument name | +| Paramore.Brighter.MessagingGateway.Kafka | `KafkaMessageProducer` | Passes `kafka` and its topic to the queue | +| Paramore.Brighter.MessagingGateway.RMQ.Sync | `RmqMessageProducer` | Passes `rabbitmq` and its topic to the queue | +| Paramore.Brighter.Extensions.Diagnostics | `BrighterMetricsBuilderExtensions` | Registers and creates `PublishConfirmationMeter` | + +#### Alternatives considered (amendment) + +- **Keep the drain serial and document why.** This was rejected, because the cap on database round + trips described above is a regression for high-rate producers. +- **Several drain threads (2–4).** This gives more threads for less overlap than a batch on one + thread, and the threads per producer would multiply on small pods. Rejected. +- **An unbounded batch.** A large backlog would open that many database connections at once. + Rejected. +- **A static `Meter` inside the queue.** Simpler, but it bypasses `IMeterFactory`, which every other + Brighter meter uses and which keeps metric tests isolated. Rejected. +- **Pass a meter role to each producer.** This needs a new parameter on every producer factory and + registry factory. That is a public API change, for a value the registry provides without it. + Rejected. + +#### Consequences (amendment) + +- **Positive:** confirmation throughput with a database outbox is bounded by `maxBatchSize` + overlapping round trips, not one. A backlog is visible before it causes resends. +- **Negative:** the in-order guarantee from #4560 goes away. Nothing documented it, and nothing in + Brighter depends on it. The approved Kafka and RMQ.Sync tests that assert "one at a time, in + order" are replaced by tests that assert confirmations overlap, never exceed the batch size, and + run on the drain thread rather than the pool. The registry is process-wide static state, so tests + that read it must filter by their own topic. + ## Consequences ### Positive From 10287babe33306a21421290774bbafabdccee720 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 01:37:30 +0100 Subject: [PATCH 12/36] refactor: rename SerialCallbackQueue to BatchedCallbackQueue Structural only, ahead of the batched drain in the ADR 0081 amendment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../KafkaMessageProducer.cs | 2 +- .../RmqMessageProducer.cs | 2 +- .../Tasks/{SerialCallbackQueue.cs => BatchedCallbackQueue.cs} | 2 +- ...hread_pool_is_starved_should_still_run_queued_callbacks.cs | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) rename src/Paramore.Brighter/Tasks/{SerialCallbackQueue.cs => BatchedCallbackQueue.cs} (99%) diff --git a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs index c3fe8545b9..71b113ea2c 100644 --- a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs @@ -81,7 +81,7 @@ event Func ISupportPublishConfirmationAsync.OnM // poll thread, and never one thread-pool item per report); the queue lets Dispose wait for those // callbacks — including the awaited Outbox mark-dispatched — after Flush() has drained the delivery // reports themselves. - private readonly SerialCallbackQueue _confirmationCallbacks = new(); + private readonly BatchedCallbackQueue _confirmationCallbacks = new(); public KafkaMessageProducer( KafkaMessagingGatewayConfiguration configuration, diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index fb4cec74f6..efdeae65df 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -63,7 +63,7 @@ public partial class RmqMessageProducer : RmqMessageGateway, IAmAMessageProducer // subscriber, so awaited callbacks run one at a time, in ack order, on the queue's single worker // (never one thread-pool item per ack). The queue lets Dispose wait for them — including the // awaited Outbox mark-dispatched — after WaitForConfirms has drained the broker acks themselves. - private readonly SerialCallbackQueue _confirmationCallbacks = new(); + private readonly BatchedCallbackQueue _confirmationCallbacks = new(); /// /// Action taken when a message is published, following receipt of a confirmation from the broker diff --git a/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs similarity index 99% rename from src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs rename to src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs index d5d538ac18..bbeea6148a 100644 --- a/src/Paramore.Brighter/Tasks/SerialCallbackQueue.cs +++ b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs @@ -42,7 +42,7 @@ namespace Paramore.Brighter.Tasks /// keeps draining. lets a producer's dispose wait for the queued callbacks to finish. /// The thread starts with the first callback and ends after once the queue is empty. /// - public sealed class SerialCallbackQueue + public sealed class BatchedCallbackQueue { private readonly BlockingCollection> _callbacks = new(); private readonly InFlightCallbackTracker _inFlight = new(); diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs index 927167b409..cb487a1e73 100644 --- a/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/When_the_thread_pool_is_starved_should_still_run_queued_callbacks.cs @@ -9,13 +9,13 @@ namespace Paramore.Brighter.Core.Tests.Tasks; [Collection(ThreadPoolStarvationCollection.Name)] -public class SerialCallbackQueueStarvationTests +public class BatchedCallbackQueueStarvationTests { [Fact] public void When_the_thread_pool_is_starved_should_still_run_queued_callbacks() { //Arrange - var queue = new SerialCallbackQueue(); + var queue = new BatchedCallbackQueue(); using var confirmed = new ManualResetEventSlim(false); var starvedPool = new StarvedThreadPool(); From 5bd972f69fe5c28c17aaa5e9c383106d9beb8870 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 08:16:11 +0100 Subject: [PATCH 13/36] test: batched callback queue runs up to 32 callbacks at once ADR 0081 amendment. RED (3/3): "Expected 32 callbacks in progress at once, but at most 1 were". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- ...e_batch_size_should_run_a_batch_at_once.cs | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once.cs new file mode 100644 index 0000000000..b39a40263f --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once.cs @@ -0,0 +1,77 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Tasks; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Tasks; + +public class BatchedCallbackQueueBatchingTests +{ + private const int BatchSize = 32; // the queue's default + private const int QueuedCallbacks = 40; // more than one batch + private static readonly TimeSpan s_wait = TimeSpan.FromSeconds(5); + + [Fact] + public void When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once() + { + //Arrange + var queue = new BatchedCallbackQueue(); + var inFlight = 0; + var mostAtOnce = 0; + var roundTrip = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + using var allDone = new CountdownEvent(QueuedCallbacks); + + // Park the drain thread, so every callback below is queued before it takes its next batch + using var drainParked = new ManualResetEventSlim(false); + using var releaseDrain = new ManualResetEventSlim(false); + queue.Enqueue(() => + { + drainParked.Set(); + releaseDrain.Wait(); + return Task.CompletedTask; + }); + Assert.True(drainParked.Wait(s_wait), "The drain thread did not start"); + + try + { + //Act + for (var i = 0; i < QueuedCallbacks; i++) + { + queue.Enqueue(async () => + { + InterlockedMax(ref mostAtOnce, Interlocked.Increment(ref inFlight)); + await roundTrip.Task; // a confirmation awaiting its outbox round trip + Interlocked.Decrement(ref inFlight); + allDone.Signal(); + }); + } + releaseDrain.Set(); + + var batchRunning = SpinWait.SpinUntil(() => Volatile.Read(ref inFlight) == BatchSize, s_wait); + Thread.Sleep(100); // give the drain the chance to start more than one batch at once + var atOnceBeforeRelease = Volatile.Read(ref mostAtOnce); + roundTrip.SetResult(); + var allRan = allDone.Wait(s_wait); + + //Assert + Assert.True(batchRunning, $"Expected {BatchSize} callbacks in progress at once, but at most {Volatile.Read(ref mostAtOnce)} were"); + Assert.Equal(BatchSize, atOnceBeforeRelease); + Assert.True(allRan, "Not every queued callback ran"); + Assert.Equal(BatchSize, mostAtOnce); + } + finally + { + releaseDrain.Set(); + roundTrip.TrySetResult(); + queue.Complete(); + } + } + + private static void InterlockedMax(ref int target, int value) + { + int current; + while ((current = Volatile.Read(ref target)) < value + && Interlocked.CompareExchange(ref target, value, current) != current) { } + } +} From cf8da1438b84a1f4df6de6846e5c83d91969966f Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 08:16:11 +0100 Subject: [PATCH 14/36] fix: drain publish confirmations in batches of up to 32 One callback at a time capped a producer with a database outbox at one mark-dispatched round trip at a time. The drain thread now takes up to 32 queued callbacks, starts them in order inside one BrighterAsyncContext and waits for all of them, so their I/O overlaps while continuations still come back to the drain thread. Callbacks may now finish out of order. Per the ADR 0081 amendment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../Tasks/BatchedCallbackQueue.cs | 58 +++++++++++++------ 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs index bbeea6148a..f334cfec16 100644 --- a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs +++ b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs @@ -25,31 +25,37 @@ THE SOFTWARE. */ using System; using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; using System.Threading; using System.Threading.Tasks; namespace Paramore.Brighter.Tasks { /// - /// Runs callbacks one at a time, in the order they were queued, on a single dedicated thread. A producer uses - /// it to raise publish confirmations off the broker's thread without queuing any work on the thread pool, so a - /// busy producer does not add to the pool's queue and confirmations keep flowing when the pool is starved (#4560). + /// Runs callbacks in batches on a single dedicated thread. A producer uses it to raise publish confirmations off + /// the broker's thread without queuing any work on the thread pool, so a busy producer does not add to the pool's + /// queue and confirmations keep flowing when the pool is starved (#4560). /// /// - /// Each callback runs inside a , so its continuations come back to the - /// dedicated thread; only a continuation that opts out of the context (ConfigureAwait(false)) can land + /// The thread takes up to 32 queued callbacks at a time, starts them in the order they were queued, and waits for + /// all of them before taking more; so their I/O, such as marking a message dispatched in an outbox, overlaps, + /// but callbacks may finish in any order. Each batch runs inside a , so + /// continuations come back to the dedicated thread; only a continuation that opts out of the context (ConfigureAwait(false)) can land /// on the pool. A callback must handle its own exceptions; one that escapes is swallowed so that the thread /// keeps draining. lets a producer's dispose wait for the queued callbacks to finish. /// The thread starts with the first callback and ends after once the queue is empty. /// public sealed class BatchedCallbackQueue { + private const int MaxBatchSize = 32; + private readonly BlockingCollection> _callbacks = new(); private readonly InFlightCallbackTracker _inFlight = new(); private int _threadStarted; /// - /// Queues a callback to run after every callback queued before it. + /// Queues a callback; it starts after every callback queued before it has started. /// /// The callback; it must not throw. public void Enqueue(Func callback) @@ -94,20 +100,34 @@ private bool TryAdd(Func callback) private void Drain() { - foreach (var callback in _callbacks.GetConsumingEnumerable()) + foreach (var first in _callbacks.GetConsumingEnumerable()) { - try - { - BrighterAsyncContext.Run(callback); - } - catch - { - // The callback owns its fault handling; swallow anything it let escape so later callbacks still run. - } - finally - { - _inFlight.End(); - } + var batch = TakeBatch(first); + BrighterAsyncContext.Run(() => Task.WhenAll(batch.Select(RunAndRelease))); + } + } + + private List> TakeBatch(Func first) + { + var batch = new List> { first }; + while (batch.Count < MaxBatchSize && _callbacks.TryTake(out var next)) + batch.Add(next); + return batch; + } + + private async Task RunAndRelease(Func callback) + { + try + { + await callback(); + } + catch + { + // The callback owns its fault handling; swallow anything it let escape so the rest still run. + } + finally + { + _inFlight.End(); } } } From 08a8ff64c8ccda28fc19696d42908c01de4a4481 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 08:17:59 +0100 Subject: [PATCH 15/36] fix: re-check the sweep's due time after creating its timer If the clock passed the due time between computing the wait and creating the timer, the timer was set from the later time and the sweep waited a whole extra interval. A test clock hits this (the tick-lag test failed 2/8 runs under load); the system clock only drifts by microseconds. Green 10/10 afterwards. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index ee49c867ac..face314af4 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -184,6 +184,10 @@ private bool WaitUntil(DateTimeOffset due) return true; using var timer = _timeProvider.CreateTimer(_ => SignalDue(), null, wait, Timeout.InfiniteTimeSpan); + // The clock may have moved on before the timer existed to see it, so look again before waiting. + if (due - _timeProvider.GetUtcNow() <= TimeSpan.Zero) + return true; + WaitHandle.WaitAny(new[] { _due, _stopping.Token.WaitHandle }, wait); } From 0ff90cfa6ef23d97146ede1a3de49b3c1ff5da3b Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:10:37 +0100 Subject: [PATCH 16/36] test: confirmations overlap in batches off the thread pool Replaces the "one at a time, in order" Kafka and RMQ.Sync tests, per the ADR 0081 amendment. They now assert every message is confirmed, 2-32 handlers run at once, and no handler runs on a thread-pool thread, before or after its await. Kafka: green 3/3; mutation-checked (serial drain -> "Actual: 1"; no BrighterAsyncContext -> "ran on a thread-pool thread"). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- ..._overlapping_batches_off_the_thread_pool.cs} | 17 +++++++++-------- ..._overlapping_batches_off_the_thread_pool.cs} | 17 +++++++++-------- 2 files changed, 18 insertions(+), 16 deletions(-) rename tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/{When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs => When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs} (82%) rename tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/{When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs => When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs} (81%) diff --git a/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs b/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs similarity index 82% rename from tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs rename to tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs index 155f1dd091..6da7f619dc 100644 --- a/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs +++ b/tests/Paramore.Brighter.Kafka.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs @@ -1,5 +1,4 @@ using System; -using System.Collections.Concurrent; using System.Linq; using System.Threading; using System.Threading.Tasks; @@ -10,13 +9,14 @@ namespace Paramore.Brighter.Kafka.Tests.MessagingGateway.Reactor; [Trait("Category", "Kafka")] [Collection("Kafka")] -public class KafkaConfirmationOrderingTests : IDisposable +public class KafkaConfirmationBatchingTests : IDisposable { private const int MessageCount = 50; + private const int MaxBatchSize = 32; private readonly string _topic = Guid.NewGuid().ToString(); private readonly IAmAProducerRegistry _producerRegistry; - public KafkaConfirmationOrderingTests() + public KafkaConfirmationBatchingTests() { _producerRegistry = new KafkaProducerRegistryFactory( new KafkaMessagingGatewayConfiguration @@ -37,7 +37,7 @@ public KafkaConfirmationOrderingTests() } [Fact] - public async Task When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order() + public async Task When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool() { //Arrange await Task.Delay(500); //Let the topic propagate in the broker @@ -48,14 +48,15 @@ public async Task When_confirming_many_messages_should_raise_confirmations_one_a var inFlight = 0; var mostAtOnce = 0; - var confirmedInOrder = new ConcurrentQueue(); + var ranOnPool = false; using var allConfirmed = new CountdownEvent(MessageCount); confirmations.OnMessagePublishedAsync += async result => { var now = Interlocked.Increment(ref inFlight); InterlockedMax(ref mostAtOnce, now); + if (Thread.CurrentThread.IsThreadPoolThread) ranOnPool = true; await Task.Delay(10); // a handler that takes a little time, like marking an outbox row dispatched - confirmedInOrder.Enqueue(result.MessageId); + if (Thread.CurrentThread.IsThreadPoolThread) ranOnPool = true; Interlocked.Decrement(ref inFlight); allConfirmed.Signal(); }; @@ -74,8 +75,8 @@ public async Task When_confirming_many_messages_should_raise_confirmations_one_a //Assert Assert.True(completed, "Not every message was confirmed"); - Assert.Equal(1, mostAtOnce); - Assert.Equal(sent.Select(m => m.Id), confirmedInOrder); + Assert.InRange(mostAtOnce, 2, MaxBatchSize); // confirmations overlap, but never more than one batch + Assert.False(ranOnPool, "A confirmation handler ran on a thread-pool thread"); } private static void InterlockedMax(ref int target, int value) diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs similarity index 81% rename from tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs rename to tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs index b734ac568e..b87ffaea09 100644 --- a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order.cs +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs @@ -1,5 +1,4 @@ using System; -using System.Collections.Concurrent; using System.Linq; using System.Threading; using System.Threading.Tasks; @@ -10,13 +9,14 @@ namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; [Trait("Category", "RMQ")] [Collection("RMQ")] -public class RmqConfirmationOrderingTests : IDisposable +public class RmqConfirmationBatchingTests : IDisposable { private const int MessageCount = 50; + private const int MaxBatchSize = 32; private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); private readonly RmqMessageProducer _messageProducer; - public RmqConfirmationOrderingTests() + public RmqConfirmationBatchingTests() { var rmqConnection = new RmqMessagingGatewayConnection { @@ -32,19 +32,20 @@ public RmqConfirmationOrderingTests() } [Fact] - public void When_confirming_many_messages_should_raise_confirmations_one_at_a_time_in_order() + public void When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool() { //Arrange var inFlight = 0; var mostAtOnce = 0; - var confirmedInOrder = new ConcurrentQueue(); + var ranOnPool = false; using var allConfirmed = new CountdownEvent(MessageCount); ((ISupportPublishConfirmationAsync)_messageProducer).OnMessagePublishedAsync += async result => { var now = Interlocked.Increment(ref inFlight); InterlockedMax(ref mostAtOnce, now); + if (Thread.CurrentThread.IsThreadPoolThread) ranOnPool = true; await Task.Delay(10); // a handler that takes a little time, like marking an outbox row dispatched - confirmedInOrder.Enqueue(result.MessageId); + if (Thread.CurrentThread.IsThreadPoolThread) ranOnPool = true; Interlocked.Decrement(ref inFlight); allConfirmed.Signal(); }; @@ -63,8 +64,8 @@ public void When_confirming_many_messages_should_raise_confirmations_one_at_a_ti //Assert Assert.True(completed, "Not every message was confirmed"); - Assert.Equal(1, mostAtOnce); - Assert.Equal(sent.Select(m => m.Id), confirmedInOrder); + Assert.InRange(mostAtOnce, 2, MaxBatchSize); // confirmations overlap, but never more than one batch + Assert.False(ranOnPool, "A confirmation handler ran on a thread-pool thread"); } private static void InterlockedMax(ref int target, int value) From 398db5c2a2fbe06cd6b4967546b93b64375a1180 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:10:37 +0100 Subject: [PATCH 17/36] test: report the publish-confirmation queue depth ADR 0081 amendment. RED: PublishConfirmationMeter and the (MessagingSystem, RoutingKey) queue constructor do not exist. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../Tasks/TestDoubles/ObservedQueueDepths.cs | 63 +++++++++++++++++++ .../Tasks/TestDoubles/QueueDepth.cs | 4 ++ ...e_waiting_should_report_the_queue_depth.cs | 55 ++++++++++++++++ 3 files changed, 122 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/ObservedQueueDepths.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/QueueDepth.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/When_confirmations_are_waiting_should_report_the_queue_depth.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/ObservedQueueDepths.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/ObservedQueueDepths.cs new file mode 100644 index 0000000000..c75546e6b6 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/ObservedQueueDepths.cs @@ -0,0 +1,63 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; + +namespace Paramore.Brighter.Core.Tests.Tasks.TestDoubles; + +/// +/// Owns an and, on , collects the confirmation queue depth from +/// meters that factory created, so a test sees only its own meter's instrument. +/// +public sealed class ObservedQueueDepths : IDisposable +{ + public const string Instrument = "paramore.brighter.publish_confirmation.queue.depth"; + + private readonly ServiceProvider _provider; + private readonly MeterListener _listener = new(); + private readonly ConcurrentQueue _depths = new(); + + public ObservedQueueDepths() + { + _provider = new ServiceCollection().AddMetrics().BuildServiceProvider(); + MeterFactory = _provider.GetRequiredService(); + + _listener.InstrumentPublished = (instrument, listener) => + { + if (ReferenceEquals(instrument.Meter.Scope, MeterFactory) && instrument.Name == Instrument) + listener.EnableMeasurementEvents(instrument); + }; + _listener.SetMeasurementEventCallback((_, value, tags, _) => Record(value, tags)); + _listener.Start(); + } + + public IMeterFactory MeterFactory { get; } + + /// Collects the instrument now and returns what it reported. + public IReadOnlyList Observe() + { + _depths.Clear(); + _listener.RecordObservableInstruments(); + return _depths.ToList(); + } + + public void Dispose() + { + _listener.Dispose(); + _provider.Dispose(); + } + + private void Record(long value, ReadOnlySpan> tags) + { + string? system = null, destination = null; + foreach (var tag in tags) + { + if (tag.Key == "messaging.system") system = tag.Value as string; + if (tag.Key == "messaging.destination.name") destination = tag.Value as string; + } + + _depths.Enqueue(new QueueDepth(system, destination, value)); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/QueueDepth.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/QueueDepth.cs new file mode 100644 index 0000000000..9675d85684 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/TestDoubles/QueueDepth.cs @@ -0,0 +1,4 @@ +namespace Paramore.Brighter.Core.Tests.Tasks.TestDoubles; + +/// One observed confirmation queue depth, with the attributes it was reported under. +public sealed record QueueDepth(string? MessagingSystem, string? Destination, long Value); diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/When_confirmations_are_waiting_should_report_the_queue_depth.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/When_confirmations_are_waiting_should_report_the_queue_depth.cs new file mode 100644 index 0000000000..5e75aca387 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/When_confirmations_are_waiting_should_report_the_queue_depth.cs @@ -0,0 +1,55 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Tasks.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Tasks; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Tasks; + +public class BatchedCallbackQueueDepthTests +{ + private const int WaitingConfirmations = 4; + + [Fact] + public void When_confirmations_are_waiting_should_report_the_queue_depth() + { + //Arrange + using var observed = new ObservedQueueDepths(); + _ = new PublishConfirmationMeter(observed.MeterFactory); + + var topic = new RoutingKey(Guid.NewGuid().ToString()); // the registry is process-wide, so find our own queue + var queue = new BatchedCallbackQueue(MessagingSystem.Kafka, topic); + using var drainParked = new ManualResetEventSlim(false); + using var releaseDrain = new ManualResetEventSlim(false); + + try + { + //Act + // One confirmation holds the drain thread; the rest wait behind it + queue.Enqueue(() => + { + drainParked.Set(); + releaseDrain.Wait(); + return Task.CompletedTask; + }); + Assert.True(drainParked.Wait(TimeSpan.FromSeconds(5)), "The drain thread did not start"); + for (var i = 1; i < WaitingConfirmations; i++) + queue.Enqueue(() => Task.CompletedTask); + + var depths = observed.Observe().Where(d => d.Destination == topic.Value).ToList(); + + //Assert + var depth = Assert.Single(depths); + Assert.Equal(WaitingConfirmations, depth.Value); + Assert.Equal("kafka", depth.MessagingSystem); + } + finally + { + releaseDrain.Set(); + queue.Complete(); + } + } +} From e4d826dbfc22aea250b36125fcecb38cda21adbb Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:10:37 +0100 Subject: [PATCH 18/36] feat: publish-confirmation queue depth metric paramore.brighter.publish_confirmation.queue.depth, an ObservableUpDownCounter tagged with messaging.system and messaging.destination.name, counts confirmations queued or running for each producer. Producers are built outside DI, so each tagged BatchedCallbackQueue registers in an internal registry of weak references; PublishConfirmationMeter reads it on collection. AddBrighterInstrumentation registers the meter and creates it when the meter provider is built. Kafka tags its queue with its topic; RMQ.Sync with its topic, or its exchange when the publication has none. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../BrighterMetricsBuilderExtensions.cs | 6 ++ .../KafkaMessageProducer.cs | 3 +- .../RmqMessageProducer.cs | 7 +- .../BrighterSemanticConventions.cs | 1 + .../Observability/PublishConfirmationMeter.cs | 66 +++++++++++++++++++ .../Tasks/BatchedCallbackQueue.cs | 43 +++++++++++- .../Tasks/BatchedCallbackQueueRegistry.cs | 64 ++++++++++++++++++ .../Tasks/InFlightCallbackTracker.cs | 14 ++++ 8 files changed, 201 insertions(+), 3 deletions(-) create mode 100644 src/Paramore.Brighter/Observability/PublishConfirmationMeter.cs create mode 100644 src/Paramore.Brighter/Tasks/BatchedCallbackQueueRegistry.cs diff --git a/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs b/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs index 868d33fa84..7172875e8c 100644 --- a/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs +++ b/src/Paramore.Brighter.Extensions.Diagnostics/BrighterMetricsBuilderExtensions.cs @@ -23,6 +23,7 @@ THE SOFTWARE. */ #endregion +using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using OpenTelemetry.Metrics; using Paramore.Brighter.Observability; @@ -38,9 +39,14 @@ public static MeterProviderBuilder AddBrighterInstrumentation(this MeterProvider services.TryAddSingleton(); services.TryAddSingleton(); services.TryAddSingleton(); + services.TryAddSingleton(); builder.AddMeter(BrighterSemanticConventions.MeterName); }); + + // Nothing depends on the queue-depth meter, so create it when the meter provider is built + if (builder is IDeferredMeterProviderBuilder deferred) + deferred.Configure((serviceProvider, _) => serviceProvider.GetRequiredService()); return builder; } diff --git a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs index 71b113ea2c..f39006f576 100644 --- a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs @@ -81,7 +81,7 @@ event Func ISupportPublishConfirmationAsync.OnM // poll thread, and never one thread-pool item per report); the queue lets Dispose wait for those // callbacks — including the awaited Outbox mark-dispatched — after Flush() has drained the delivery // reports themselves. - private readonly BatchedCallbackQueue _confirmationCallbacks = new(); + private readonly BatchedCallbackQueue _confirmationCallbacks; public KafkaMessageProducer( KafkaMessagingGatewayConfiguration configuration, @@ -95,6 +95,7 @@ public KafkaMessageProducer( throw new ConfigurationException("Topic is required for a publication"); Publication = publication; + _confirmationCallbacks = new BatchedCallbackQueue(MessagingSystem.Kafka, publication.Topic!); ClientConfig = new ClientConfig { diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index efdeae65df..76b0df5941 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -63,7 +63,7 @@ public partial class RmqMessageProducer : RmqMessageGateway, IAmAMessageProducer // subscriber, so awaited callbacks run one at a time, in ack order, on the queue's single worker // (never one thread-pool item per ack). The queue lets Dispose wait for them — including the // awaited Outbox mark-dispatched — after WaitForConfirms has drained the broker acks themselves. - private readonly BatchedCallbackQueue _confirmationCallbacks = new(); + private readonly BatchedCallbackQueue _confirmationCallbacks; /// /// Action taken when a message is published, following receipt of a confirmation from the broker @@ -122,6 +122,11 @@ public RmqMessageProducer(RmqMessagingGatewayConnection connection, RmqPublicati { _publication = publication ?? new RmqPublication { MakeChannels = OnMissingChannel.Create }; _waitForConfirmsTimeOutInMilliseconds = _publication.WaitForConfirmsTimeOutInMilliseconds; + // A publication without a topic publishes to many routing keys on its exchange, so report it by the exchange + var destination = RoutingKey.IsNullOrEmpty(_publication.Topic) + ? new RoutingKey(connection.Exchange?.Name ?? string.Empty) + : _publication.Topic!; + _confirmationCallbacks = new BatchedCallbackQueue(MessagingSystem.RabbitMQ, destination); } /// diff --git a/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs b/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs index 0ea9a73594..a27fed12d0 100644 --- a/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs +++ b/src/Paramore.Brighter/Observability/BrighterSemanticConventions.cs @@ -106,6 +106,7 @@ public static class BrighterSemanticConventions public const string OutboxSweeperSweepDuration = "paramore.brighter.outbox_sweeper.sweep.duration"; public const string OutboxSweeperSweeps = "paramore.brighter.outbox_sweeper.sweeps"; public const string OutboxSweeperTickLag = "paramore.brighter.outbox_sweeper.tick.lag"; + public const string PublishConfirmationQueueDepth = "paramore.brighter.publish_confirmation.queue.depth"; public const string OutboxType = "paramore.brighter.outbox.type"; public const string ReplyTo = "paramore.brighter.replyto"; public const string RequestId = "paramore.brighter.request.id"; diff --git a/src/Paramore.Brighter/Observability/PublishConfirmationMeter.cs b/src/Paramore.Brighter/Observability/PublishConfirmationMeter.cs new file mode 100644 index 0000000000..cd64bc999e --- /dev/null +++ b/src/Paramore.Brighter/Observability/PublishConfirmationMeter.cs @@ -0,0 +1,66 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using Paramore.Brighter.Tasks; + +namespace Paramore.Brighter.Observability; + +/// +/// Reports how many publish confirmations are waiting to be raised, for each producer that raises them through a +/// . A growing depth means confirmations arrive faster than their handlers, such +/// as marking a message dispatched in an outbox, finish; left alone, messages stay unmarked and are swept again. +/// +public sealed class PublishConfirmationMeter +{ + /// + /// Creates the queue-depth instrument on the Brighter meter. + /// + /// Creates the meter. + public PublishConfirmationMeter(IMeterFactory meterFactory) + { + var meter = meterFactory.Create(BrighterSemanticConventions.MeterName); + + meter.CreateObservableUpDownCounter( + name: BrighterSemanticConventions.PublishConfirmationQueueDepth, + observeValues: ObserveQueueDepths, + unit: "{confirmation}", + description: "Number of publish confirmations queued or running, by producer."); + } + + private static IEnumerable> ObserveQueueDepths() + { + foreach (var queue in BatchedCallbackQueueRegistry.LiveQueues()) + { + if (queue.MessagingSystem is not { } system || queue.Destination is null) continue; + + yield return new Measurement( + queue.Depth, + new KeyValuePair(BrighterSemanticConventions.MessagingSystem, system.ToMessagingSystemName()), + new KeyValuePair(BrighterSemanticConventions.MessagingDestination, queue.Destination.Value)); + } + } +} diff --git a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs index f334cfec16..3d9315d436 100644 --- a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs +++ b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs @@ -29,6 +29,7 @@ THE SOFTWARE. */ using System.Linq; using System.Threading; using System.Threading.Tasks; +using Paramore.Brighter.Observability; namespace Paramore.Brighter.Tasks { @@ -53,6 +54,33 @@ public sealed class BatchedCallbackQueue private readonly BlockingCollection> _callbacks = new(); private readonly InFlightCallbackTracker _inFlight = new(); private int _threadStarted; + private readonly long? _registryId; + + /// + /// Creates a queue that the confirmation queue-depth metric does not report. + /// + public BatchedCallbackQueue() { } + + /// + /// Creates a queue whose depth reports under the producer's attributes. + /// + /// The broker the owning producer publishes to. + /// The topic the owning producer publishes to. + public BatchedCallbackQueue(MessagingSystem messagingSystem, RoutingKey destination) + { + MessagingSystem = messagingSystem; + Destination = destination; + _registryId = BatchedCallbackQueueRegistry.Register(this); + } + + internal MessagingSystem? MessagingSystem { get; } + + internal RoutingKey? Destination { get; } + + /// + /// How many callbacks are queued or running. + /// + internal int Depth => _inFlight.Count; /// /// Queues a callback; it starts after every callback queued before it has started. @@ -82,7 +110,12 @@ public void Enqueue(Func callback) /// /// Stops accepting callbacks; the thread ends once it has run those already queued. /// - public void Complete() => _callbacks.CompleteAdding(); + public void Complete() + { + _callbacks.CompleteAdding(); + if (Volatile.Read(ref _threadStarted) == 0) + Unregister(); + } // A confirmation that arrives after Complete (a late broker ack during dispose) is dropped rather than // thrown back onto the broker's thread; BlockingCollection throws once adding is complete. @@ -105,6 +138,14 @@ private void Drain() var batch = TakeBatch(first); BrighterAsyncContext.Run(() => Task.WhenAll(batch.Select(RunAndRelease))); } + + Unregister(); + } + + private void Unregister() + { + if (_registryId is { } id) + BatchedCallbackQueueRegistry.Unregister(id); } private List> TakeBatch(Func first) diff --git a/src/Paramore.Brighter/Tasks/BatchedCallbackQueueRegistry.cs b/src/Paramore.Brighter/Tasks/BatchedCallbackQueueRegistry.cs new file mode 100644 index 0000000000..db8ccc3d45 --- /dev/null +++ b/src/Paramore.Brighter/Tasks/BatchedCallbackQueueRegistry.cs @@ -0,0 +1,64 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; + +namespace Paramore.Brighter.Tasks +{ + /// + /// The live s whose depth is reported. Producers are built outside DI, so a + /// queue registers itself here and a meter reads the registry when its instrument is collected. It holds weak + /// references, so a queue that is never completed still goes when it is collected. + /// + internal static class BatchedCallbackQueueRegistry + { + private static readonly ConcurrentDictionary> s_queues = new(); + private static long s_nextId; + + /// Registers a queue and returns the id that unregisters it. + public static long Register(BatchedCallbackQueue queue) + { + var id = Interlocked.Increment(ref s_nextId); + s_queues[id] = new WeakReference(queue); + return id; + } + + public static void Unregister(long id) => s_queues.TryRemove(id, out _); + + public static IEnumerable LiveQueues() + { + foreach (var entry in s_queues) + { + if (entry.Value.TryGetTarget(out var queue)) + yield return queue; + else + s_queues.TryRemove(entry.Key, out _); + } + } + } +} diff --git a/src/Paramore.Brighter/Tasks/InFlightCallbackTracker.cs b/src/Paramore.Brighter/Tasks/InFlightCallbackTracker.cs index 48f4d4b4a9..c35a00f66d 100644 --- a/src/Paramore.Brighter/Tasks/InFlightCallbackTracker.cs +++ b/src/Paramore.Brighter/Tasks/InFlightCallbackTracker.cs @@ -47,6 +47,20 @@ public sealed class InFlightCallbackTracker private int _inFlight; private TaskCompletionSource? _drained; + /// + /// How many callbacks have begun and not yet ended. + /// + public int Count + { + get + { + lock (_lock) + { + return _inFlight; + } + } + } + /// /// Records that a callback has been started. Every call must be balanced by exactly one /// (call it in a finally), or will wait out its From 964a85cc8a64a46eb03e234fb3edf2023b94b472 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:14:03 +0100 Subject: [PATCH 19/36] test: sweeper and confirmation threads must not inherit ambient state Review of #4562, finding #4. RED (3/3): the drain thread ran every callback with the first enqueuer's AsyncLocal value, and sweeps ran with the value of the flow that called StartAsync. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TestDoubles/AmbientRecordingLock.cs | 28 +++++++++++ ...t_state_should_not_carry_it_into_sweeps.cs | 39 +++++++++++++++ ...tate_should_not_carry_it_into_callbacks.cs | 48 +++++++++++++++++++ 3 files changed, 115 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/AmbientRecordingLock.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Tasks/When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/AmbientRecordingLock.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/AmbientRecordingLock.cs new file mode 100644 index 0000000000..1a33924f12 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/AmbientRecordingLock.cs @@ -0,0 +1,28 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// An that always grants the lock and records the value an +/// held each time a sweep asked for it, so a test can see what ambient state a sweep runs with. +/// +public sealed class AmbientRecordingLock(AsyncLocal ambient) : IDistributedLock +{ + private readonly ManualResetEventSlim _obtained = new(false); + + public string? AmbientWhenObtained { get; private set; } + + public bool WaitForSweep(TimeSpan timeout) => _obtained.Wait(timeout); + + public Task ObtainLockAsync(string resource, CancellationToken cancellationToken) + { + AmbientWhenObtained = ambient.Value; + _obtained.Set(); + return Task.FromResult(Guid.NewGuid().ToString()); + } + + public Task ReleaseLockAsync(string resource, string lockId, CancellationToken cancellationToken) => + Task.CompletedTask; +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps.cs new file mode 100644 index 0000000000..d225290d20 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps.cs @@ -0,0 +1,39 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperAmbientStateTests +{ + private static readonly AsyncLocal s_ambient = new(); + + [Fact] + public async Task When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps() + { + //Arrange + using var outbox = new SweepableOutbox(); + var distributedLock = new AmbientRecordingLock(s_ambient); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + distributedLock, + new TimedOutboxSweeperOptions { TimerInterval = 1, MinimumMessageAge = TimeSpan.Zero }); + + //Act + await Task.Run(async () => + { + s_ambient.Value = "the starting caller's state"; // e.g. a logging scope or baggage + await sweeper.StartAsync(CancellationToken.None); + }); + var swept = distributedLock.WaitForSweep(TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(swept, "The sweeper did not sweep"); + Assert.Null(distributedLock.AmbientWhenObtained); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/Tasks/When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks.cs b/tests/Paramore.Brighter.Core.Tests/Tasks/When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks.cs new file mode 100644 index 0000000000..3d0a835825 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Tasks/When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks.cs @@ -0,0 +1,48 @@ +using System; +using System.Collections.Concurrent; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Tasks; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Tasks; + +public class BatchedCallbackQueueAmbientStateTests +{ + private const int Callbacks = 2; + private static readonly AsyncLocal s_ambient = new(); + + [Fact] + public async Task When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks() + { + //Arrange + var queue = new BatchedCallbackQueue(); + var seen = new ConcurrentQueue(); + using var allRan = new CountdownEvent(Callbacks); + Func recordAmbient = () => + { + seen.Enqueue(s_ambient.Value); + allRan.Signal(); + return Task.CompletedTask; + }; + + try + { + //Act + await Task.Run(() => + { + s_ambient.Value = "the first request's state"; // e.g. a logging scope or baggage + queue.Enqueue(recordAmbient); // starts the drain thread + }); + queue.Enqueue(recordAmbient); // a later confirmation, from a flow with no ambient state + + //Assert + Assert.True(allRan.Wait(TimeSpan.FromSeconds(3)), "The callbacks did not run"); + Assert.All(seen, Assert.Null); + } + finally + { + queue.Complete(); + } + } +} From 5e08f53acc2ac16aa99c765e5aa6343c34ceaee7 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:14:03 +0100 Subject: [PATCH 20/36] fix: start sweeper and confirmation threads without the caller's context new Thread().Start() captures the starting flow's ExecutionContext, so the drain thread kept the first enqueuer's AsyncLocals (logging scopes, baggage) for the producer's lifetime and every later confirmation ran with them; sweeps likewise ran with the StartAsync caller's state. Both starts now run under ExecutionContext.SuppressFlow(). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs | 4 +++- src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs | 6 +++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index face314af4..b3dcb4e1ae 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -100,7 +100,9 @@ public Task StartAsync(CancellationToken cancellationToken) Log.OutboxSweeperServiceIsStarting(s_logger); _sweepThread = new Thread(SweepUntilStopped) { IsBackground = true, Name = "Brighter Outbox Sweeper" }; - _sweepThread.Start(); + // Start with an empty context, so sweeps do not carry the starting caller's ambient state (AsyncLocals) + using (ExecutionContext.SuppressFlow()) + _sweepThread.Start(); return Task.CompletedTask; } diff --git a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs index 3d9315d436..5a63fbee5d 100644 --- a/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs +++ b/src/Paramore.Brighter/Tasks/BatchedCallbackQueue.cs @@ -96,7 +96,11 @@ public void Enqueue(Func callback) } if (Interlocked.CompareExchange(ref _threadStarted, 1, 0) == 0) - new Thread(Drain) { IsBackground = true, Name = "Brighter Confirmation Callbacks" }.Start(); + { + // Start with an empty context, so callbacks do not carry the first caller's ambient state (AsyncLocals) + using (ExecutionContext.SuppressFlow()) + new Thread(Drain) { IsBackground = true, Name = "Brighter Confirmation Callbacks" }.Start(); + } } /// From 232d069dc0703d885e15a026536067a3125c396a Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:17:10 +0100 Subject: [PATCH 21/36] test: a backward wall-clock step must not hold up the next sweep Review of #4562, finding #5. RED (3/3): after the wall clock stepped back an hour, one interval of elapsed time produced no sweep. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TestDoubles/SteppableWallClock.cs | 28 +++++++++++++ ...s_should_still_sweep_one_interval_later.cs | 41 +++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SteppableWallClock.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later.cs diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SteppableWallClock.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SteppableWallClock.cs new file mode 100644 index 0000000000..0d4bf56d97 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/TestDoubles/SteppableWallClock.cs @@ -0,0 +1,28 @@ +using System; +using System.Threading; +using Microsoft.Extensions.Time.Testing; + +namespace Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; + +/// +/// A whose wall clock can be stepped backwards, as an NTP correction or a resumed VM +/// does, while its elapsed time (timestamps) and timers only ever move forward with . +/// +public sealed class SteppableWallClock : TimeProvider +{ + private readonly FakeTimeProvider _elapsed = new(); + private TimeSpan _wallClockOffset = TimeSpan.Zero; + + public void Advance(TimeSpan delta) => _elapsed.Advance(delta); + + public void StepWallClockBack(TimeSpan step) => _wallClockOffset -= step; + + public override DateTimeOffset GetUtcNow() => _elapsed.GetUtcNow() + _wallClockOffset; + + public override long GetTimestamp() => _elapsed.GetTimestamp(); + + public override long TimestampFrequency => _elapsed.TimestampFrequency; + + public override ITimer CreateTimer(TimerCallback callback, object? state, TimeSpan dueTime, TimeSpan period) => + _elapsed.CreateTimer(callback, state, dueTime, period); +} diff --git a/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later.cs b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later.cs new file mode 100644 index 0000000000..4c54dd869f --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/Sweeping/When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later.cs @@ -0,0 +1,41 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Core.Tests.Sweeping.TestDoubles; +using Paramore.Brighter.Observability; +using Paramore.Brighter.Outbox.Hosting; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.Sweeping; + +public class TimedOutboxSweeperWallClockTests +{ + [Fact] + public async Task When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later() + { + //Arrange + var clock = new SteppableWallClock(); + using var outbox = new SweepableOutbox(); + using var measurements = new RecordedSweeperMeasurements(); + var sweeper = new TimedOutboxSweeper( + outbox.ScopeFactory, + new InMemoryLock(), + new TimedOutboxSweeperOptions { TimerInterval = 5, MinimumMessageAge = TimeSpan.Zero }, + timeProvider: clock, + meter: new SweeperMeter(measurements.MeterFactory)); + + await sweeper.StartAsync(CancellationToken.None); + var firstSweep = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.Sweeps).Count == 1, TimeSpan.FromSeconds(3)); + + //Act + clock.StepWallClockBack(TimeSpan.FromHours(1)); // e.g. an NTP correction + clock.Advance(TimeSpan.FromSeconds(5)); // one interval of elapsed time + var secondSweep = measurements.WaitFor(m => m.Of(RecordedSweeperMeasurements.Sweeps).Count == 2, TimeSpan.FromSeconds(3)); + await sweeper.StopAsync(CancellationToken.None); + sweeper.Dispose(); + + //Assert + Assert.True(firstSweep, "The sweeper did not sweep at start"); + Assert.True(secondSweep, "The sweeper did not sweep one interval later after the wall clock stepped back an hour"); + } +} From 3ad95ee1c333cb6fe13e56c8df45ffe1b316aa8d Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 09:17:11 +0100 Subject: [PATCH 22/36] fix: schedule sweeps by elapsed time, not the wall clock Due times, the re-anchor check and tick lag were computed from GetUtcNow, so a wall clock stepping backwards (an NTP correction, a resumed VM) pushed the next sweep out by the size of the step. They are now elapsed time from GetTimestamp on the sweeper's TimeProvider. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../TimedOutboxSweeper.cs | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs index b3dcb4e1ae..9dca9f2304 100644 --- a/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs +++ b/src/Paramore.Brighter.Outbox.Hosting/TimedOutboxSweeper.cs @@ -158,10 +158,14 @@ private void SweepUntilStopped() { try { - var due = _timeProvider.GetUtcNow(); - while (WaitUntil(due)) + // Schedule by elapsed time, not the wall clock, so a clock that steps backwards cannot hold up a sweep + var origin = _timeProvider.GetTimestamp(); + TimeSpan Elapsed() => _timeProvider.GetElapsedTime(origin); + + var due = Elapsed(); + while (WaitUntil(due, Elapsed)) { - var started = _timeProvider.GetUtcNow(); + var started = Elapsed(); Record(() => _meter.RecordTickLag(started - due)); Sweep(); @@ -177,17 +181,17 @@ private void SweepUntilStopped() // Waits on this thread, never on the pool: the real-time timeout wakes the sweeper even when no pool // thread is free to run the TimeProvider's timer callback. The callback lets a test clock wake it too. - private bool WaitUntil(DateTimeOffset due) + private bool WaitUntil(TimeSpan due, Func elapsed) { while (!_stopping.IsCancellationRequested) { - var wait = due - _timeProvider.GetUtcNow(); + var wait = due - elapsed(); if (wait <= TimeSpan.Zero) return true; using var timer = _timeProvider.CreateTimer(_ => SignalDue(), null, wait, Timeout.InfiniteTimeSpan); // The clock may have moved on before the timer existed to see it, so look again before waiting. - if (due - _timeProvider.GetUtcNow() <= TimeSpan.Zero) + if (due - elapsed() <= TimeSpan.Zero) return true; WaitHandle.WaitAny(new[] { _due, _stopping.Token.WaitHandle }, wait); @@ -204,7 +208,7 @@ private void SignalDue() // Keeps to the schedule, so a late or overrunning sweep shows as lag on the next one. A sweep that started // a whole interval or more late re-anchors the schedule, so the sweeper never sweeps back to back to catch up. - private DateTimeOffset NextDue(DateTimeOffset due, DateTimeOffset started) + private TimeSpan NextDue(TimeSpan due, TimeSpan started) { var anchor = started - due >= _interval ? started : due; return anchor + _interval; From 6f180d632b49d49ae8ecc2dd048e1042fc557b54 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 10:27:05 +0100 Subject: [PATCH 23/36] test: a Proactor handler's outbox clear keeps Polly on the pump thread Characterisation test for the continueOnCapturedContext change in 80512a496. A Proactor pump's handler clears the outbox (as PostAsync does) through a retry pipeline to a producer that fails its first send from its own thread. The post completes, OnRetry runs on the pump thread, and the handler resumes there. RED under the named mutation (Polly given its default ContinueOnCapturedContext = false): OnRetry ran on pool threads 5 and 10, not pump thread 19. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- ..._resilience_pipeline_on_the_pump_thread.cs | 97 +++++++++++++++++++ .../TestDoubles/FailsOnceOffPoolProducer.cs | 64 ++++++++++++ .../OutboxClearingCommandProcessor.cs | 31 ++++++ 3 files changed, 192 insertions(+) create mode 100644 tests/Paramore.Brighter.Core.Tests/MessageDispatch/Proactor/When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/FailsOnceOffPoolProducer.cs create mode 100644 tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/OutboxClearingCommandProcessor.cs diff --git a/tests/Paramore.Brighter.Core.Tests/MessageDispatch/Proactor/When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread.cs b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/Proactor/When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread.cs new file mode 100644 index 0000000000..4515dcc248 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/Proactor/When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread.cs @@ -0,0 +1,97 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using System.Transactions; +using Microsoft.Extensions.Time.Testing; +using Paramore.Brighter.Core.Tests.CommandProcessors.TestDoubles; +using Paramore.Brighter.Core.Tests.MessageDispatch.TestDoubles; +using Paramore.Brighter.Extensions; +using Paramore.Brighter.Observability; +using Paramore.Brighter.ServiceActivator; +using Polly; +using Polly.Registry; +using Polly.Retry; +using Xunit; + +namespace Paramore.Brighter.Core.Tests.MessageDispatch.Proactor; + +public class ProactorOutboxClearContextTests +{ + private readonly RoutingKey _incoming = new("MyCommand"); + private readonly RoutingKey _outgoing = new("Outgoing"); + + [Fact] + public async Task When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread() + { + //Arrange + var retryThread = 0; + var resiliencePipelines = new ResiliencePipelineRegistry(); + resiliencePipelines.TryAddBuilder(CommandProcessor.OutboxProducer, (builder, _) => builder.AddRetry(new RetryStrategyOptions + { + MaxRetryAttempts = 1, + Delay = TimeSpan.Zero, + OnRetry = _ => + { + retryThread = Environment.CurrentManagedThreadId; // where Polly resumed after the failed send + return default; + } + })); + resiliencePipelines.AddBrighterDefault(); + + var outgoingBus = new InternalBus(); + var producer = new FailsOnceOffPoolProducer(outgoingBus, new Publication { Topic = _outgoing }); + var outbox = new InMemoryOutbox(TimeProvider.System); + var outstanding = new Message(new MessageHeader(Id.Random(), _outgoing, MessageType.MT_EVENT), new MessageBody("posted by the handler")); + outbox.Add(outstanding, new RequestContext()); + var mediator = new OutboxProducerMediator( + new ProducerRegistry(new Dictionary { { _outgoing, producer } }), + resiliencePipelines, + new MessageMapperRegistry(new SimpleMessageMapperFactory(_ => null!), null), + new EmptyMessageTransformerFactory(), + new EmptyMessageTransformerFactoryAsync(), + new BrighterTracer(TimeProvider.System), + new FindPublicationByPublicationTopicOrRequestType(), + outbox); + + var commandProcessor = new OutboxClearingCommandProcessor(mediator, outstanding.Id); + var (pump, channel) = CreatePump(commandProcessor); + + //Act + var pumping = Task.Factory.StartNew(() => pump.Run(), TaskCreationOptions.LongRunning); + var handled = commandProcessor.WaitForHandler(TimeSpan.FromSeconds(5)); + channel.Enqueue(new Message(new MessageHeader(string.Empty, RoutingKey.Empty, MessageType.MT_QUIT), new MessageBody(""))); + await pumping; + await producer.DisposeAsync(); + + //Assert + Assert.True(handled, "The handler did not finish clearing the outbox"); + Assert.Single(outgoingBus.Stream(_outgoing)); // sent on the retry + Assert.Equal(commandProcessor.HandlerThread, retryThread); // Polly kept the pump's context + Assert.Equal(commandProcessor.HandlerThread, commandProcessor.ThreadAfterClear); + } + + private (IAmAMessagePump Pump, ChannelAsync Channel) CreatePump(IAmACommandProcessor commandProcessor) + { + var incomingBus = new InternalBus(); + var channel = new ChannelAsync(new("myChannel"), _incoming, + new InMemoryMessageConsumer(_incoming, incomingBus, new FakeTimeProvider(), ackTimeout: TimeSpan.FromMilliseconds(1000))); + + var mappers = new MessageMapperRegistry(null, new SimpleMessageMapperFactoryAsync(_ => new MyCommandMessageMapperAsync())); + mappers.RegisterAsync(); + + var pump = new ServiceActivator.Proactor(commandProcessor, _ => typeof(MyCommand), + mappers, new EmptyMessageTransformerFactoryAsync(), new InMemoryRequestContextFactory(), channel) + { + Channel = channel, TimeOut = TimeSpan.FromMilliseconds(5000), RequeueCount = 1 + }; + + var command = new TransformPipelineBuilderAsync(mappers, null, InstrumentationOptions.All) + .BuildWrapPipeline() + .WrapAsync(new MyCommand(), new RequestContext(), new Publication { Topic = _incoming }) + .Result; + incomingBus.Enqueue(command); + + return (pump, channel); + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/FailsOnceOffPoolProducer.cs b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/FailsOnceOffPoolProducer.cs new file mode 100644 index 0000000000..7011f8b768 --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/FailsOnceOffPoolProducer.cs @@ -0,0 +1,64 @@ +using System; +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; + +namespace Paramore.Brighter.Core.Tests.MessageDispatch.TestDoubles; + +/// +/// An async producer that completes each send from its own delivery thread, as Confluent's ProduceAsync +/// does (a task awaited plainly inside the +/// producer), and fails its first send, so a retry pipeline has to retry it. +/// +public sealed class FailsOnceOffPoolProducer : IAmAMessageProducerAsync +{ + private readonly InternalBus _bus; + private readonly BlockingCollection<(Message Message, TaskCompletionSource Delivered)> _deliveries = new(); + private readonly Thread _deliveryThread; + private int _sends; + + public FailsOnceOffPoolProducer(InternalBus bus, Publication publication) + { + _bus = bus; + Publication = publication; + _deliveryThread = new Thread(Deliver) { IsBackground = true, Name = "Off-pool delivery" }; + _deliveryThread.Start(); + } + + public Publication Publication { get; } + public Activity? Span { get; set; } + public IAmAMessageScheduler? Scheduler { get; set; } + + public async Task SendAsync(Message message, CancellationToken cancellationToken = default) + { + var delivered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _deliveries.Add((message, delivered), cancellationToken); + await delivered.Task; + } + + public Task SendWithDelayAsync(Message message, TimeSpan? delay, CancellationToken cancellationToken = default) + => SendAsync(message, cancellationToken); + + public ValueTask DisposeAsync() + { + _deliveries.CompleteAdding(); + _deliveryThread.Join(TimeSpan.FromSeconds(5)); + return ValueTask.CompletedTask; + } + + private void Deliver() + { + foreach (var (message, delivered) in _deliveries.GetConsumingEnumerable()) + { + if (Interlocked.Increment(ref _sends) == 1) + { + delivered.SetException(new InvalidOperationException("The broker refused the first send")); + continue; + } + + _bus.Enqueue(message); + delivered.SetResult(); + } + } +} diff --git a/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/OutboxClearingCommandProcessor.cs b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/OutboxClearingCommandProcessor.cs new file mode 100644 index 0000000000..3aec99995e --- /dev/null +++ b/tests/Paramore.Brighter.Core.Tests/MessageDispatch/TestDoubles/OutboxClearingCommandProcessor.cs @@ -0,0 +1,31 @@ +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Testing; + +namespace Paramore.Brighter.Core.Tests.MessageDispatch.TestDoubles; + +/// +/// Stands in for a handler that clears the outbox while handling a command, as PostAsync does after it +/// deposits a message: it calls the mediator's ClearOutboxAsync and records the threads it ran on. +/// +internal sealed class OutboxClearingCommandProcessor(IAmAnOutboxProducerMediator mediator, Id outstanding) : SpyCommandProcessor +{ + private readonly ManualResetEventSlim _handled = new(false); + + public int HandlerThread { get; private set; } + public int ThreadAfterClear { get; private set; } + + public bool WaitForHandler(System.TimeSpan timeout) => _handled.Wait(timeout); + + public override async Task SendAsync( + T command, + RequestContext? requestContext = null, + bool continueOnCapturedContext = true, + CancellationToken cancellationToken = default) + { + HandlerThread = System.Environment.CurrentManagedThreadId; + await mediator.ClearOutboxAsync([outstanding], new RequestContext(), continueOnCapturedContext, cancellationToken: cancellationToken); + ThreadAfterClear = System.Environment.CurrentManagedThreadId; + _handled.Set(); + } +} From 6d2a8d7f192d2170e593b6397f1afce0a18a6dc1 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 11:24:29 +0100 Subject: [PATCH 24/36] docs: record the #4562 review fixes in the release note, ADR and bugfix record - Release note: the sweep stays on its thread across awaited sends, elapsed-time scheduling, batched confirmations, empty ExecutionContext, the queue-depth metric, and continueOnCapturedContext reaching Polly (including the Proactor case). - ADR 0081 amendment: the batch size is a private constant of 32. - Bugfix record: review tests 9-13, fixes, and re-verification. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../bugfix.md | 100 ++++++++++++++++++ ...081-timed-outbox-sweeper-health-metrics.md | 10 +- release_notes.md | 49 +++++++-- 3 files changed, 143 insertions(+), 16 deletions(-) diff --git a/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md b/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md index 12eca28a77..c4bc282fd4 100644 --- a/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md +++ b/bugfixes/0052-timed-outbox-sweeper-starvation/bugfix.md @@ -633,6 +633,43 @@ A `dotnet-dump` of an affected pod (`threadpool`, `clrstack -all`) would settle - **RED:** compilation fails because `IAmABrighterSweeperMeter`, `SweepOutcome`, `SweeperMeter` and the `meter:` and `timeProvider:` constructor parameters do not exist yet. +9. **Reopened after review of #4562 (2026-10-09), finding #1/#2.** + `.../Sweeping/When_thread_pool_workers_are_all_blocked_and_sends_complete_off_the_pool_should_still_sweep_a_batch.cs` + (uses `TestDoubles/OffPoolCompletingProducer`). + - **Why:** test 1 uses `InMemoryMessageProducer`, whose `SendAsync` completes inline, so it only + proves the tick fires. Confluent.Kafka 2.15.1's `TypedTaskDeliveryHandlerShim` completes + `ProduceAsync` with `RunContinuationsAsynchronously` (checked in its IL), from librdkafka's thread. + On the sweeper thread, which has no `SynchronizationContext`, every continuation after an awaited + send goes to the pool. + - **The double** completes each send from a dedicated delivery thread through a + `RunContinuationsAsynchronously` TCS, as Confluent does. Five messages are outstanding. + - **RED (3/3):** "dispatched 1 of 5 outstanding messages within 00:00:03". + +10. **Review #3 (batched drain), ADR 0081 amendment.** + - `Tasks/When_more_callbacks_are_queued_than_the_batch_size_should_run_a_batch_at_once.cs`. RED + (3/3): "Expected 32 callbacks in progress at once, but at most 1 were". + - `Tasks/When_confirmations_are_waiting_should_report_the_queue_depth.cs` (doubles + `Tasks/TestDoubles/ObservedQueueDepths`, `QueueDepth`). RED: the meter and the tagged queue + constructor did not exist. + - **Replaced** tests 6 and 7 with `When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool` + (Kafka and RMQ.Sync): 2–32 handlers at once and none on a pool thread. Mutation-checked on both + brokers: a serial drain gives "Actual: 1"; no `BrighterAsyncContext` gives "ran on a + thread-pool thread". +11. **Review #4.** `Tasks/When_a_callback_is_first_queued_from_a_flow_with_ambient_state_should_not_carry_it_into_callbacks.cs` + and `Sweeping/When_the_sweeper_is_started_from_a_flow_with_ambient_state_should_not_carry_it_into_sweeps.cs` + (`TestDoubles/AmbientRecordingLock`). RED (3/3): both saw the starting caller's `AsyncLocal` value. +12. **Review #5.** `Sweeping/When_the_wall_clock_steps_backwards_should_still_sweep_one_interval_later.cs` + (`TestDoubles/SteppableWallClock`). RED (3/3): no sweep one interval after a one-hour backward step. + +13. **Characterisation of the Polly `continueOnCapturedContext` change on a Proactor pump** (raised by the + maintainer): `MessageDispatch/Proactor/When_a_handler_clears_the_outbox_should_run_the_resilience_pipeline_on_the_pump_thread.cs` + (`TestDoubles/FailsOnceOffPoolProducer`, `OutboxClearingCommandProcessor`). + - **Green on arrival.** RED under the named mutation (Polly given its default of `false`): `OnRetry` + ran on pool threads 5 and 10, not pump thread 19. + - It pins that a Proactor handler's post or clear completes, runs Polly's callbacks on the pump + thread, and resumes there. Blocking on `PostAsync` from the pump thread already deadlocked, so + no new deadlock is introduced. + ## Fix Four commits on `bugfix/4560-timed-outbox-sweeper-starvation`: @@ -678,6 +715,37 @@ Four commits on `bugfix/4560-timed-outbox-sweeper-starvation`: | Existing Kafka confirmation and dispose tests | 12/12 | | Existing RMQ.Sync confirmation and dispose tests | 30/30 | + +### Fixes after review of #4562 (2026-10-09) + +1. **Review #1/#2:** `eece13a93` test, `80512a496` fix. + - The sweep runs in `BrighterAsyncContext.Run` on the sweeper thread. + - **Running it in a context was not enough on its own.** A `ConfigureAwait(false)` continuation is + not inlined on a thread with a `SynchronizationContext`; it goes to the pool. Two such hops were + found by tracing threads in the test double: + - `BackgroundDispatchUsingAsync`, whose only caller is the sweeper, now passes + `continueOnCapturedContext: true`; + - `ExecuteWithResiliencePipelineAsync` now passes the caller's `continueOnCapturedContext` to + Polly through a pooled `ResilienceContext`. It used to get Polly's default of `false`. + - Removing either change turns the test red again. + - The reviewer's claim that `ConfigureAwait(false)` "resumes on whichever thread completed the + inner task" is wrong in exactly this case. + - Confluent.Kafka 2.15.1's `TypedTaskDeliveryHandlerShim` passes `RunContinuationsAsynchronously` + (`ldc.i4.s 0x40`); this was checked in its IL. +2. **Review #3:** `10287babe` rename; `5bd972f69`/`cf8da1438` batched drain; `0ff90cfa6` replaced + broker tests; `398db5c2a`/`e4d826dbf` queue-depth metric; ADR 0081 amended (`docs:` commit). +3. **Race fix, found while testing #3:** `08a8ff64c` re-checks the due time after creating the timer. + The tick-lag test had failed 2 of 8 runs under load; it was green 10/10 afterwards. +4. **Review #4:** `964a85cc8`/`5e08f53ac`: `ExecutionContext.SuppressFlow()` around both thread starts. +5. **Review #5:** `232d069dc`/`3ad95ee1c`: the schedule uses elapsed time from `GetTimestamp`. + +**Review points not acted on:** +- **6a:** `StopAsync`'s continuation needs a pool thread. This is bounded by the host's shutdown + timeout. +- **6b:** a sweep in flight is not cancelled on stop. This is by design, and a test covers it. +- **6c:** "log the backlog at dispose". This is already done: `WaitForConfirmationCallbacks` logs + `stillInFlight`. + ## Verify Full suites run on 2026-10-08 against `ea92447c9` (all branch commits), on net9.0 and net10.0. The @@ -706,3 +774,35 @@ them come from the local environment rather than this change: RMQ.Async.Tests was not run: RMQ.Async's `RmqMessageProducer` is unchanged. **Dropped from scope:** the `samples/` harness (see Scope Notes). + +### Re-verify after the review fixes (2026-10-09) + +The run was at `6f180d632`, on net9.0 and net10.0. A fresh Kafka container ran with +`schema-registry`, alongside a stock RabbitMQ on 5672 and the native-delay RabbitMQ on 5673. + +| Suite | net9.0 | net10.0 | +| --- | --- | --- | +| Core.Tests | 1661 passed, 7 skipped | 1661 passed, 7 skipped | +| Kafka.Tests | **265/265** | **265/265** | +| RMQ.Sync.Tests | 186 passed, 1 skipped, without mTLS | 193 passed, 1 skipped, 9 mTLS failed | +| Extensions.Tests | 753 / 750, all passed | | +| Testing.Tests, InMemory.Tests | all passed | all passed | +| RMQ.Async.Tests | 358 passed, 10 failed: 8 mTLS and 2 `mary` | same | +| Extensions.AspNetCore.Tests | flaky, see below | flaky | + +**Environment problems found and fixed along the way:** +- **Clock drift:** the Podman VM's clock was 1 h 19 min behind the host, so Kafka produced + `Invalid timestamp` and about 110 consumer tests failed. It was stepped with `chronyc makestep`. +- **No native-delay broker:** port 5673 had no broker. The image from `docker/RabbitMQ/Dockerfile` + crashes with `.erlang.cookie: eacces`, probably because `rabbitmq-plugins enable --offline` runs as + root during the build. It was run with `--user 0`, so the entrypoint fixes ownership and drops + privileges. + +**Failures that remain are environmental:** +- The mTLS tests need generated certificates. +- `DispatchBuilderTests` fail on a `mary` durable-queue mismatch, because RMQ.Sync and RMQ.Async + share the queue name. + +**Extensions.AspNetCore is flaky.** `UseInboxHandler`'s static initializer throws +`ObjectDisposedException`, because the shared logger factory has already been disposed. The failure +count varies from run to run (1–6), and a rerun passes 65/65 on both frameworks. diff --git a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md index 044992124a..f34722b635 100644 --- a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md +++ b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md @@ -256,18 +256,18 @@ The queue becomes `BatchedCallbackQueue` (`Paramore.Brighter.Tasks`). It keeps t never the thread pool. - **The drain thread takes a batch.** It waits for one callback, then takes whatever else is already - queued, up to `maxBatchSize` (default **32**) in total. + queued, up to **32** in total. - **It starts every callback in the batch, in queue order,** inside one `BrighterAsyncContext.Run`, and waits for all of them before taking the next batch. So their I/O overlaps, while their continuations still come back to the drain thread. - **The batch size bounds concurrency.** A batch of at most 32 stays below the default connection pool size of SqlClient and Npgsql (100), which leaves room for the application. The size is a - constructor parameter, not a public option: no producer exposes it in this amendment. + private constant: no test or producer needs to vary it, so it is not a parameter or an option. - **Callbacks still run in a `BrighterAsyncContext`.** Only a continuation that opts out with `ConfigureAwait(false)` can land on the pool. This is unchanged. - **Unchanged:** `TryWait`, `Complete`, dropping a confirmation added after `Complete`, and swallowing a callback's escaped exception. -- **Changed:** callbacks may now complete out of order, and up to `maxBatchSize` may be in progress +- **Changed:** callbacks may now complete out of order, and up to 32 may be in progress at once. #### Queue depth @@ -297,7 +297,7 @@ never the thread pool. | Assembly | Type | Change | | --- | --- | --- | -| Paramore.Brighter | `SerialCallbackQueue` | Renamed `BatchedCallbackQueue`; batched drain; new constructor parameters for its attributes and `maxBatchSize` | +| Paramore.Brighter | `SerialCallbackQueue` | Renamed `BatchedCallbackQueue`; batched drain; a new constructor that takes its attributes (`MessagingSystem`, `RoutingKey`) | | Paramore.Brighter | `BatchedCallbackQueueRegistry` | New, internal | | Paramore.Brighter | `PublishConfirmationMeter` | New | | Paramore.Brighter | `BrighterSemanticConventions` | New constant for the instrument name | @@ -321,7 +321,7 @@ never the thread pool. #### Consequences (amendment) -- **Positive:** confirmation throughput with a database outbox is bounded by `maxBatchSize` +- **Positive:** confirmation throughput with a database outbox is bounded by 32 overlapping round trips, not one. A backlog is visible before it causes resends. - **Negative:** the in-order guarantee from #4560 goes away. Nothing documented it, and nothing in Brighter depends on it. The approved Kafka and RMQ.Sync tests that assert "one at a time, in diff --git a/release_notes.md b/release_notes.md index 35968abb17..4ade563443 100644 --- a/release_notes.md +++ b/release_notes.md @@ -15,11 +15,23 @@ per publish confirmation, which added to that pressure under load. - **A failed sweep no longer crashes the process.** The sweeper logs the error and tries again on the next sweep. It also releases its lock on every path, so one failure no longer blocks every later sweep. -- **Kafka and `RMQ.Sync` raise publish confirmations on one thread per producer**, one at a time, in - delivery order. -- **New sweeper metrics.** `AddBrighterInstrumentation` registers - `paramore.brighter.outbox_sweeper.tick.lag`, `.sweep.duration` and `.sweeps` (by outcome). See - [ADR 0081](docs/adr/0081-timed-outbox-sweeper-health-metrics.md). +- **A sweep stays on the sweeper's thread across awaited sends.** A Kafka send completes on + Confluent's own thread, so before this each later message in a sweep waited for a pool thread. Now + a starved pool cannot slow a Kafka sweep down. A producer or outbox that itself awaits with + `ConfigureAwait(false)`, as database clients and the AWS SDK do, still resumes on the pool. +- **The schedule uses elapsed time, not the wall clock**, so a clock that steps backwards (an NTP + correction, a resumed VM) no longer delays the next sweep. +- **Kafka and `RMQ.Sync` raise publish confirmations on one thread per producer**, in batches of up + to 32 rather than one work item per confirmation. +- **The sweeper and confirmation threads start with an empty `ExecutionContext`**, so they no longer + keep the `AsyncLocal` state (logging scopes, baggage) of whichever caller started them. +- **New metrics.** `AddBrighterInstrumentation` registers: + - `paramore.brighter.outbox_sweeper.tick.lag`, `.sweep.duration` and `.sweeps` (by outcome); + - `paramore.brighter.publish_confirmation.queue.depth`, by `messaging.system` and + `messaging.destination.name`. It counts confirmations waiting or running for each producer, so a + backlog is visible before messages are swept again. + + See [ADR 0081](docs/adr/0081-timed-outbox-sweeper-health-metrics.md). #### Behaviour change: `TimerInterval` below 1 is rejected @@ -27,12 +39,27 @@ per publish confirmation, which added to that pressure under load. a `ConfigurationException` when the sweeper is created. Before, 0 swept once and then never again, and a negative value threw when the sweeper started. -#### Behaviour change: publish-confirmation subscribers run one at a time +#### Behaviour change: publish-confirmation subscribers run in batches on one thread On Kafka and `RMQ.Sync`, subscribers to `OnMessagePublished` and to the awaited confirmation event -used to run concurrently. They now run one at a time, in the order the broker confirmed the -messages. A slow subscriber therefore delays the confirmations behind it. Brighter's own subscriber -marks the message dispatched in the outbox. +used to run each on its own thread-pool work item. They now run on the producer's confirmation +thread: up to 32 start together, in the order the broker confirmed the messages, and may finish in +any order. A subscriber's continuations come back to that thread unless it uses +`ConfigureAwait(false)`. Brighter's own subscriber marks the message dispatched in the outbox. + +#### Behaviour change: `continueOnCapturedContext` now reaches the resilience pipeline + +`OutboxProducerMediator` passes the caller's `continueOnCapturedContext` to Polly when the request +has no `ResilienceContext` of its own. Before, that path always used Polly's default of `false`. A +caller that posts with `continueOnCapturedContext: true` (the default for `PostAsync` and +`ClearOutboxAsync`) from inside a `SynchronizationContext` now has Polly's continuations come back to +that context too. + +The `SynchronizationContext` that matters most here is the Proactor's own. A handler on a Proactor pump +that posts or clears the outbox now runs the outbox producer's resilience pipeline, including callbacks +such as a retry's `OnRetry`, on the pump's thread rather than on a thread-pool thread. The mediator's +own awaits, and Kafka's, already resumed there. This adds no new way to deadlock. Blocking on `PostAsync` +from a pump thread already deadlocked. #### Cost: one thread per confirming producer @@ -42,8 +69,8 @@ app with twenty publications runs twenty confirmation threads, plus one sweeper An idle thread costs little. Under load, though, those threads compete with the rest of the app for CPU. That can matter on a pod limited to 1–2 CPUs. If you run many publications on a small pod, watch -CPU throttling (`container_cpu_cfs_throttled_periods_total` on Kubernetes) and the new sweeper -tick-lag metric. +CPU throttling (`container_cpu_cfs_throttled_periods_total` on Kubernetes), the sweeper tick-lag +metric and the confirmation queue depth. ### AWS SQS, GCP Pub/Sub and RocketMQ: `requeueCount` now runs down (#4341, spec 0037) From 6b546641b80050fe7454640ab5b60695bb475af1 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 14:37:05 +0100 Subject: [PATCH 25/36] fix: the RMQ.Sync producer confirms every message a coalesced ack covers RabbitMQ may coalesce publisher confirms: one basic.ack with multiple=true settles every delivery tag up to and including its own. The Sync producer settled only the exact tag, so the other messages the ack covered never raised a publish confirmation, and an outbox would send them again. RMQ.Async already handles this. Exposed by the intermittent rabbitmq-sync-ci failure of RmqConfirmationBatchingTests on #4562. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../RmqMessageProducer.cs | 19 ++++- ...ld_confirm_every_message_the_ack_covers.cs | 57 +++++++++++++ .../CoalescingConfirmsRmqChannel.cs | 82 +++++++++++++++++++ .../CoalescingConfirmsRmqProducer.cs | 44 ++++++++++ 4 files changed, 198 insertions(+), 4 deletions(-) create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers.cs create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqProducer.cs diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index 76b0df5941..9a2a94af72 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -25,8 +25,10 @@ THE SOFTWARE. */ using System; using System.Collections.Concurrent; +using System.Collections.Generic; using System.Diagnostics; using System.IO; +using System.Linq; using System.Text.Json; using System.Threading; using System.Threading.Tasks; @@ -277,14 +279,23 @@ private void OnPublishFailed(object? sender, BasicNackEventArgs e) private void OnPublishSucceeded(object? sender, BasicAckEventArgs e) { - if (_pendingConfirmations.TryGetValue(e.DeliveryTag, out PendingConfirmation confirmation)) + foreach (var deliveryTag in ConfirmedDeliveryTags(e.DeliveryTag, e.Multiple)) { - RaisePublishConfirmation(new PublishConfirmationResult(true, confirmation.MessageId, confirmation.Topic, confirmation.Context)); - _pendingConfirmations.TryRemove(e.DeliveryTag, out PendingConfirmation _); - Log.PublishedMessageInformation(s_logger, confirmation.MessageId.Value); + if (_pendingConfirmations.TryGetValue(deliveryTag, out PendingConfirmation confirmation)) + { + RaisePublishConfirmation(new PublishConfirmationResult(true, confirmation.MessageId, confirmation.Topic, confirmation.Context)); + _pendingConfirmations.TryRemove(deliveryTag, out PendingConfirmation _); + Log.PublishedMessageInformation(s_logger, confirmation.MessageId.Value); + } } } + // The broker may coalesce confirms: with multiple set, one frame settles every tag up to and including its own + private IEnumerable ConfirmedDeliveryTags(ulong deliveryTag, bool multiple) + => multiple + ? _pendingConfirmations.Keys.Where(pending => pending <= deliveryTag).OrderBy(pending => pending).ToArray() + : [deliveryTag]; + private void RaisePublishConfirmation(PublishConfirmationResult result) { // The sync event stays on the connection loop thread, matching its long-standing behavior. diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers.cs new file mode 100644 index 0000000000..d8191bbb5f --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers.cs @@ -0,0 +1,57 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; +using Xunit; + +namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; + +[Trait("Category", "RMQ")] +[Collection("RMQ")] +public class RmqCoalescedAckConfirmationTests : IDisposable +{ + private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); + private readonly CoalescingConfirmsRmqProducer _messageProducer; + + public RmqCoalescedAckConfirmationTests() + { + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672/%2f")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + _messageProducer = new CoalescingConfirmsRmqProducer(rmqConnection); + + //we need a queue to avoid a discard + new QueueFactory(rmqConnection, new ChannelName(Guid.NewGuid().ToString()), new RoutingKeys(_routingKey)) + .Create(TimeSpan.FromMilliseconds(1000)); + } + + [Fact] + public void When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers() + { + //Arrange + var confirmations = new List(); + _messageProducer.OnMessagePublished += confirmations.Add; + + var sent = Enumerable.Range(0, 3) + .Select(_ => new Message( + new MessageHeader(Id.Random(), _routingKey, MessageType.MT_COMMAND), + new MessageBody("confirm me"))) + .ToArray(); + + foreach (var message in sent) + _messageProducer.Send(message); + + //Act — one frame acks delivery tags 1 to 3 + _messageProducer.Confirms.RaiseAck(deliveryTag: 3, multiple: true); + + //Assert — each covered message is confirmed once, in send order + Assert.Equal(sent.Select(m => m.Id), confirmations.Select(c => c.MessageId)); + Assert.All(confirmations, c => Assert.True(c.Success)); + } + + public void Dispose() => _messageProducer.Dispose(); +} diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs new file mode 100644 index 0000000000..312a2fc0d3 --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs @@ -0,0 +1,82 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +#nullable enable + +using System; +using System.Reflection; +using System.Runtime.ExceptionServices; +using RabbitMQ.Client; +using RabbitMQ.Client.Events; + +namespace Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; + +// Forward broker I/O to the real channel, but hold back the producer's publisher-confirm handlers, so +// the test, not broker timing, decides how acks and nacks are framed. +public class CoalescingConfirmsRmqChannel : DispatchProxy +{ + private IModel _channel = null!; + private EventHandler? _acks; + private EventHandler? _nacks; + + public static CoalescingConfirmsRmqChannel Wrap(IModel channel) + { + var proxy = Create(); + var coalescing = (CoalescingConfirmsRmqChannel)proxy; + coalescing._channel = channel; + return coalescing; + } + + // One broker frame: the client raises a single event, carrying the broker's multiple flag as-is + public void RaiseAck(ulong deliveryTag, bool multiple) + => _acks?.Invoke(this, new BasicAckEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); + + public void RaiseNack(ulong deliveryTag, bool multiple) + => _nacks?.Invoke(this, new BasicNackEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); + + protected override object? Invoke(MethodInfo? targetMethod, object?[]? args) + { + var method = targetMethod!; + switch (method.Name) + { + case "add_BasicAcks": + _acks += (EventHandler)args![0]!; + return null; + case "add_BasicNacks": + _nacks += (EventHandler)args![0]!; + return null; + } + + try + { + return method.Invoke(_channel, args); + } + catch (TargetInvocationException error) when (error.InnerException != null) + { + ExceptionDispatchInfo.Capture(error.InnerException).Throw(); + throw; + } + } +} diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqProducer.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqProducer.cs new file mode 100644 index 0000000000..0eb29b0aab --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqProducer.cs @@ -0,0 +1,44 @@ +#region Licence + +/* The MIT License (MIT) +Copyright © 2026 Ian Cooper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. */ + +#endregion + +#nullable enable + +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using RabbitMQ.Client; + +namespace Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; + +public class CoalescingConfirmsRmqProducer(RmqMessagingGatewayConnection connection) + : RmqMessageProducer(connection) +{ + public CoalescingConfirmsRmqChannel Confirms => (CoalescingConfirmsRmqChannel)Channel!; + + protected override void ConnectToBroker(OnMissingChannel makeExchange) + { + base.ConnectToBroker(makeExchange); + if (Channel is not CoalescingConfirmsRmqChannel) + Channel = (IModel)(object)CoalescingConfirmsRmqChannel.Wrap(Channel!); + } +} From 0af74dcf18d6adcf70e7af0ee8eb9a485aedcc61 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 15:10:31 +0100 Subject: [PATCH 26/36] fix: the RMQ.Sync producer fails every message a coalesced nack covers A basic.nack with multiple=true rejects every delivery tag up to and including its own. Settle that whole range, as for acks, and have both handlers share one SettleConfirmations method. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../RmqMessageProducer.cs | 27 ++++----- ...ould_fail_every_message_the_nack_covers.cs | 57 +++++++++++++++++++ 2 files changed, 69 insertions(+), 15 deletions(-) create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers.cs diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index 9a2a94af72..53fcdd7b95 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -267,25 +267,22 @@ protected override void Dispose(bool disposing) } } - private void OnPublishFailed(object? sender, BasicNackEventArgs e) - { - if (_pendingConfirmations.TryGetValue(e.DeliveryTag, out PendingConfirmation confirmation)) - { - RaisePublishConfirmation(new PublishConfirmationResult(false, confirmation.MessageId, confirmation.Topic, confirmation.Context)); - _pendingConfirmations.TryRemove(e.DeliveryTag, out PendingConfirmation _); - Log.FailedToPublishMessage(s_logger, confirmation.MessageId.Value); - } - } + private void OnPublishFailed(object? sender, BasicNackEventArgs e) => SettleConfirmations(e.DeliveryTag, e.Multiple, success: false); + + private void OnPublishSucceeded(object? sender, BasicAckEventArgs e) => SettleConfirmations(e.DeliveryTag, e.Multiple, success: true); - private void OnPublishSucceeded(object? sender, BasicAckEventArgs e) + private void SettleConfirmations(ulong deliveryTag, bool multiple, bool success) { - foreach (var deliveryTag in ConfirmedDeliveryTags(e.DeliveryTag, e.Multiple)) + foreach (var confirmedTag in ConfirmedDeliveryTags(deliveryTag, multiple)) { - if (_pendingConfirmations.TryGetValue(deliveryTag, out PendingConfirmation confirmation)) + if (_pendingConfirmations.TryGetValue(confirmedTag, out PendingConfirmation confirmation)) { - RaisePublishConfirmation(new PublishConfirmationResult(true, confirmation.MessageId, confirmation.Topic, confirmation.Context)); - _pendingConfirmations.TryRemove(deliveryTag, out PendingConfirmation _); - Log.PublishedMessageInformation(s_logger, confirmation.MessageId.Value); + RaisePublishConfirmation(new PublishConfirmationResult(success, confirmation.MessageId, confirmation.Topic, confirmation.Context)); + _pendingConfirmations.TryRemove(confirmedTag, out PendingConfirmation _); + if (success) + Log.PublishedMessageInformation(s_logger, confirmation.MessageId.Value); + else + Log.FailedToPublishMessage(s_logger, confirmation.MessageId.Value); } } } diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers.cs new file mode 100644 index 0000000000..446b7cab6c --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers.cs @@ -0,0 +1,57 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; +using Xunit; + +namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; + +[Trait("Category", "RMQ")] +[Collection("RMQ")] +public class RmqCoalescedNackConfirmationTests : IDisposable +{ + private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); + private readonly CoalescingConfirmsRmqProducer _messageProducer; + + public RmqCoalescedNackConfirmationTests() + { + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672/%2f")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + _messageProducer = new CoalescingConfirmsRmqProducer(rmqConnection); + + //we need a queue to avoid a discard + new QueueFactory(rmqConnection, new ChannelName(Guid.NewGuid().ToString()), new RoutingKeys(_routingKey)) + .Create(TimeSpan.FromMilliseconds(1000)); + } + + [Fact] + public void When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers() + { + //Arrange + var confirmations = new List(); + _messageProducer.OnMessagePublished += confirmations.Add; + + var sent = Enumerable.Range(0, 3) + .Select(_ => new Message( + new MessageHeader(Id.Random(), _routingKey, MessageType.MT_COMMAND), + new MessageBody("reject me"))) + .ToArray(); + + foreach (var message in sent) + _messageProducer.Send(message); + + //Act — one frame nacks delivery tags 1 to 3 + _messageProducer.Confirms.RaiseNack(deliveryTag: 3, multiple: true); + + //Assert — each covered message is reported as failed once, in send order + Assert.Equal(sent.Select(m => m.Id), confirmations.Select(c => c.MessageId)); + Assert.All(confirmations, c => Assert.False(c.Success)); + } + + public void Dispose() => _messageProducer.Dispose(); +} From dda706a4723cd8fe4b76928ee0b26216ca764851 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 15:49:40 +0100 Subject: [PATCH 27/36] fix: a throwing RMQ.Sync confirmation subscriber neither loses nor repeats a confirmation The producer raised a confirmation before removing it from the pending set. If an OnMessagePublished subscriber threw, the rest of a coalesced range was skipped, and because the ack handler is subscribed once per Send, the next invocation raised the same confirmation again. Claim each pending confirmation with TryRemove before raising it, and log a faulting sync subscriber rather than letting it escape the settle loop. The test channel now calls each ack/nack subscriber in turn and swallows its exception, as RabbitMQ.Client does. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../RmqMessageProducer.cs | 16 +++-- ...should_still_confirm_every_message_once.cs | 60 +++++++++++++++++++ .../CoalescingConfirmsRmqChannel.cs | 22 ++++++- 3 files changed, 92 insertions(+), 6 deletions(-) create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once.cs diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index 53fcdd7b95..c68c98ea11 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -275,10 +275,10 @@ private void SettleConfirmations(ulong deliveryTag, bool multiple, bool success) { foreach (var confirmedTag in ConfirmedDeliveryTags(deliveryTag, multiple)) { - if (_pendingConfirmations.TryGetValue(confirmedTag, out PendingConfirmation confirmation)) + // Claim before raising: the handlers are subscribed once per Send, so only one invocation may win each tag + if (_pendingConfirmations.TryRemove(confirmedTag, out PendingConfirmation confirmation)) { RaisePublishConfirmation(new PublishConfirmationResult(success, confirmation.MessageId, confirmation.Topic, confirmation.Context)); - _pendingConfirmations.TryRemove(confirmedTag, out PendingConfirmation _); if (success) Log.PublishedMessageInformation(s_logger, confirmation.MessageId.Value); else @@ -295,8 +295,16 @@ private IEnumerable ConfirmedDeliveryTags(ulong deliveryTag, bool multipl private void RaisePublishConfirmation(PublishConfirmationResult result) { - // The sync event stays on the connection loop thread, matching its long-standing behavior. - OnMessagePublished?.Invoke(result); + // The sync event stays on the connection loop thread, matching its long-standing behavior. A subscriber + // that throws must not stop the rest of a coalesced ack from settling. + try + { + OnMessagePublished?.Invoke(result); + } + catch (Exception ex) + { + Log.ConfirmationCallbackFault(s_logger, result.MessageId.Value, ex); + } var handlers = _onMessagePublishedAsync; if (handlers is null) diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once.cs new file mode 100644 index 0000000000..36d4530868 --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once.cs @@ -0,0 +1,60 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; +using Xunit; + +namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; + +[Trait("Category", "RMQ")] +[Collection("RMQ")] +public class RmqThrowingConfirmationSubscriberTests : IDisposable +{ + private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); + private readonly CoalescingConfirmsRmqProducer _messageProducer; + + public RmqThrowingConfirmationSubscriberTests() + { + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672/%2f")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + _messageProducer = new CoalescingConfirmsRmqProducer(rmqConnection); + + //we need a queue to avoid a discard + new QueueFactory(rmqConnection, new ChannelName(Guid.NewGuid().ToString()), new RoutingKeys(_routingKey)) + .Create(TimeSpan.FromMilliseconds(1000)); + } + + [Fact] + public void When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once() + { + //Arrange — the subscriber fails on the first confirmation it is given + var confirmations = new List(); + _messageProducer.OnMessagePublished += result => + { + confirmations.Add(result); + if (confirmations.Count == 1) throw new InvalidOperationException("Subscriber failed"); + }; + + var sent = Enumerable.Range(0, 3) + .Select(_ => new Message( + new MessageHeader(Id.Random(), _routingKey, MessageType.MT_COMMAND), + new MessageBody("confirm me"))) + .ToArray(); + + foreach (var message in sent) + _messageProducer.Send(message); + + //Act — one frame acks delivery tags 1 to 3 + _messageProducer.Confirms.RaiseAck(deliveryTag: 3, multiple: true); + + //Assert — the failure loses no confirmation and repeats none + Assert.Equal(sent.Select(m => m.Id), confirmations.Select(c => c.MessageId)); + } + + public void Dispose() => _messageProducer.Dispose(); +} diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs index 312a2fc0d3..f2adbb34da 100644 --- a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs @@ -51,10 +51,28 @@ public static CoalescingConfirmsRmqChannel Wrap(IModel channel) // One broker frame: the client raises a single event, carrying the broker's multiple flag as-is public void RaiseAck(ulong deliveryTag, bool multiple) - => _acks?.Invoke(this, new BasicAckEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); + => Raise(_acks, new BasicAckEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); public void RaiseNack(ulong deliveryTag, bool multiple) - => _nacks?.Invoke(this, new BasicNackEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); + => Raise(_nacks, new BasicNackEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); + + // As the client does, call each subscriber in turn and swallow its exception, so one cannot stop the next + private void Raise(EventHandler? handlers, TArgs args) + { + if (handlers is null) return; + + foreach (var handler in handlers.GetInvocationList()) + { + try + { + ((EventHandler)handler)(this, args); + } + catch (Exception) + { + // the client reports a subscriber's exception through CallbackException and carries on + } + } + } protected override object? Invoke(MethodInfo? targetMethod, object?[]? args) { From 229908c4ebda88383255290bc3ff54873c1e0371 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:03:13 +0100 Subject: [PATCH 28/36] fix: a reset RMQ.Sync channel does not confirm messages pending on the old one After an IOException the producer resets its connection, and the next channel numbers its delivery tags from 1 again. Pending confirmations from the old channel were never cleared, so the new messages' entries collided with them, and an ack on the new channel (above all a coalesced one) confirmed messages the broker had never confirmed. Clear them before the reset, as RMQ.Async does. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../RmqMessageProducer.cs | 2 + ...irm_messages_pending_on_the_old_channel.cs | 62 +++++++++++++++++++ .../CoalescingConfirmsRmqChannel.cs | 8 +++ 3 files changed, 72 insertions(+) create mode 100644 tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel.cs diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index c68c98ea11..db2324fade 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -202,6 +202,8 @@ private void SendWithDelay(Message message, TimeSpan? delay, bool useSchedulerAs catch (IOException io) { Log.ErrorTalkingToSocket(s_logger, io, Connection.AmpqUri!.GetSanitizedUri()); + // The new channel numbers its delivery tags from 1 again, so its acks must not settle the old channel's + _pendingConfirmations.Clear(); ResetConnectionToBroker(); throw new ChannelFailureException("Error talking to the broker, see inner exception for details", io); } diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel.cs new file mode 100644 index 0000000000..a6234a120c --- /dev/null +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel.cs @@ -0,0 +1,62 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using Paramore.Brighter.MessagingGateway.RMQ.Sync; +using Paramore.Brighter.RMQ.Sync.Tests.TestDoubles; +using Xunit; + +namespace Paramore.Brighter.RMQ.Sync.Tests.MessagingGateway.Reactor; + +[Trait("Category", "RMQ")] +[Collection("RMQ")] +public class RmqChannelResetConfirmationTests : IDisposable +{ + private readonly RoutingKey _routingKey = new(Guid.NewGuid().ToString()); + private readonly CoalescingConfirmsRmqProducer _messageProducer; + + public RmqChannelResetConfirmationTests() + { + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672/%2f")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + _messageProducer = new CoalescingConfirmsRmqProducer(rmqConnection); + + //we need a queue to avoid a discard + new QueueFactory(rmqConnection, new ChannelName(Guid.NewGuid().ToString()), new RoutingKeys(_routingKey)) + .Create(TimeSpan.FromMilliseconds(1000)); + } + + [Fact] + public void When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel() + { + //Arrange — two messages await confirmation on the old channel when its socket fails + var confirmations = new List(); + _messageProducer.OnMessagePublished += confirmations.Add; + + _messageProducer.Send(NewMessage()); + _messageProducer.Send(NewMessage()); + + _messageProducer.Confirms.FailNextPublish(); + Assert.Throws(() => _messageProducer.Send(NewMessage())); + + // the new channel numbers its delivery tags from 1 again + var sentOnNewChannel = new[] { NewMessage(), NewMessage() }; + foreach (var message in sentOnNewChannel) + _messageProducer.Send(message); + + //Act — one frame on the new channel acks its delivery tags 1 to 2 + _messageProducer.Confirms.RaiseAck(deliveryTag: 2, multiple: true); + + //Assert — only the messages sent on the new channel are confirmed + Assert.Equal(sentOnNewChannel.Select(m => m.Id), confirmations.Select(c => c.MessageId)); + } + + private Message NewMessage() => new( + new MessageHeader(Id.Random(), _routingKey, MessageType.MT_COMMAND), + new MessageBody("confirm me")); + + public void Dispose() => _messageProducer.Dispose(); +} diff --git a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs index f2adbb34da..6b33ac52ad 100644 --- a/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs +++ b/tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/CoalescingConfirmsRmqChannel.cs @@ -26,6 +26,7 @@ THE SOFTWARE. */ #nullable enable using System; +using System.IO; using System.Reflection; using System.Runtime.ExceptionServices; using RabbitMQ.Client; @@ -40,6 +41,7 @@ public class CoalescingConfirmsRmqChannel : DispatchProxy private IModel _channel = null!; private EventHandler? _acks; private EventHandler? _nacks; + private bool _failNextPublish; public static CoalescingConfirmsRmqChannel Wrap(IModel channel) { @@ -49,6 +51,9 @@ public static CoalescingConfirmsRmqChannel Wrap(IModel channel) return coalescing; } + // The next publish fails as a dropped socket would, so the producer resets its connection + public void FailNextPublish() => _failNextPublish = true; + // One broker frame: the client raises a single event, carrying the broker's multiple flag as-is public void RaiseAck(ulong deliveryTag, bool multiple) => Raise(_acks, new BasicAckEventArgs { DeliveryTag = deliveryTag, Multiple = multiple }); @@ -85,6 +90,9 @@ private void Raise(EventHandler? handlers, TArgs args) case "add_BasicNacks": _nacks += (EventHandler)args![0]!; return null; + case "BasicPublish" when _failNextPublish: + _failNextPublish = false; + throw new IOException("Injected socket failure."); } try From bb7139df8c6c318b187fc6bd54cc14b859f3a80d Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:03:14 +0100 Subject: [PATCH 29/36] docs: bugfix record for coalesced RMQ.Sync publisher confirms Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../0053-rmq-sync-multiple-confirm/bugfix.md | 294 ++++++++++++++++++ 1 file changed, 294 insertions(+) create mode 100644 bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md diff --git a/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md new file mode 100644 index 0000000000..ccff26b024 --- /dev/null +++ b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md @@ -0,0 +1,294 @@ +# Bugfix: RMQ.Sync producer ignores coalesced (multiple=true) publisher confirms + +**Linked Issue**: none yet (found by CI on PR #4562; related to #4560) +**Status**: Tested + +## Symptom + +**Observed** + +- With publisher confirms on, the RMQ.Sync `RmqMessageProducer` sometimes raises no publish + confirmation for some of the messages it sends. Neither `OnMessagePublished` nor + `OnMessagePublishedAsync` fires, even though the broker accepted the messages. +- CI on PR #4562 caught it (rabbitmq-sync-ci, run 37918253244). The failing test was + `RmqConfirmationBatchingTests.When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool`. + - It failed with "Not every message was confirmed" after the full 30 s wait on one target + framework. + - It passed in about 1 s on the other framework in the same run, so the failure is intermittent. +- In production, an outbox that relies on publish confirmation would never mark those messages + dispatched. The sweeper would send them again, so consumers would see duplicates. + +**Expected** + +Every message the broker acks or nacks raises exactly one confirmation. That includes each message +covered by a coalesced ack or nack (`multiple=true`, meaning "every delivery tag up to and including +`DeliveryTag`"). + +**Reproduction** (as in the test) + +1. Create a Sync `RmqMessageProducer` with a bound queue. +2. Subscribe to `OnMessagePublishedAsync`. +3. Call `Send` 50 times back-to-back, without waiting for confirms between sends. +4. Wait for 50 confirmations. + +The test passes or fails depending on whether the broker happens to coalesce acks. See +`tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool.cs:53-66`. + +## Suspected Location + +The handlers and the dictionary in the Sync producer: + +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:268-276`: `OnPublishFailed` + looks up and removes only `_pendingConfirmations[e.DeliveryTag]`. It never reads `e.Multiple`. +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:278-286`: + `OnPublishSucceeded` has the same problem for acks. +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:58`: `_pendingConfirmations` + is a `ConcurrentDictionary` keyed by delivery tag. + - The two handlers above are the only places that remove entries. + - Nothing clears it on dispose or on connection reset. +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:178`: each `Send` adds an + entry keyed by `Channel.NextPublishSeqNo`, so each message gets its own tag. A coalesced ack + therefore has to settle a range of entries. +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:168-170`: every `Send` + subscribes `BasicAcks`/`BasicNacks` again and calls `ConfirmSelect()` again. + - This is a handler leak. + - It does not lose confirmations: the second and later invocations find no entry to settle. +- `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs:247-254`: `WaitForConfirms` + is called only from `Dispose`. + - `Send` never waits per message, so many publishes can be unconfirmed at once. That is what lets + the broker coalesce acks. + +Reference implementation: + +- `src/Paramore.Brighter.MessagingGateway.RMQ.Async/RmqMessageProducer.cs:535-537` pass `e.Multiple` + to `SettleConfirmationsAsync`. +- `RemovePendingConfirmations` (`:473-512`) removes every pending tag that `IsConfirmedBy` + (`:516-517`) matches: `multiple ? pending <= tag : pending == tag`. +- The Async producer also clears its dictionary on channel reset (`:445`). + +Elsewhere in RMQ.Sync: + +- No other code in `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/` handles publisher confirms. + The `DeliveryTag` uses in `RmqMessageConsumer.cs` and `PullConsumer.cs` are consumer-side acks. +- RMQ.Sync uses RabbitMQ.Client 6.x, which passes the broker's `multiple` flag straight to + `BasicAcks`/`BasicNacks` subscribers. + - The client's own `WaitForConfirms` bookkeeping does honour `multiple`. + - That is why `Dispose` does not hang, even though Brighter's dictionary still holds the entries + that were never settled. + +## Root-Cause Hypothesis + +**Hypothesis** + +- RabbitMQ sometimes acks or nacks several publishes with one frame: `multiple=true` and + `DeliveryTag = N`. +- When that happens, the Sync producer settles only the entry for tag `N` (`RmqMessageProducer.cs:270` + and `:280`). +- Every pending entry with a tag `< N` that the frame covered stays in `_pendingConfirmations` + forever, and no `PublishConfirmationResult` is ever raised for it. +- RabbitMQ decides whether to coalesce confirms based on timing and load, so the failure is + intermittent. + +**How to falsify it** + +1. Capture `e.Multiple` and `e.DeliveryTag` in the Sync handlers during a 50-message burst. + - A failing run should show at least one `Multiple == true` event. + - The number of entries left in `_pendingConfirmations` should equal the number of missing + confirmations. +2. Raise `BasicAcks` with `Multiple=true` deterministically, with no broker timing involved. The + current code will raise one confirmation where it should raise several. +3. If failing runs show no `Multiple=true` frames, the hypothesis is refuted. + +**Alternatives considered** + +- *`BatchedCallbackQueue` drops callbacks.* `Enqueue` drops a callback only after `Complete()`, and + `Complete()` runs only in `Dispose`, after the test's wait. `Drain` runs every item it takes. This + is unlikely, but `/bugfix:confirm` should rule it out. +- *The 30 s timeout is too short.* The passing framework finished in about 1 s. A complete run would + not need anywhere near 30 s, so lost confirmations fit the evidence better than slowness. +- *A new channel resets delivery tags.* If a new channel were created mid-burst, `NextPublishSeqNo` + would restart at 1. + - Old entries would then be orphaned, and `TryAdd` silently ignores duplicate keys. + - That needs an `IOException` in `Send`, which would fail the test differently. + - This is a real but secondary weakness: unlike the Async producer, the Sync producer never clears + `_pendingConfirmations` on reset. +- *The handlers are subscribed once per `Send`.* This only adds no-op invocations; it doesn't lose + confirmations. +- *This branch introduced the bug.* No: origin/master has the same `e.DeliveryTag`-only lookups. The + branch changed only the callback queue, which may change how often the broker coalesces acks. + +**Suggested direction** (**UNVERIFIED — to be proven or refuted in /bugfix:confirm**) + +- When `Multiple` is true, settle every pending tag `<= DeliveryTag`. +- Mirror the Async producer's `RemovePendingConfirmations` and `IsConfirmedBy`. +- Raise the confirmations in ascending tag order. + +**Scope-note candidates** (not part of the root cause) + +- The comments at `RmqMessageProducer.cs:62-63` and `:298` are out of date. They say callbacks run + "one at a time, in ack order", but they now run in batches of up to 32. +- Every `Send` subscribes the handlers and calls `ConfirmSelect()` again (`:168-170`). +- `_pendingConfirmations` is never cleared on channel reset or dispose. + +## Confirmed Root Cause + +**Verdict: CONFIRMED.** + +- **The client passes coalesced acks through unchanged.** RMQ.Sync uses RabbitMQ.Client 6.8.1. + - One broker `basic.ack`/`basic.nack` frame produces exactly one `BasicAcks`/`BasicNacks` event. + - The broker's `multiple` flag arrives unchanged in `e.Multiple`; the client does not expand it + into one event per tag. +- **Brighter settles only the exact tag.** `RmqMessageProducer.OnPublishSucceeded` and + `OnPublishFailed` (`RmqMessageProducer.cs:268-286`) never read `e.Multiple`. They look up and + remove only the key `e.DeliveryTag`. +- **The covered messages are lost.** For a coalesced frame with `DeliveryTag = N`, every pending + entry with a lower tag stays in `_pendingConfirmations` forever. No `OnMessagePublished` or + `OnMessagePublishedAsync` is raised for those messages. +- **Nothing visible goes wrong.** The client's own `WaitForConfirms` bookkeeping does honour + `multiple`, so `Dispose` returns normally. +- **It is intermittent.** Whether the broker coalesces acks depends on timing. + +## Evidence + +- [x] **Code-trace.** The client was checked by disassembling the shipped 6.8.1 DLL; no source + package is on disk. + 1. **Client version.** The RMQ.Sync csproj uses `VersionOverride="$(RabbitMQClientV6)"`, and + `Directory.Packages.props:13` sets it to `6.8.1`. + 2. **The event args expose `Multiple`.** See + `~/.nuget/packages/rabbitmq.client/6.8.1/lib/netstandard2.0/RabbitMQ.Client.xml:3100` + (`BasicAckEventArgs.Multiple`) and `:3149` (`BasicNackEventArgs.Multiple`). Both types have + public constructors and public setters. + 3. **There is no per-tag expansion.** `ModelBase.HandleBasicAck(deliveryTag, multiple)`: + - builds one `BasicAckEventArgs` with `DeliveryTag` and `Multiple` set; + - invokes each subscriber in `GetInvocationList()` once, with each call wrapped in a try/catch + that routes to `OnCallbackException`; + - then calls `HandleAckNack(deliveryTag, multiple, false)`. + + `HandleBasicNack` does the same. + 4. **The client's own bookkeeping honours `multiple`.** `ModelBase.HandleAckNack` removes every + pending tag `< deliveryTag`, then tag N itself, and signals the countdown. This is why + `WaitForConfirms` in `Dispose` (`RmqMessageProducer.cs:251`) never times out. + 5. **Each message gets its own tag.** `RmqMessageProducer.cs:178` calls `TryAdd` with + `Channel.NextPublishSeqNo` before each publish. Calling `ConfirmSelect` again does not reset the + sequence number on a live channel. + 6. **Settlement is by exact tag only.** The nack handler keys only on `e.DeliveryTag` at `:270` + and `:273`; the ack handler does the same at `:280` and `:283`. Nothing else removes entries + from `_pendingConfirmations`. + 7. **The test's burst is what triggers coalescing.** `Send` never waits for confirms per message. + The test sends 50 messages back-to-back (`...overlapping_batches_off_the_thread_pool.cs:53-63`), + so any coalesced frame leaves the `CountdownEvent` short. + 8. **The reference implementation is correct.** In RMQ.Async (`RmqMessageProducer.cs:535-537`), + the handlers pass `e.Multiple` on, and `IsConfirmedBy` (`:516-517`) treats a multiple ack as + covering every tag up to and including `DeliveryTag`. +- **Alternatives ruled out** + - *`BatchedCallbackQueue` drops callbacks.* No. The queue drops only after `Complete()`, which runs + only in `Dispose`. `Drain` runs every item it takes. In any case, the missing confirmations are + never enqueued: they are lost upstream of the queue. + - *The 30 s timeout is too short.* No. The other framework finished in about 1 s, and orphaned + entries never settle however long you wait. + - *A new channel resets delivery tags.* Not this failure. That path rethrows a + `ChannelFailureException` out of `Send`, which would fail the test differently. It is a real + secondary weakness (see Scope Notes). + - *Duplicate handler subscriptions.* No. The first invocation's `TryRemove` wins, and the later + invocations are no-ops. + - *This branch introduced the bug.* No. The same code is on origin/master. +- [ ] **Red repro:** not run in Confirm. A deterministic one is feasible; see Regression Test. + +**Suggested-fix assessment: PARTIAL.** Settling every tag up to and including `DeliveryTag` when +`Multiple` is set is right, but the fix also needs to: + +1. **Claim each entry with `TryRemove` before raising it.** Today the code raises, then removes. + - If a sync `OnMessagePublished` subscriber throws, the client swallows the exception and the + remove is skipped. + - The handler is subscribed once per `Send`, so the next duplicate invocation finds the entry + still there and raises it again. + - In a range loop, the same throw would also abort settlement of the rest of the range. + - So raise each confirmation only after its `TryRemove` succeeds, and isolate each raise in a + try/catch that logs. This also makes the fix safe when the handler is subscribed many times. +2. **Settle in ascending tag order, sorted explicitly.** `ConcurrentDictionary` enumeration order is + unspecified. The Async producer doesn't sort either. +3. **Concurrency needs no lock.** `Send` adds each entry before it publishes, so any entry added + during enumeration has a tag above anything the broker could have acked. +4. **Log each settled message,** as today. + +## Scope Notes + +- **`_pendingConfirmations` is never cleared on channel reset or dispose.** The user directed that + this be raised as a separate issue at the end. But range settlement makes this weakness worse: + - After `ResetConnectionToBroker` (`:203`), the new channel restarts `NextPublishSeqNo` at 1. + - `TryAdd` (`:178`) then silently fails on a colliding stale key. The new message is never tracked. + - Today, one wrong message is confirmed per collision. + - With range settlement, a coalesced ack on the new channel would confirm **every** stale entry up + to and including N. Messages that were never confirmed would be marked dispatched. + - The RMQ.Async producer clears on reset (`Async/RmqMessageProducer.cs:230`, `:441`). + - The minimal parity fix is `_pendingConfirmations.Clear()` in the Sync `catch (IOException)` + before `ResetConnectionToBroker()`. + - Not verified: RabbitMQ.Client auto-recovery (on by default) could also restart tags without the + `IOException` path running. + - **Decided at the Confirm gate (2026-10-09): clear in the `IOException` path as part of this fix**, + with its own regression test. The separate issue still covers dispose and auto-recovery. +- **Every `Send` subscribes the handlers again and calls `ConfirmSelect()` again (`:168-170`).** + Not related to this root cause. **Raise it as a separate GitHub issue at the end**, per the user's + direction. + - It causes unbounded invocation-list growth. + - Every `Send` makes a synchronous `confirm.select` round-trip. + - It amplifies the raise-before-remove duplicate described above. + - The fix here only has to be safe under N subscriptions, which item 1 above covers. +- **Out-of-date comments** at `RmqMessageProducer.cs:62-63` and `:297-298`. They come from the #4560 + batching change on this branch, not from this bug. Correct them separately, as a tidy-first or docs + commit. +- **Other transports:** none settle publisher confirms by exact tag; RMQ.Async already handles + `Multiple`. + +## Regression Test + +Four tests, each written red-first and approved before its fix, in +`tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/`: + +| Test | Red today | Commit | +| --- | --- | --- | +| `When_the_broker_coalesces_acks_should_confirm_every_message_the_ack_covers.cs` | 1 of 3 confirmations, for the third message only | `6b546641b` | +| `When_the_broker_coalesces_nacks_should_fail_every_message_the_nack_covers.cs` | 1 of 3 failed confirmations | `0af74dcf1` | +| `When_a_confirmation_subscriber_throws_should_still_confirm_every_message_once.cs` | the first message confirmed twice | `dda706a47` | +| `When_the_channel_resets_should_not_confirm_messages_pending_on_the_old_channel.cs` | the new channel's ack confirms the two old-channel messages | `229908c4e` | + +Test doubles in `tests/Paramore.Brighter.RMQ.Sync.Tests/TestDoubles/`: + +- **`CoalescingConfirmsRmqChannel`**: a `DispatchProxy` over `IModel`. + - It forwards every call to the real channel except the `BasicAcks`/`BasicNacks` subscriptions. + It keeps those, so the broker's real per-tag acks never reach the producer. + - `RaiseAck`/`RaiseNack` simulate one broker frame. Each subscriber is called separately and its + exception swallowed, as RabbitMQ.Client 6.8.1's `HandleBasicAck` does. + - `FailNextPublish()` makes the next `BasicPublish` throw an `IOException`. +- **`CoalescingConfirmsRmqProducer`**: overrides `ConnectToBroker` to wrap each new channel in the + proxy, and exposes it as `Confirms`. + +The original CI failure, +`RmqConfirmationBatchingTests.When_confirming_many_messages_should_raise_confirmations_in_overlapping_batches_off_the_thread_pool`, +depends on broker timing. These four tests reproduce its cause every time. + +**Recommended approach** (from Confirm). A deterministic, broker-backed test in +`tests/Paramore.Brighter.RMQ.Sync.Tests/MessagingGateway/Reactor/`, driven through the public `Send` +and `OnMessagePublished` API. + +It needs two test doubles, one class per file, following the precedent of `FaultingRmqChannel` and +`CleanupFailureRmqProducer`: + +- **A channel proxy** (a `DispatchProxy` over `IModel`). + - It forwards every call to the real channel except `add_BasicAcks` and `add_BasicNacks`. It + captures those handlers and does **not** forward them, so real per-tag acks cannot race in. + - It exposes `RaiseAck(tag, multiple)` and `RaiseNack(tag, multiple)`. Each invokes the captured + handlers once, simulating one broker frame. +- **A producer subclass** whose `ConnectToBroker` wraps `Channel` in that proxy. + +The test: + +- **Arrange:** subscribe the sync `OnMessagePublished`, collecting results into a list. +- **Act:** `Send` 3 messages, then call `RaiseAck(3, multiple: true)`. +- **Assert:** exactly 3 results, for the 3 sent ids in send order, all successful. + - Today it fails with 1 result. + - The "exactly 3" assertion also guards against duplicates from the 3 handler subscriptions. +- **Nack variant:** the same with `RaiseNack(3, multiple: true)`, expecting 3 failed results. + +## Fix +_(left blank — filled by /bugfix:fix)_ From fa786716cdfebc783ea82bcc997677ba137a311b Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:32:02 +0100 Subject: [PATCH 30/36] docs: RMQ.Sync confirmation callbacks run in batches, not one at a time The comments predate BatchedCallbackQueue, which starts callbacks in ack order but runs up to 32 at once. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../RmqMessageProducer.cs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs index db2324fade..82b70db4c1 100644 --- a/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs @@ -62,7 +62,7 @@ public partial class RmqMessageProducer : RmqMessageGateway, IAmAMessageProducer private readonly int _waitForConfirmsTimeOutInMilliseconds; private event Func? _onMessagePublishedAsync; // The ack/nack handlers run on the client's connection loop, which must never block on a - // subscriber, so awaited callbacks run one at a time, in ack order, on the queue's single worker + // subscriber, so awaited callbacks run on the queue's single worker, in batches started in ack order // (never one thread-pool item per ack). The queue lets Dispose wait for them — including the // awaited Outbox mark-dispatched — after WaitForConfirms has drained the broker acks themselves. private readonly BatchedCallbackQueue _confirmationCallbacks; @@ -313,7 +313,7 @@ private void RaisePublishConfirmation(PublishConfirmationResult result) return; // Awaited callbacks must not block the connection loop (WaitForConfirms depends on it to - // process acks), so they run on the confirmation queue's worker, one at a time, in ack order. + // process acks), so they run on the confirmation queue's worker, in batches started in ack order. _confirmationCallbacks.Enqueue(async () => { try From a19787bd2b99aefd832a3791c49649e3dd860d54 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:32:02 +0100 Subject: [PATCH 31/36] docs: record verification of the coalesced RMQ.Sync confirms fix Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../0053-rmq-sync-multiple-confirm/bugfix.md | 39 ++++++++++++++++++- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md index ccff26b024..c44625ea38 100644 --- a/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md +++ b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md @@ -1,7 +1,7 @@ # Bugfix: RMQ.Sync producer ignores coalesced (multiple=true) publisher confirms **Linked Issue**: none yet (found by CI on PR #4562; related to #4560) -**Status**: Tested +**Status**: Verified ## Symptom @@ -291,4 +291,39 @@ The test: - **Nack variant:** the same with `RaiseNack(3, multiple: true)`, expecting 3 failed results. ## Fix -_(left blank — filled by /bugfix:fix)_ + +Each fix was made in the GREEN step of its own regression test, so `/bugfix:fix` had nothing left to +do. All the changes are in `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqMessageProducer.cs`. + +- **Range settlement** (`6b546641b`, `0af74dcf1`) + - `OnPublishSucceeded` and `OnPublishFailed` delegate to `SettleConfirmations(deliveryTag, multiple, + success)`. + - `ConfirmedDeliveryTags` returns every pending tag up to and including `DeliveryTag` when `multiple` + is set, otherwise just that tag. The tags are sorted ascending and copied before the loop. +- **Claim, then raise** (`dda706a47`) + - Each entry is removed with `TryRemove` before its confirmation is raised, so each tag is raised + at most once. + - An exception from a sync `OnMessagePublished` subscriber is caught and logged with + `ConfirmationCallbackFault`, so the rest of the range still settles. +- **Clear on reset** (`229908c4e`): `_pendingConfirmations.Clear()` before `ResetConnectionToBroker()` + in the `IOException` path, as RMQ.Async does. + +## Verification (2026-10-09) + +- **Full suite.** `Paramore.Brighter.RMQ.Sync.Tests` with the CI filter + (`Fragile!=CI&Requires!=Docker-mTLS&Category!=RMQNativeDelay`): **143 passed, 1 skipped, 0 failed** + on both net9.0 and net10.0. +- **Repetition.** The 4 regression tests and the original CI failure + (`RmqConfirmationBatchingTests...overlapping_batches_off_the_thread_pool`) were run 10 times on both + frameworks: **20 of 20 runs green**. +- **Not counted.** Without the CI filter, 9 mutual-TLS acceptance tests fail locally. They need a TLS + broker on port 5671, which this machine doesn't have, and CI excludes them. + +## Follow-ups (to raise as issues, per the user) + +- RMQ.Sync `Send` subscribes `BasicAcks`/`BasicNacks` again and calls `ConfirmSelect()` again on every + call (`RmqMessageProducer.cs:168-170`). This is not the cause of this bug, but it means: + - the invocation list grows without bound; + - every `Send` makes a synchronous `confirm.select` round trip. +- `_pendingConfirmations` is still not cleared on `Dispose`, or when the client's automatic recovery + replaces the channel. Auto-recovery is on by default, and the recovery case has not been verified. From e765ba02c0cc0c3b6c52af62205d0d2f7aaf3e15 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:33:12 +0100 Subject: [PATCH 32/36] docs: link the coalesced-confirm follow-up issues #4567 and #4568 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md index c44625ea38..117d02c123 100644 --- a/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md +++ b/bugfixes/0053-rmq-sync-multiple-confirm/bugfix.md @@ -319,11 +319,13 @@ do. All the changes are in `src/Paramore.Brighter.MessagingGateway.RMQ.Sync/RmqM - **Not counted.** Without the CI filter, 9 mutual-TLS acceptance tests fail locally. They need a TLS broker on port 5671, which this machine doesn't have, and CI excludes them. -## Follow-ups (to raise as issues, per the user) +## Follow-ups (raised 2026-10-09, assigned to the maintainer, labelled `Bug` and `1 - Up Next`) -- RMQ.Sync `Send` subscribes `BasicAcks`/`BasicNacks` again and calls `ConfirmSelect()` again on every +- **#4567**: RMQ.Sync `Send` subscribes `BasicAcks`/`BasicNacks` again and calls `ConfirmSelect()` again on every call (`RmqMessageProducer.cs:168-170`). This is not the cause of this bug, but it means: - the invocation list grows without bound; - every `Send` makes a synchronous `confirm.select` round trip. -- `_pendingConfirmations` is still not cleared on `Dispose`, or when the client's automatic recovery - replaces the channel. Auto-recovery is on by default, and the recovery case has not been verified. +- **#4568**: `_pendingConfirmations` is not cleared when the client's automatic recovery replaces the + channel. Auto-recovery is on by default in 6.8.1; that tags restart on recovery has not been + verified. Not clearing on `Dispose` turned out not to be a defect: the entries are collected with + the producer, and those messages were correctly never confirmed. From c97d7935e3095dace016154661aa22773dc2152d Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:33:26 +0100 Subject: [PATCH 33/36] docs: record the Confirm gate approval for bugfix 0053 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- bugfixes/0053-rmq-sync-multiple-confirm/.confirm-approved | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 bugfixes/0053-rmq-sync-multiple-confirm/.confirm-approved diff --git a/bugfixes/0053-rmq-sync-multiple-confirm/.confirm-approved b/bugfixes/0053-rmq-sync-multiple-confirm/.confirm-approved new file mode 100644 index 0000000000..e69de29bb2 From 9dca4d5b3496dba1efbb81e35b8fe21523b6567a Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 16:41:25 +0100 Subject: [PATCH 34/36] docs: release note for coalesced RMQ.Sync publisher confirms Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- release_notes.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/release_notes.md b/release_notes.md index ff5433f733..9f593e7c57 100644 --- a/release_notes.md +++ b/release_notes.md @@ -72,6 +72,35 @@ CPU. That can matter on a pod limited to 1–2 CPUs. If you run many publication CPU throttling (`container_cpu_cfs_throttled_periods_total` on Kubernetes), the sweeper tick-lag metric and the confirmation queue depth. +### RabbitMQ (`RMQ.Sync`): every message a coalesced publisher confirm covers is now confirmed (#4562) + +RabbitMQ can confirm several publishes with one ack or nack. The ack carries `multiple=true` and +covers every delivery tag up to and including its own. The `RMQ.Sync` producer settled only that one +tag. The other messages the ack covered never raised `OnMessagePublished` or the awaited +confirmation event, so an outbox never marked them dispatched and the sweeper sent them again. The +broker coalesces confirms under load, so this showed up as occasional duplicates. `RMQ.Async` already +handled it. + +- **A coalesced ack or nack settles every message it covers,** once each, in the order they were + sent. +- **A confirmation is raised at most once.** A subscriber that throws no longer stops the rest of a + coalesced ack from settling, and no longer causes the same confirmation to be raised again. +- **A connection reset forgets confirmations still pending on the old channel.** When a send fails + with an `IOException`, the producer resets its connection, and the new channel numbers its delivery + tags from 1 again. The new channel's acks used to settle the old channel's pending entries, marking + messages dispatched that the broker had never confirmed. Those messages are now left for the + sweeper to send again, as `RMQ.Async` already does. + +**Known gap:** when RabbitMQ.Client's automatic recovery (on by default) rebuilds the channel instead, +nothing yet forgets the old channel's pending confirmations. #4568 tracks it. + +#### Behaviour change: a throwing `OnMessagePublished` subscriber is logged, not rethrown + +On `RMQ.Sync`, an exception from a subscriber to the synchronous `OnMessagePublished` event is now +caught and logged as a Warning ("Confirmation callback for message {MessageId} faulted"), as faults +in awaited subscribers already were. Before, it went to RabbitMQ.Client, which also swallowed it, so +no caller could have relied on seeing it. + ### Dispatcher shutdown drains consumers still being created (#4541) The Dispatcher now tracks accepted consumer operations until their channels have either been From e373af9deb3fba6119b835a754e92aa06a7bbd2b Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 17:03:06 +0100 Subject: [PATCH 35/36] docs: ADR 0081 says a hung sweep goes silent, so alert on missing completed sweeps Sweeps now run one after another on the sweeper's thread, so a hung sweep records no tick lag, duration or outcome until it ends. lock_unavailable shows only an overrun on another instance sharing the lock. The stall alert is no increase in completed sweeps across instances; an alert on any outcome would never fire while the lock holder hangs. Also records the sync-over-async deadlock risk on the sweeper's thread. Raised in the re-review of #4562. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- docs/adr/0081-timed-outbox-sweeper-health-metrics.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md index f34722b635..77c99967a8 100644 --- a/docs/adr/0081-timed-outbox-sweeper-health-metrics.md +++ b/docs/adr/0081-timed-outbox-sweeper-health-metrics.md @@ -220,7 +220,10 @@ the values with a `MeterListener`. The test then needs no OpenTelemetry pipeline #### Why three outcomes and no more? These three outcomes are the ones that need different responses. `failed` means read the logs. -`lock_unavailable` means a sweep is overrunning or hung. `completed` is the baseline. A sweep that +`lock_unavailable` means a sweep is overrunning or hung on **another instance** that shares the +distributed lock. It cannot show a hang in the same process: the sweeper runs its sweeps one after +another on one thread, so while a sweep is hung, no later tick starts and nothing is recorded at all +(see Risks and Mitigations). `completed` is the baseline. A sweep that the mediator skipped internally is still `completed` from the sweeper's point of view. That case can only happen with a second caller of `ClearOutstandingFromOutboxAsync`. @@ -354,6 +357,8 @@ never the thread pool. | The names conflict with the outbox metrics in #4484 | Both use the `paramore.brighter.` prefix, and this ADR uses the `outbox_sweeper` segment. Neither counts messages twice. The #4484 review should cross-reference this ADR | | A meter failure breaks sweeping | The role's contract says it never throws. The sweeper records outside the lock and inside its catch-all. A test checks that a throwing meter does not stop sweeps | | Tick lag is misread when the sweeper is stopped | The sweeper records lag only for sweeps it starts, never for the wait that `StopAsync` cancels | +| A sweep hangs and the metrics go silent. Sweeps run one after another on the sweeper's thread, so while one is hung no tick starts: nothing records `tick.lag`, `sweep.duration` or `sweeps`, and tick lag is recorded only when a sweep finally starts. `lock_unavailable` appears only on other instances sharing the lock, so with one instance or the in-memory lock a hang records nothing | The primary stall alert is **no increase in `completed` sweeps**, summed across instances, over several intervals, paired with `absent()` where the backend drops stale series. It must be `completed`, not any outcome: while the lock holder is hung, the other instances keep recording `lock_unavailable`. Treat `lock_unavailable` as a sign of an overrun on another instance, not as the stall alert | +| A sweep deadlocks on the sweeper's thread. Since the sweep runs in a `BrighterAsyncContext`, code in its path that blocks on a task (sync-over-async), such as a custom `IDistributedLock`, producer or outbox calling `.GetAwaiter().GetResult()` on an async method, can deadlock there. Before #4560 the same code only tied up a pool thread | Brighter's own sweep path does not block on tasks: its `GetAwaiter().GetResult()` sites are in sync-only methods. A third-party implementation that does is caught by the stall alert above | ## Alternatives Considered From 26cb7b9d857b4497c27616e575fc44cb8eac8b70 Mon Sep 17 00:00:00 2001 From: Ian Cooper Date: Fri, 9 Oct 2026 17:03:06 +0100 Subject: [PATCH 36/36] docs: Kafka confirmation callbacks run in batches, not one at a time Raised in the re-review of #4562; the RMQ.Sync comments were corrected in fa786716c. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SpjnqGWbARr3Z3ofGc6C2x --- .../KafkaMessageProducer.cs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs index f39006f576..f3bd58e30d 100644 --- a/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs +++ b/src/Paramore.Brighter.MessagingGateway.Kafka/KafkaMessageProducer.cs @@ -77,7 +77,7 @@ event Func ISupportPublishConfirmationAsync.OnM private int _disposed; private readonly InstrumentationOptions _instrumentation; private event Func? _onMessagePublishedAsync; - // Confirmation raises run one at a time, in delivery order, on a single worker (never on Confluent's + // Confirmation raises run on a single worker, in batches started in delivery order (never on Confluent's // poll thread, and never one thread-pool item per report); the queue lets Dispose wait for those // callbacks — including the awaited Outbox mark-dispatched — after Flush() has drained the delivery // reports themselves. @@ -458,7 +458,7 @@ private void PublishResults(PersistenceStatus status, Headers headers, RoutingKe private void RaisePublishConfirmation(PublishConfirmationResult result) { // Raise on the confirmation queue's worker so we never block Confluent's delivery-report handler, - // and confirmations reach subscribers one at a time, in delivery order. Wrap the invoke so a + // and confirmations reach subscribers in batches started in delivery order. Wrap the invoke so a // faulting subscriber is logged. Brighter's own mediator callback is already self-contained; // this guards any other subscriber and the queue's worker. _confirmationCallbacks.Enqueue(async () =>