Repository navigation
mcpx: /v1/events replay drops everything past 256 events without a gap marker, though reconnection is documented as lossless #1914
Description
Relocated from dezren39/darwin-nix#193 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #177
Summary
GET /v1/events promises lossless reconnection: "Reconnect with Last-Event-ID ... or ?since=N and everything
after N is replayed", and a reconnect older than the retained history "gets an in-band gap event"
(docs/story.md:2203-2214). A replay of more than 256 matching events keeps the oldest 256 and silently drops the
rest, with no gap event. The retained history is 1024 by default (events.history), so a ?since=0 against a busy
daemon can lose up to three quarters of what it asked for, and the client is told nothing.
Found by reading the code in the 2026-09-30 issues pass (worker "ext", confirmed by the architect). Not yet
reproduced against a running daemon; the first step is the reproduction.
Current state (05c78b2)
Bus.SubscribeFrom (internal/events/events.go:176-199):
- makes the subscription channel with a buffer of 256 (
:177); - holds the bus lock for the whole replay (
:178-179), and the subscription has not been returned yet, so nothing
can drain the channel while the replay fills it; - replays each retained event into the channel with a non-blocking send; when the buffer is full it increments
sub.Droppedand moves on (:190-194); - sets
gaponly whensincepredates the retained history (:185-187), never for drops.
Dropped has no reader outside internal/events (:147, :193, :230), and the daemon's handler
(internal/daemon/server.go:826) sees only gap. The same counter is raised for a slow live reader (:230), which
is the documented trade-off ("one slow reader cannot stall the daemon"); the replay case is different because it
happens before the reader has had any chance to read.
Approaches
A -- enlarge the buffer to events.history. One line; replays of up to the whole history fit. Costs memory per
subscriber (1024 events each) and still loses silently if a live burst arrives before the reader drains; the promise
remains unguarded.
B -- replay outside the lock, from a snapshot, with blocking sends. Copy the matching history under the lock,
register the subscription, then feed the snapshot to the reader at its own pace before live events. Lossless for the
replay; live-burst loss still silent.
C -- recommended: B, plus make every loss visible with the gap contract #6 owns. When Dropped rises -- during
replay or live -- emit the in-band gap {since, latest} marker #6 defines (#181 collision 13), so a consumer always
knows it missed something and can resynchronise with ?since=. Add the test that would have caught this: replay more
than the buffer and assert either every event or a gap. Recommended because "lossless" is a documented promise and
the only honest versions of it are "you got everything" or "you were told you did not".
Open questions
- Should
/v1/eventsreportDroppedin its stream or only as thegapmarker? - Does the MCP notification path (
internal/pool/pool.go:411) share the bus and the same failure?
Ease · Impact · Related
Ease S (A, B) / M (C, with #6's gap contract) · Impact med -- silent loss on the one path documented as
lossless; #81's hooks and #86's browser surface consume this stream.
Related: #177 (class), #6 (owns gap semantics, collision 13), #81 (hooks over the bus), #86 (browser reconnect,
Last-Event-ID), #87 (lifetime), #129 (guard mechanics).
Security/auth deferred; see tracker.
Acceptance criteria
- Reproduction first: a replay through
Bus.SubscribeFromwith more than 256 matching retained events demonstrates the silent drop on current code (the issue is code-read, not yet reproduced against a running daemon). - After the fix, a replay larger than the channel buffer loses nothing silently: either all retained events after
sinceare delivered, or the client receives an in-bandgapevent naming the loss — matching thedocs/story.mdreconnection promise. - The
Droppedcounter has a reader that surfaces drops (gap event, metric, or log), including drops after the subscription is live. - A
?since=0replay against a full 1024-event history is covered by a test; tests pass in CI.
Activity
developing-today commentedon Oct 28, 2026
CODE-840
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#193 (verbatim, oldest first):
@dezren39 · 2026-10-19T11:49:30Z
Partial fix landed — reopening for the remainder (the auto-close came from the PR body saying "does not close" this issue; GitHub's keyword parser matched the verb anyway).
Done (replay half, approach B's losslessness): PR #460 merged here (a047f67d) + the identical mcpx port in developing-today/code PR #351 merged (0111161), as one cross-repo set.
SubscribeFrom(since=0, replay=true)→ received 256,Dropped=44,gap=false— silent loss, exactly as filed.SubscribeFromnow collects the matching replay under the lock, sizes the subscription channel for the whole replay plus the usual live buffer (bounded by the retained history), and setsgapif a replay send ever still cannot land.TestAReplayLargerThanTheSubscriberBufferMissesNothing(300-event replay, all in order,Dropped=0): fails pre-fix, passes post-fix; events/daemon/pool suites green locally in both repos; both PRs' CI fully green incl. all conformance legs and server-all.Remaining (approach C's second half): drops of live events for a slow reader are still only counted on
Subscription.Dropped, not emitted as an in-band mid-streamgap {since, latest}marker. That visibility depends on the gap contract owned by #6, so this issue stays open for it.@dezren39 · 2026-10-28T05:41:53Z
Intake review — fleet monitor
Liveness: Verified still live at intake: the issue is open and no merged PR closes it. Evidence caveat (same as the rest of this register batch): the filer's evidence was verified against
05c78b2(2026-09-30) — re-verify against current main before fixing; if a row has since been fixed, say so on the issue and narrow the scope instead of redoing it.Related: Parent tracker named in the issue body; sibling register-batch issues in dezren39/nix (the #209–#220 batch was enriched in the previous monitor run). mcpx also lives in developing-today/code — check for a code-side twin before assuming a nix-only fix.
Notes / risks: Flagged honestly in the issue: found by code reading, not yet reproduced against a running daemon. The first acceptance criterion is the reproduction; if the drop cannot be reproduced, close with that evidence instead of fixing blind.
Suggested approach: Re-verify the cited code paths on current main, land failing tests for the acceptance criteria first, then the smallest fix that satisfies them.
🤖 Agent note · state: filed
dezren39 commentedon Oct 29, 2026
Cross-reference from triage of #1850 (queue task 2740, 2026-10-29): #1850's 'Gap semantics' section describes this same defect (replay past the 256-slot channel silently drops events, contradicting the lossless-reconnection promise) as the design question its event store would inherit. The defect itself is tracked here (#1914, and #1679 for the since=0/malformed-cursor face) — #1850 remains open only for the event-store design/implementation, which should consume whatever gap semantics is decided here rather than inventing its own.
89 remaining items
Merge pull request #2564 from developing-today/queue/3192-1914-events…
dezren39 commentedon Oct 30, 2026
Agent note — done (queue task 3192)
basic: testfinally ran (queued ~3h) and failed — but in internal/api, internal/defaults, internal/cli, internal/settings, none of them packages this PR touches. Verified pre-existing on clean main e616aef (defaults guard fails there; settings failures are mcpx: three internal/settings guard tests fail on main (compendium chapter + undeclared adapter env) #2697/mcpx docs: 13 pages missing from compendium.manifest.json — TestEveryDocumentHasAChapter red on main #2552; api passes on current main) and the composition is recorded on the main-broken tracker main-broken: developing-today/code mcpx @9f3c829 — basic: test #2534. The PR's own packages (daemon/events) were verified by the previous lane (vet clean, events + daemon tests pass) and Codex had no findings — merged under the standing criteria for unrelated pre-existing failures.dezren39 commentedon Oct 30, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 30, 2026
🤖 Agent note · done
event: gapfor live drops + full 1024-event retained-history replay test). Itsbasic: testfailure is in packages this PR does not touch (api/defaults/cli/settings), verified pre-existing on clean main and tracked on main-broken: developing-today/code mcpx @9f3c829 — basic: test #2534; the PR's own packages were verified (vet clean, events + daemon focused tests pass, Codex no findings). Closing mcpx: /v1/events replay drops everything past 256 events without a gap marker, though reconnection is documented as lossless #1914 — the replay half was fixed earlier on main, this PR closed the live-drop half.