Repository navigation
chat: broadcast fan-out runs each Send in its own goroutine — messages arrive out of order and a stalled session piles up unbounded blocked goroutines #4389
Description
Problem
app.send in pkgs/chat/main.go fans a chat message out with one new goroutine per recipient (go p.Send(msg), main.go:52-54). Nothing serializes those goroutines, so:
- Broadcast order is not preserved. Two messages sent in order can be delivered to a session in the opposite order whenever the first
Sendis delayed (a busy program, scheduler timing). Different sessions can therefore show the same conversation in different orders, and one session can show a reply before the message it answers. - A stalled session accumulates unbounded goroutines.
tea.Program.Sendblocks until the program's update loop reads the message (or the program ends). A session that stops reading — a wedged terminal, a dead-but-not-yet-pruned connection — leaves one blocked goroutine per broadcast message, forever, with no bound, coalescing, or drop.
Evidence (main @ 30ce6be, verified this run)
pkgs/chat/main.go:46-55—sendcopies the registry underRLock, thenfor _, p := range progs { go p.Send(msg) }. Thegois presumably there becausesendis called from inside a model'sUpdate(main.go:205-211) and the sender's own program is in the registry, so a naive synchronousSendto itself could block its own update loop — but the goroutine-per-recipient shape trades that for ordering and pile-up.- Laptop probe (verbatim
main.go/main_test.gofrom current main, md5 main.go ed60f0ca59fe0cec95101087ca792655; probe dir~/git/fleet/probe-2026-11-02-laneB-chatorder/on the laptop):- PROBE1: a recipient whose
Sendfor the textfirstsleeps 150 ms before recording received two broadcasts issued in order[first second]as[second first]— order inverted at one recipient. - PROBE2: 40 broadcasts to one recipient whose
Sendnever returns grew the goroutine count by exactly 40 — one permanently blocked goroutine per message, unbounded. - The existing registry tests (
TestSendReachesOnlyLivePrograms,TestRemoveUnknownIsNoOp,TestRegistryConcurrentRegisterSendRemove) all pass alongside the probes.
- PROBE1: a recipient whose
Why it matters
A chat room's one shared truth is the order of its messages. Today that order is per-session luck: two participants can read the same exchange in opposite orders with no error anywhere. And the pile-up half turns one stalled SSH session into slow, unbounded goroutine growth on the server for as long as the room stays busy — the registry pruning from #3201 only helps once the program actually ends, not while it is wedged.
Acceptance criteria
- Broadcasts are delivered to each live program in the order
sendwas called: a probe in the style of PROBE1 (a recipient whose firstSendis delayed) records[first second], and a multi-recipient probe shows every recipient recording the same order. - A recipient that never reads cannot accumulate goroutines without bound: the fan-out drops, coalesces, or queues with a stated bound for a stalled program (PROBE2-style measurement stays bounded as broadcasts continue), and the session is still pruned normally when its program ends.
- Sending a message from a session still delivers it to that same session (no self-send deadlock): the sender sees its own message exactly once, in order.
- Existing
pkgs/chattests pass; new tests pin both the ordering and the bounded-stalled-recipient behaviour using the existingsenderinterface /fakeProgharness.
Distinct from #3201 (closed: registry never pruned — this is the fan-out call shape that fix kept, not the registry lifecycle) and from #3209 (graceful shutdown exits 1, the Start/ListenAndServe path, unrelated to send).
Metadata
Metadata
Assignees
Labels
Package/app: chat (pkgs/ or apps/ in this repo)Small: expected under 2 hours of workSomething isn't working (prefixed form of the classic bug label)Ordinary priority (the default)Filed by the new-issue-creator jobTriaged and routed (see agent note)Queue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Nov 2, 2026
🤖 Agent note · filed
go p.Send(msg)per recipient, so delivery order is scheduler luck (two messages can arrive reversed at a session, and sessions disagree with each other) and a stalled session leaves one permanently blocked goroutine per broadcast, unbounded.dezren39 commentedon Nov 2, 2026
Agent note — task 6272 triage verdict: CONFIRMED on current main (lane code-k)
381fecb60607b34288fea966c7ea5a3433310b29:pkgs/chat/main.gomd5ed60f0ca59fe0cec95101087ca792655— byte-identical to the file this issue's probe exercised, so both probe results carry over unchanged (PROBE1 recorded[second first]; PROBE2: +40 goroutines after 40 broadcasts to a blocked recipient).app.send(main.go:47-56) copies the registry underRLock, then runsgo p.Send(msg)per program. Nothing serializes successive broadcasts to the same program, so delivery order per recipient is scheduler luck; and each broadcast to a stalled session parks one more goroutine inSenduntil that program ends — growth is unbounded and driven purely by room traffic.removedrops finished programs; the gap is only the fan-out shape chat: session program registry never pruned — unbounded growth, per-message fan-out to dead programs, and a data race on the shared slice #3201 kept).12 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: pr_opened
task: 6421 (repo-health-code-pilot, repo-worker-code, lane code-h)
agent: OpenCode worker
session: ses_e5e6276d091ffeZfMzx2BsZnIa
state: Opened PR #4530 from queue/6421-4389-chat-broadcast-order at commit 8a12197. Per-recipient dispatch is ordered; each recipient has one in-flight Send plus 16 queued messages, and queue overflow prunes it.
checks: go test ./... -count=1, go test -race ./... -count=1, go vet ./..., and git diff --check passed in pkgs/chat.
next: review fresh PR/check state and finish task 6421 as pr_opened.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: done
task: 6421 (repo-health-code-pilot, repo-worker-code, lane code-h)
agent: OpenCode worker
session: ses_e5e6276d091ffeZfMzx2BsZnIa
state: Terminal pr_opened. Issue #4389 OPEN; PR #4530 OPEN at commit 8a12197 from queue/6421-4389-chat-broadcast-order. Removed work/assigned, queue/claimed, and status/in-progress so no in-flight labels remain.
checks: go test ./... -count=1 pass; go test -race ./... -count=1 pass; go vet ./... pass; git diff --check pass; PR CI shows CodeRabbit neutral plus queued metered-provider guard.
next: merge #4530 when checks are green; issue #4389 closes on merge.
developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged
queue: repo-health-code-pilot
task: 8394 (completed, outcome
triaged)by: openchamber-scheduled-lane (repo-worker-code, lane bf419306)
at: 2026-11-03T19:42Z
Triage at current main (
pkgs/chat/main.gosendstill doesgo p.Send(msg)per recipient, so the bug is live).Fix path already exists: PR #4530
fix(chat): preserve broadcast order with bounded recipient queues(branchqueue/6421-4389-chat-broadcast-order, head 8a12197, OPEN, not draft, no held/user-owned labels). It adds one ordered delivery worker per recipient (ordering) and a depth-16 queue that prunes a stalled recipient on overflow (bounded goroutines), with tests inmain_test.go.No competing PR opened here. No follow-up task from triage. Remaining step is PR #4530 review and merge once its checks are green; that is a merge-verify step, not new triage work.
Labels: removed
queue/queuedandstatus/needs-triage; addedstatus/triaged.