Repository navigation
The connective tissue: one authority for location + liveness — the five-router architecture is one bug with many costumes #1760
Description
The diagnosis
Every failure in the 2026-10-10/11 stream-chain saga — silent merged streams, ghost question cards, "question request not found", split-brain busy sessions, stuck-busy leaks, boot-looping webviews — is the same bug wearing different costumes: state and location get resolved differently by different components at different times. No single authority answers "where does this live, and is it alive?"
The five routing authorities (all live today, all capable of disagreeing)
- The frontdoor — sticky session map, event-learned question map (seeded by polling), directory pools, weighted placement, and now per-request directory injection (
f20ee140). Each emergency added a responsibility; it is now a pub/sub broker, cache, snapshot builder, request rewriter, placement engine, and alarm system. Complexity budget: blown. - The engines — runtime truth (busy, pending questions, permissions, coordinator registration) is process-local memory while the shared SQLite says every session exists everywhere. The
/questionlist reads memory; the reply resolves directory→instance→memory. Any component resolving the wrong instance gets a confident 404 for state that provably exists (f20ee140's root cause). - The panel — its own protocol detection (V1/V2 wire shapes), a multi-server registry (
server/:serverKey/session/:id), per-directory sync children with silent-drop gates (SESSION_CONTENT_EVENTS×sessionContent: false), and a question dock whose SDK apparently resolves to a different door than its own view (evidence below). - The extension — server discovery plus a local engine pinned to a four-month-old fork build (
~/.vscode-serverMachine settings,opencode-local, pinned 2026-07-13) — a whole class of "the panel talks to WHAT?" confusion. - The fleet sessions themselves — they talk to engines directly on erlich (localhost:4094/4194/4294), bypassing every gateway fix. The rd-framework session's question (
que_12b45b0b0) got "reply for unknown request" on wrong engines at the user's exact answer times, with zero reply POSTs in the frontdoor's full log — a client population outside the fixed path.
Evidence trail
dd2788a6— merged event upstreams bound the engines' default instance (empty bus) and streamed silence for a day while extension, dist, encoder, and coalescing were each bisected and exonerated. The upstreams didn't know their directory; nobody was wrong, everybody disagreed.f20ee140— the reply'slocation.directorynever reached the query string; the engine resolved its default instance and 404'd a question it was actively holding. Verified: same reply, same shard, with directory → 200, turn resumed.ses_eda8295eregistered busy on two shards simultaneously after a mis-routed request lazily created a ghost copy (lazy rehydration + no ownership).- Engine: a hung LLM stream never times out — the turn and its busy registration leak forever #1739 (now
ffdeccde): hung LLM streams leaked turn + busy forever — runtime state invisible to the watchdog until it's irrecoverable. - The panel rendered question cards from list pulls for weeks while live
question.askedevents were silently dropped by thesessionContentgate — every card that ever rendered was stale-by-design. - fix(app+ops): client telemetry is dead — restore the ingest + a liveness canary so silence alarms #1745: client telemetry was gated behind a per-origin localStorage flag and a wrong base URL — silent for six weeks — so every panel failure in this saga was debugged blind, via server logs and headless probes (
/tmp/opencode/panel-probe/, the playwright harness that finally made the client visible). - Shard 1's engine restarted three times in one day with the cause unattributed at the time of writing.
The fix ladder (phased, cheapest first; each phase independently shippable)
- P1 — One door. Bind engines to loopback and require a frontdoor-only shared-secret header on their API ports. Every client — panel, extension, fleet sessions, CLI, probes — must use the gateway. This makes every routing fix universal instead of per-door. (Also: the engines'
SPA-fallback serves index.html to SSE requestsclass dies here — 404 JSON, never HTML, enforced at the door.) - P2 — Runtime state moves into the shared store. busy / pending questions / pending permissions / coordinator registration get shared rows with TTL heartbeats. Then any component can answer "where is this, is it alive": routing becomes a lookup, not a learned guess; ghost sessions become visible at creation; stale cards become detectable (card liveness = the row's TTL); the watchdog gets fleet-wide busy vision; replies become idempotent against the store rather than hostage to one process's memory.
- P3 — The panel's contracts collapse to one. The gateway normalizes to a single wire shape (delete protocol detection at the edge, then in the panel); one server per fleet panel (delete the multi-server registry for fleet setups); every mutation carries its full location (sessionID + directory); the silent-drop gates (
sessionContent,recent,existing-children) become unnecessary as the event flow becomes uniform. The dock's SDK resolves to the view's server, provably (add the request-URL assert to the headless probe suite). - P4 — The extension pin dies. Ship the third-actor feature to the lane or drop the pin; a four-month-old fork build as the panel's local engine is unattributable drift.
- P5 — Observability stays on, permanently. fix(app+ops): client telemetry — ungate the capture, root-relative ingest, heartbeat canary + freshness check #1748's telemetry + heartbeat canary + freshness check are preconditions for every phase above — the next bug must take an hour, not a day.
Verification
Every phase ships with headless end-to-end probes (the panel-probe harness) plus the frontdoor suite; P1/P2 additionally need a multi-client matrix (panel + fleet session + CLI answering the same question through the same door).
Honest scope note
This is not a rewrite. Engines, frontdoor, panel, and fleet all keep their shapes. It is a contract: one door, one source of location truth, one wire shape, one observability floor. The bugs this contract cannot prevent should at least all be findable in an hour.
Activity
feat(extension): amicode.hubServerUrl setting + posture decision (#1760…
feat(extension): reserved hub port guard at the spawn choke point (#1760
feat(extension): the hub client session — probe/attach loop, no spawn…
feat(extension): wire the hub-host client posture into activation (#1760
feat(app): the global event stream URL rides the #1748 telemetry chan…
Merge pull request #1784 from harmoniqs/opencode/issue-1760-p4-hub-cl…
feat(hub+app): attach protocol — one-wire snapshot reattach (#1760, r…