Skip to content

The connective tissue: one authority for location + liveness — the five-router architecture is one bug with many costumes #1760

Open

Description

@aarontrowbridge

The diagnosis

Every failure in the 2026-10-10/11 stream-chain saga — silent merged streams, ghost question cards, "question request not found", split-brain busy sessions, stuck-busy leaks, boot-looping webviews — is the same bug wearing different costumes: state and location get resolved differently by different components at different times. No single authority answers "where does this live, and is it alive?"

The five routing authorities (all live today, all capable of disagreeing)

  1. The frontdoor — sticky session map, event-learned question map (seeded by polling), directory pools, weighted placement, and now per-request directory injection (f20ee140). Each emergency added a responsibility; it is now a pub/sub broker, cache, snapshot builder, request rewriter, placement engine, and alarm system. Complexity budget: blown.
  2. The engines — runtime truth (busy, pending questions, permissions, coordinator registration) is process-local memory while the shared SQLite says every session exists everywhere. The /question list reads memory; the reply resolves directory→instance→memory. Any component resolving the wrong instance gets a confident 404 for state that provably exists (f20ee140's root cause).
  3. The panel — its own protocol detection (V1/V2 wire shapes), a multi-server registry (server/:serverKey/session/:id), per-directory sync children with silent-drop gates (SESSION_CONTENT_EVENTS × sessionContent: false), and a question dock whose SDK apparently resolves to a different door than its own view (evidence below).
  4. The extension — server discovery plus a local engine pinned to a four-month-old fork build (~/.vscode-server Machine settings, opencode-local, pinned 2026-07-13) — a whole class of "the panel talks to WHAT?" confusion.
  5. The fleet sessions themselves — they talk to engines directly on erlich (localhost:4094/4194/4294), bypassing every gateway fix. The rd-framework session's question (que_12b45b0b0) got "reply for unknown request" on wrong engines at the user's exact answer times, with zero reply POSTs in the frontdoor's full log — a client population outside the fixed path.

Evidence trail

  • dd2788a6 — merged event upstreams bound the engines' default instance (empty bus) and streamed silence for a day while extension, dist, encoder, and coalescing were each bisected and exonerated. The upstreams didn't know their directory; nobody was wrong, everybody disagreed.
  • f20ee140 — the reply's location.directory never reached the query string; the engine resolved its default instance and 404'd a question it was actively holding. Verified: same reply, same shard, with directory → 200, turn resumed.
  • ses_eda8295e registered busy on two shards simultaneously after a mis-routed request lazily created a ghost copy (lazy rehydration + no ownership).
  • Engine: a hung LLM stream never times out — the turn and its busy registration leak forever #1739 (now ffdeccde): hung LLM streams leaked turn + busy forever — runtime state invisible to the watchdog until it's irrecoverable.
  • The panel rendered question cards from list pulls for weeks while live question.asked events were silently dropped by the sessionContent gate — every card that ever rendered was stale-by-design.
  • fix(app+ops): client telemetry is dead — restore the ingest + a liveness canary so silence alarms #1745: client telemetry was gated behind a per-origin localStorage flag and a wrong base URL — silent for six weeks — so every panel failure in this saga was debugged blind, via server logs and headless probes (/tmp/opencode/panel-probe/, the playwright harness that finally made the client visible).
  • Shard 1's engine restarted three times in one day with the cause unattributed at the time of writing.

The fix ladder (phased, cheapest first; each phase independently shippable)

  • P1 — One door. Bind engines to loopback and require a frontdoor-only shared-secret header on their API ports. Every client — panel, extension, fleet sessions, CLI, probes — must use the gateway. This makes every routing fix universal instead of per-door. (Also: the engines' SPA-fallback serves index.html to SSE requests class dies here — 404 JSON, never HTML, enforced at the door.)
  • P2 — Runtime state moves into the shared store. busy / pending questions / pending permissions / coordinator registration get shared rows with TTL heartbeats. Then any component can answer "where is this, is it alive": routing becomes a lookup, not a learned guess; ghost sessions become visible at creation; stale cards become detectable (card liveness = the row's TTL); the watchdog gets fleet-wide busy vision; replies become idempotent against the store rather than hostage to one process's memory.
  • P3 — The panel's contracts collapse to one. The gateway normalizes to a single wire shape (delete protocol detection at the edge, then in the panel); one server per fleet panel (delete the multi-server registry for fleet setups); every mutation carries its full location (sessionID + directory); the silent-drop gates (sessionContent, recent, existing-children) become unnecessary as the event flow becomes uniform. The dock's SDK resolves to the view's server, provably (add the request-URL assert to the headless probe suite).
  • P4 — The extension pin dies. Ship the third-actor feature to the lane or drop the pin; a four-month-old fork build as the panel's local engine is unattributable drift.
  • P5 — Observability stays on, permanently. fix(app+ops): client telemetry — ungate the capture, root-relative ingest, heartbeat canary + freshness check #1748's telemetry + heartbeat canary + freshness check are preconditions for every phase above — the next bug must take an hour, not a day.

Verification

Every phase ships with headless end-to-end probes (the panel-probe harness) plus the frontdoor suite; P1/P2 additionally need a multi-client matrix (panel + fleet session + CLI answering the same question through the same door).

Honest scope note

This is not a rewrite. Engines, frontdoor, panel, and fleet all keep their shapes. It is a contract: one door, one source of location truth, one wire shape, one observability floor. The bugs this contract cannot prevent should at least all be findable in an hour.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions