2.0: Scope event streams and bound event payloads #36441

Activity

1. kitlangton

   added

   tui

   discussionUsed for feature requests, proposals, ideas, etc. Open discussionUsed for feature requests, proposals, ideas, etc. Open discussion

   perfIndicates a performance issue or need for optimizationIndicates a performance issue or need for optimization

   coreAnything pertaining to core functionality of the application (opencode server stuff)Anything pertaining to core functionality of the application (opencode server stuff)

   2.0

   on Jul 11, 2026
2. kitlangton commented on Jul 19, 2026

   

   kitlangton

   on Jul 19, 2026

   ContributorAuthor

   More actions

   Reproduced this on 0.0.0-next-15834 with one healthy, version-matched managed service, nine TUI clients, and eight running sessions. This is a concrete current incident rather than a synthetic benchmark:

   Measurement	Observed
   TUI CPU during six 2-second top samples	13-59% per older client, with an earlier 122% spike
   Managed-service CPU	21-60%
   Older TUI memory reported by top	413 MiB-2.1 GiB each
   Identical per-client receive deltas in consecutive nettop snapshots	50,120; 31,611; 35,462; 21,399 bytes
   Rolling 10-minute durable activity	4,136 events / 7.0 MiB in one capture
   Local durable state	2,121 sessions, 1,293,728 event rows, 3.1 GiB event JSON, 6.1 GiB SQLite file

   The clients were connected from unrelated locations, but the older clients had each received about 209 MiB from the service since its 12:56 launch. A newer client had already received 106 MiB. session.tool.success and session.tool.progress dominated the rolling payload volume; one recent success event was 684 KiB. The client receive counters advanced by the same amount even for TUIs whose displayed session was unrelated.

   The current origin/v2 source matches the measurement. ClientProvider opens unscoped api.event.subscribe() and dispatches every event (packages/tui/src/context/client.tsx:67-93). DataProvider.handleEvent has no location/session interest gate (packages/tui/src/context/data.tsx:296-884), and message.update creates draft[sessionID] for event-owned sessions even when that session was never hydrated in this TUI. This means every client can accumulate and reactively mutate foreign-session message state. EventFeed now encodes each event once, so the issue body's per-subscriber schema-encoding point is no longer current, but queueing, SSE writes, client parsing/dispatch, and projection are still multiplied by the number of TUIs.

   The saved-session count is therefore not the direct cause of the continuous load. The immediate multiplier is concurrently active sessions times globally subscribed TUIs; the large database separately worsens cold reads/startup. A useful regression should assert that an event for session A neither changes session B's transport byte counter nor creates a session A message-store entry in a client interested only in B, while still supporting explicitly followed cross-location child sessions.
3. kitlangton commented on Jul 19, 2026

   

   kitlangton

   on Jul 19, 2026

   ContributorAuthor

   More actions

   The production follow-up has separated four additional failure boundaries from the global transport fan-out in this issue:

   • tui: ignore foreign-session events until the session is followed #37792: TUI projection needs its own followed-Session/location gate, even before server-side scoping lands.
   • core: cold VCS location acquisition blocked the service for 53 seconds #37794: one cold VCS Location blocked the incumbent service for 53.4 seconds; the aggregate Location timer cannot identify the slow node, although eager synchronous FFF creation is a strong candidate and fix(core): initialize fff lazily #36449 already implements the lazy shape.
   • core: project-copy refresh can report a 48-second no-op #37793: the following post-boot ProjectCopy.refresh timer remained incomplete for another 48.1 seconds with no changes, but current evidence cannot distinguish refresh work from scheduler starvation by another synchronous post-boot task.
   • cli: service restart can silently reuse an unresponsive incumbent #37795: an explicit service restart issued while the incumbent was unresponsive silently reused the same PID after it recovered because Service.stop treats a failed health probe as no existing service.

   I am filing these as focused issues and will link them here rather than mixing cold-start and lifecycle acceptance criteria into the scoped-event implementation.
4. kitlangton

   self-assigned this

   on Jul 19, 2026
5. kitlangton commented on Jul 20, 2026

   

   kitlangton

   on Jul 20, 2026

   ContributorAuthor

   More actions

   A live 0.0.0-next-15839 snapshot shows this is an active-runtime amplification problem, not only a cold-start or historical-database problem. Across the managed server and connected OpenCode clients, the sample accounted for about 13 GiB RSS and 457% CPU while system CPU was fully occupied. The managed health endpoint still returned healthy, but took 1.78 seconds. No process was restarted or terminated for the measurement.

   The newest 10,000 durable events covered about 22 minutes and 23 MiB. Tool success/progress dominated. The largest recent success was 2.43 MiB and came from webfetch; its structured.output alone was 2.22 MiB. In the 200 largest recent tool events, webfetch accounted for 4.55 MiB, with 3.57 MiB in structured and another 0.97 MiB in content. read and grep showed the same duplication pattern. ToolOutputStore.bound bounds model-facing content but preserves the full structured value, so large structured output bypasses the practical event budget.

   With many TUIs open, the process-global event feed turns this into N-way JSON decode and store projection. This reinforces the landing order: scope delivery by client interest first, then enforce a complete encoded-event budget and store one full blob/reference rather than retaining the same body in both structured and content. The blob path must remain replay/export aware.
6. kitlangton commented on Jul 20, 2026

   

   kitlangton

   on Jul 20, 2026

   ContributorAuthor

   More actions

   Design options

   There are three reasonable levels of response to oversized tool results:

   1. Hard truncate at settlement. Encode the complete event, cap it, and discard everything beyond a bounded preview. This is the smallest safe change and immediately prevents oversized database rows and SSE frames, but it loses irreproducible shell, web, and MCP output. It works better for pageable sources such as read, grep, and glob, where the model can request another page.
   2. Managed file plus preview. Extend the existing ToolOutputStore approach so the complete result is written once and the durable event contains only a preview, byte count, truncation metadata, and managed path. This preserves output, but raw paths are weak durable identities: export/copy must discover them, cleanup can invalidate replay, and paths are not portable across placement.
   3. Content-addressed artifact plus preview. Store complete output behind a stable artifact ID/hash and put only a bounded descriptor in the durable event. Replay, export, copy, and GC understand artifact references explicitly. This is more work, but it gives the cleanest invariant and is the recommended end state.

   Recommendation

   Use a content-addressed artifact, with hard truncation as the failure fallback. The durable result should look conceptually like:

   {
     content: boundedContent,
     metadata: boundedMetadata,
     artifact: { id, bytes, mediaType },
     truncated: true
   }

   The important invariant is: complete output is written once; every routine path carries only a bounded preview and stable reference. structured must be included in the same budget as content, not retained as an unbounded second copy. A blob write failure must never fall back to publishing the original oversized value; return an explicit lossy preview or fail settlement instead.

   Policies can differ by output kind:

   • read, grep, and glob: bound at the producer and return continuation/page information. Do not archive an entire reproducible source by default.
   • shell/process output: retain a tail or first/last preview and store the complete stream as one artifact.
   • web/MCP JSON: retain a preview plus bounded shape/schema metadata and store the complete value as one artifact.
   • images/binary data: use attachments/artifacts, never duplicate base64 across structured output, model content, events, and UI state.

   Also enforce two independent budgets:

   • Per-result and aggregate continuation budget. Parallel tools can each fit individually while their combined result is still too large. Bound the complete batch entering one continuation step, replacing the largest results first.
   • Complete encoded-event budget. Measure the final serialized event, including envelope, escaping, content, structured metadata, and references, before database insertion and SSE publication. The 16 MiB client limit should remain an emergency ceiling, not the normal server contract.

   Progress should be ephemeral, throttled/coalesced, and delta-based. Cap each delta independently, track total bytes separately, and persist only one bounded terminal result. Repeated cumulative progress snapshots should not become durable history.

   Replay should use the preview/reference directly and must not eagerly hydrate the full artifact into the next model request or every TUI. Artifact fetches should be interest-gated to the visible Session/tool call, with monotonic hydration so late reads cannot overwrite newer progress or settlement. Export/database copy must include referenced artifacts; a missing artifact should project an explicit unavailable marker rather than fail replay.

   Suggested landing order

   1. Add a universal encoded-event guard and remove the structured bypass.
   2. Scope event delivery by Session/Location interest so one result is not decoded by every TUI.
   3. Land artifact-backed settlement with replay/export/copy coverage and an explicit retention policy.
   4. Add lazy, interest-gated UI artifact loading with stale-result protection.
   5. Migrate or quarantine existing oversized rows so old data cannot continue breaking replay/SSE.

   This keeps the immediate safety fix small while avoiding a truncation-only format that would need another durable-event migration later.