Skip to content

chunkTimeout resets on SSE comment heartbeats — keepalive-warm stalled streams never time out #43519

Closed as not planned

Description

@bpross

chunkTimeout resets on SSE comment heartbeats — keepalive-warm stalled streams never time out

Version: 1.18.18 (behavior also observed on 1.17.5 before the option existed)

Summary

provider.<id>.options.chunkTimeout arms a timer per SSE reader.read() and
aborts with the typed, retryable APIError: SSE read timed out
(ProviderResponseStreamError). That works for a fully silent stream — but the
timer resets on any chunk, including SSE comment lines (: keepalive) that
carry no data: event. SSE comments are explicitly allowed by the SSE spec for
connection keep-alive, so a provider or load balancer that holds a stalled
generation open with comment heartbeats keeps the connection "warm" forever:
the chunk timer never fires, and opencode run hangs indefinitely with no
stdout events, no log lines, and no error.

Impact

In long agentic sessions we see provider streams stall mid-generation (e.g. a
tool-call part is created, zero argument bytes ever arrive, the stream then
sits open). With a comment heartbeat interval shorter than chunkTimeout, the
session hangs until an external watchdog kills the process. The stall is
invisible to every in-process signal: no stdout JSON event, no native log
line, no session-DB write, no error.

Deterministic repro (no provider, no network)

Save as stall_server.py and run with STALL_MODE=keepalive KEEPALIVE_SECS=10:

#!/usr/bin/env python3
"""OpenAI-compatible SSE stall server. Modes:
  silent    - role delta + partial tool-call args, then hold the body open
              with zero further bytes.
  keepalive - same prefix, then an SSE comment (": keepalive") every
              KEEPALIVE_SECS forever. Chunks keep arriving but no data event
              ever completes.
"""
import http.server, json, os, sys, time

MODE = os.environ.get("STALL_MODE", "silent")
KEEPALIVE_SECS = float(os.environ.get("KEEPALIVE_SECS", "20"))
PORT = int(os.environ.get("PORT", "8123"))

def sse(obj):
    return ("data: " + json.dumps(obj) + "\n\n").encode()

class Handler(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, fmt, *args):
        sys.stderr.write("[stall-server] %s\n" % (fmt % args)); sys.stderr.flush()
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.send_header("Cache-Control", "no-cache")
        self.send_header("Connection", "keep-alive")
        self.end_headers()
        base = {"id": "chatcmpl-stall", "object": "chat.completion.chunk",
                "created": int(time.time()), "model": "stall"}
        self.wfile.write(sse({**base, "choices": [{"index": 0,
            "delta": {"role": "assistant"}, "finish_reason": None}]}))
        self.wfile.flush()
        self.wfile.write(sse({**base, "choices": [{"index": 0, "delta":
            {"tool_calls": [{"index": 0, "id": "chatcmpl-tool-stall",
            "type": "function", "function": {"name": "edit",
            "arguments": "{\"filePa"}}]}, "finish_reason": None}]}))
        self.wfile.flush()
        try:
            if MODE == "keepalive":
                while True:
                    time.sleep(KEEPALIVE_SECS)
                    self.wfile.write(b": keepalive\n\n")
                    self.wfile.flush()
                    self.log_message("keepalive sent")
            else:
                while True:
                    time.sleep(3600)
        except (BrokenPipeError, ConnectionResetError):
            self.log_message("client closed connection")
    do_GET = do_POST

if __name__ == "__main__":
    http.server.ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()

Config (test-config.json) pointing any OpenAI-compatible provider at it, with
a 15s chunk timeout:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "fireworks-ai": {
      "options": {
        "chunkTimeout": 15000,
        "baseURL": "http://127.0.0.1:8123/v1"
      }
    }
  }
}

Run:

python3 stall_server.py &  # STALL_MODE=keepalive KEEPALIVE_SECS=10
OPENCODE_CONFIG=test-config.json timeout 300 opencode run \
  --model "fireworks-ai/<any-model-id>" --format json \
  --dangerously-skip-permissions "say hi"

Observed

  • STALL_MODE=silent: chunkTimeout fires — retries, then terminal typed
    APIError: SSE read timed out (isRetryable: true). Correct.
  • STALL_MODE=keepalive with KEEPALIVE_SECS=10 (heartbeats inside the 15s
    chunk window): one step_start event, then permanent silence. No timeout
    after 10x the chunk window; the external 300s bound is the only thing that
    ends the process.
  • Control with KEEPALIVE_SECS=20 (gap longer than the 15s chunk window):
    the timeout fires on every gap — opencode retries with exponential backoff
    and exits 1 with the typed error at ~5 minutes. So the timer itself works;
    it is specifically reset by non-data chunks.

Expected

chunkTimeout should bound event progress, not transport bytes: ignore
non-data: chunks (comments/heartbeats) for the timer, or add a separate
event-level timeout (no complete SSE data: event for N seconds) so a
heartbeat-warm but generation-dead stream still fails with the typed,
retryable error.

Possibly related to #28729 (silent mid-session stream drops); this issue is
the complementary case where the connection is kept alive by non-data bytes.

Activity

  1. royalskynet commented on Oct 3, 2026

    @royalskynet

    Real-world data point, in case it helps prioritize: we hit this against OpenCode Zen itself (free tier, opencode/longcat-2.5-preview-free), not a third-party proxy.

    What we saw

    • opencode run with no external timeout sat for ~30 min: no stdout events, no error, process alive. Other runs in the same window on the same model hung the same way.
    • The stalled steps were not slow-to-start: first output arrived within 2–8 s, then the stream went quiet mid-generation and never finished — matching the "stalls mid-generation" pattern described above.

    Why a shorter timeout alone isn't a good workaround
    We measured per-step "longest silence" (gap between consecutive part updates) from the local session DB: 429 longcat steps total; the shares below are over the 376 coding-agent steps (the rest were long-form analysis, which is quieter still):

    longest silence in a step that completed normally share of steps
    > 120 s ~5.6%
    > 180 s ~3.5%
    > 300 s ~1%
    max observed 882 s

    So legitimate long reasoning/writing gaps overlap heavily with stalls. Dropping a watchdog from 300 s to 180 s roughly triples false kills (each one restarts the whole task), which in our workload costs more than the ~120 s saved per real stall.

    Workaround we use
    An external watchdog on the --format json event stream (kill after N seconds with no new event, N = 300), plus a hard wall-clock limit. Heartbeats don't produce events, so this catches the keepalive-warm case. Every manual opencode run gets wrapped in timeout too.

    A fix that only resets the chunk timer on data: events (ignoring SSE comments) would let chunkTimeout replace the external watchdog for us.

  2. thdxr commented on Oct 11, 2026

    @thdxr
    Member

    Thanks for the report! OpenCode 2.0 is now out and development is focused on V2, so we're closing issues filed against OpenCode 1. Please upgrade (opencode upgrade --major, details: https://anoma.ly/notes/opencode-2-0), and if this is still relevant on 2.0, reopen this issue or open a new one with your 2.0 version.

    — from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions