Repository navigation
chunkTimeout resets on SSE comment heartbeats — keepalive-warm stalled streams never time out #43519
Description
chunkTimeout resets on SSE comment heartbeats — keepalive-warm stalled streams never time out
Version: 1.18.18 (behavior also observed on 1.17.5 before the option existed)
Summary
provider.<id>.options.chunkTimeout arms a timer per SSE reader.read() and
aborts with the typed, retryable APIError: SSE read timed out
(ProviderResponseStreamError). That works for a fully silent stream — but the
timer resets on any chunk, including SSE comment lines (: keepalive) that
carry no data: event. SSE comments are explicitly allowed by the SSE spec for
connection keep-alive, so a provider or load balancer that holds a stalled
generation open with comment heartbeats keeps the connection "warm" forever:
the chunk timer never fires, and opencode run hangs indefinitely with no
stdout events, no log lines, and no error.
Impact
In long agentic sessions we see provider streams stall mid-generation (e.g. a
tool-call part is created, zero argument bytes ever arrive, the stream then
sits open). With a comment heartbeat interval shorter than chunkTimeout, the
session hangs until an external watchdog kills the process. The stall is
invisible to every in-process signal: no stdout JSON event, no native log
line, no session-DB write, no error.
Deterministic repro (no provider, no network)
Save as stall_server.py and run with STALL_MODE=keepalive KEEPALIVE_SECS=10:
#!/usr/bin/env python3
"""OpenAI-compatible SSE stall server. Modes:
silent - role delta + partial tool-call args, then hold the body open
with zero further bytes.
keepalive - same prefix, then an SSE comment (": keepalive") every
KEEPALIVE_SECS forever. Chunks keep arriving but no data event
ever completes.
"""
import http.server, json, os, sys, time
MODE = os.environ.get("STALL_MODE", "silent")
KEEPALIVE_SECS = float(os.environ.get("KEEPALIVE_SECS", "20"))
PORT = int(os.environ.get("PORT", "8123"))
def sse(obj):
return ("data: " + json.dumps(obj) + "\n\n").encode()
class Handler(http.server.BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args):
sys.stderr.write("[stall-server] %s\n" % (fmt % args)); sys.stderr.flush()
def do_POST(self):
self.rfile.read(int(self.headers.get("Content-Length", 0)))
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.send_header("Connection", "keep-alive")
self.end_headers()
base = {"id": "chatcmpl-stall", "object": "chat.completion.chunk",
"created": int(time.time()), "model": "stall"}
self.wfile.write(sse({**base, "choices": [{"index": 0,
"delta": {"role": "assistant"}, "finish_reason": None}]}))
self.wfile.flush()
self.wfile.write(sse({**base, "choices": [{"index": 0, "delta":
{"tool_calls": [{"index": 0, "id": "chatcmpl-tool-stall",
"type": "function", "function": {"name": "edit",
"arguments": "{\"filePa"}}]}, "finish_reason": None}]}))
self.wfile.flush()
try:
if MODE == "keepalive":
while True:
time.sleep(KEEPALIVE_SECS)
self.wfile.write(b": keepalive\n\n")
self.wfile.flush()
self.log_message("keepalive sent")
else:
while True:
time.sleep(3600)
except (BrokenPipeError, ConnectionResetError):
self.log_message("client closed connection")
do_GET = do_POST
if __name__ == "__main__":
http.server.ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()Config (test-config.json) pointing any OpenAI-compatible provider at it, with
a 15s chunk timeout:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"fireworks-ai": {
"options": {
"chunkTimeout": 15000,
"baseURL": "http://127.0.0.1:8123/v1"
}
}
}
}Run:
python3 stall_server.py & # STALL_MODE=keepalive KEEPALIVE_SECS=10
OPENCODE_CONFIG=test-config.json timeout 300 opencode run \
--model "fireworks-ai/<any-model-id>" --format json \
--dangerously-skip-permissions "say hi"Observed
STALL_MODE=silent: chunkTimeout fires — retries, then terminal typed
APIError: SSE read timed out(isRetryable: true). Correct.STALL_MODE=keepalivewithKEEPALIVE_SECS=10(heartbeats inside the 15s
chunk window): onestep_startevent, then permanent silence. No timeout
after 10x the chunk window; the external 300s bound is the only thing that
ends the process.- Control with
KEEPALIVE_SECS=20(gap longer than the 15s chunk window):
the timeout fires on every gap — opencode retries with exponential backoff
and exits 1 with the typed error at ~5 minutes. So the timer itself works;
it is specifically reset by non-data chunks.
Expected
chunkTimeout should bound event progress, not transport bytes: ignore
non-data: chunks (comments/heartbeats) for the timer, or add a separate
event-level timeout (no complete SSE data: event for N seconds) so a
heartbeat-warm but generation-dead stream still fails with the typed,
retryable error.
Possibly related to #28729 (silent mid-session stream drops); this issue is
the complementary case where the connection is kept alive by non-data bytes.
Activity
royalskynet commentedon Oct 3, 2026
Real-world data point, in case it helps prioritize: we hit this against OpenCode Zen itself (free tier,
opencode/longcat-2.5-preview-free), not a third-party proxy.What we saw
opencode runwith no external timeout sat for ~30 min: no stdout events, no error, process alive. Other runs in the same window on the same model hung the same way.Why a shorter timeout alone isn't a good workaround
We measured per-step "longest silence" (gap between consecutive part updates) from the local session DB: 429 longcat steps total; the shares below are over the 376 coding-agent steps (the rest were long-form analysis, which is quieter still):
So legitimate long reasoning/writing gaps overlap heavily with stalls. Dropping a watchdog from 300 s to 180 s roughly triples false kills (each one restarts the whole task), which in our workload costs more than the ~120 s saved per real stall.
Workaround we use
An external watchdog on the
--format jsonevent stream (kill after N seconds with no new event, N = 300), plus a hard wall-clock limit. Heartbeats don't produce events, so this catches the keepalive-warm case. Every manualopencode rungets wrapped intimeouttoo.A fix that only resets the chunk timer on
data:events (ignoring SSE comments) would letchunkTimeoutreplace the external watchdog for us.fix(ai): bound native stream stalls on framed events
Merge pull request anomalyco#52866 from felixzsh/opencode
thdxr commentedon Oct 11, 2026
Thanks for the report! OpenCode 2.0 is now out and development is focused on V2, so we're closing issues filed against OpenCode 1. Please upgrade (
opencode upgrade --major, details: https://anoma.ly/notes/opencode-2-0), and if this is still relevant on 2.0, reopen this issue or open a new one with your 2.0 version.— from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊