Empty stop completions are recorded as successful responses #37735

Activity

1. github-actions

   assigned

   nexxeln

   on Jul 19, 2026
2. github-actions commented on Jul 19, 2026

   

   github-actionsBot

   on Jul 19, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue might be a duplicate of an existing issue. Please check:

   • v2: empty reasoning-only response is recorded as successful completion #37372: v2 records an empty reasoning-only response (no text, no tool calls, finish: "stop") as a successful completion — the same core invariant failure described here, observed in the V2 event model.

   #37735 is distinct in that it focuses on the case where a provider emits a stop finish after a tool call (not a reasoning-only turn), but the root cause and fix surface are likely the same: V2 accepts a terminal step with no usable assistant output as successful. It may be worth consolidating or at least cross-referencing these before implementing a fix.
3. bekandev commented on Aug 26, 2026

   

   bekandev

   on Aug 26, 2026

   More actions

   I hit this in production and built a deterministic reproduction plus a tested fix. Sharing the evidence since this issue currently has no repro.

   Deterministic reproduction

   A minimal OpenAI-compatible SSE server that opens the assistant role and then closes on stop without ever sending a content delta:

   data: {"choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
   data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
   data: [DONE]
   

   Point a provider at it and run any prompt. Reproduces 3/3 on 1.18.23 (Homebrew binary) and from source on dev @ 902e67e, so it is not a packaging artifact:

   {"type":"step_start","part":{"type":"step-start"}}
   {"type":"step_finish","part":{"type":"step-finish","reason":"stop",
     "tokens":{"input":0,"output":0,"reasoning":0,"cache":{"write":0,"read":0}},"cost":0}}

   Exit code 0, no error event, no WARN/ERROR log line. The assistant message is persisted with finish: "stop" and only step-start / step-finish parts.

   Production occurrences

   From my local session database, assistant turns with finish: "stop" and no text, tool, or reasoning part:

   Provider / model	Turns
   gpt-5.6-terra	7
   claude-sonnet-5	2
   gpt-5.6-luna	1
   claude-opus-4-8	1

   Four models across two families, all through one OpenAI-compatible gateway. Query used:

   SELECT json_extract(m.data,'$.modelID') AS model, COUNT(*)
   FROM message m
   WHERE json_extract(m.data,'$.role') = 'assistant'
     AND json_extract(m.data,'$.finish') = 'stop'
     AND NOT EXISTS (SELECT 1 FROM part p WHERE p.message_id = m.id
                     AND json_extract(p.data,'$.type') IN ('text','tool','reasoning'))
   GROUP BY model;

   All of them report input=0, output=0, so nothing was accounted for — the request produced no usage at all.

   Why #41466 does not cover this

   PR #41466 (rebase of #40531) guards only unknown finish reasons:

   if (ctx.assistantMessage.finish === "unknown" && !generated) { ... }

   Every occurrence above finishes with stop, so that guard never fires for them. Confirmed by cherry-picking #41466 onto dev and rerunning the repro: still silent.

   A second, opposite defect from the same place

   While reproducing, the unknown variant turned out to fail the other way. Same server, only finish_reason changed to a non-standard value that normalizes to unknown:

   Repro mode	Requests to the fake provider
   finish_reason: "stop"	2, then silent success
   non-standard reason → unknown	2517 before I killed it

   It never terminates. Both behaviors trace to prompt.ts:

   const finished = handle.message.finish && !["tool-calls", "unknown"].includes(handle.message.finish)

   stop counts as finished, so the loop breaks without ever checking whether the turn produced output. unknown never counts as finished, so the loop keeps re-requesting. Neither branch asks the question that matters: did this turn generate anything?

   Note this is the turn loop, not SessionRetry.policy — the policy already bounds at RETRY_MAX_RETRIES. The two retry levels are distinct.

   I could not find an existing issue for the unbounded unknown case (#15533 is auto-compaction, #17648 predates the current bound). Happy to file it separately if that is preferred.

   Tested fix

   Guarding on output produced rather than on the finish reason covers both. In processor.ts, track whether the stream opened any content block and fail the attempt when it did not:

   const EMPTY_TURN_FINISH = new Set(["stop", "unknown"])
   
   // in the stream tap
   if (event.type === "text-start" || event.type === "reasoning-start" ||
       event.type === "tool-input-start" || event.type === "tool-call") generated = true
   
   // after runDrain
   if (!generated && !ctx.needsCompaction && EMPTY_TURN_FINISH.has(ctx.assistantMessage.finish ?? "")) {
     yield* new SessionRetry.EmptyResponseError({ ... })
   }

   Then make EmptyResponseError retryable by identity in SessionRetry.policy, so it flows through the existing RETRY_MAX_RETRIES bound.

   Three details that differ from #41466:

   • stop is included, which is what covers this issue.
   • !ctx.needsCompaction — compaction interrupts the stream deliberately, so without this the guard fires a false positive on every compaction.
   • Tracking text-start, not text-delta with a length check. A provider that opened a text block did respond even if the block is empty. This matters: an SSE delta: {content: ""} produces no text-start/text-delta events at all from the AI SDK (verified against the SDK directly — the stream is just start → start-step → finish-step → finish), so on the wire an empty-string completion is indistinguishable from a fully empty turn.

   Results against the repro, same server:

   Mode	Before	After
   stop, empty	2 requests, silent success, exit 0	7 requests, then The model returned an empty response (finish reason: stop)
   unknown, empty	2517+, never terminates	7 requests, then a surfaced error
   stop with text	1 request	1 request, unchanged
   reasoning-only	—	1 request, guard does not fire

   7 requests = 1 attempt + 5 retries + 1, consistent with RETRY_MAX_RETRIES = 5.

   Verification on dev @ 902e67e: bun test test/session/ → 417 pass, 7 skip, 1 todo, 0 fail (425 tests, 18 files). bun typecheck clean. oxlint on the touched files reports no new warnings (diffed against a clean tree). Five tests added, covering the empty stop retry, the empty unknown retry, a short non-empty stop that must not retry, reasoning-only staying out of scope, and the retry bound.

   One existing test needed a one-line change: publish retry status updates used llm.text("") as filler to close the loop after a 503. Since an empty completion is now itself retryable, that blank reply adds a second retry and masks the 503 the test measures, so it now sends actual content. It is the only llm.text("") in the suite.

   Scope

   This is a harness guard: it converts a silent false success into a bounded, visible failure. It does not address why a gateway returns an empty stream — that stays a provider-side question. Reasoning-only empty turns (#37372) are deliberately out of scope here, since reasoning counts as generated output.

   Happy to open a PR if the approach looks right.
4. bekandev commented on Aug 26, 2026

   

   bekandev

   on Aug 26, 2026

   More actions

   Filed the unbounded unknown case separately as #45315, since it is a distinct failure mode (never terminates) even though it shares the predicate described above.
5. bekandev commented on Aug 26, 2026

   

   bekandev

   on Aug 26, 2026

   More actions

   Correction to my earlier comment: the request counts I posted were each inflated by one. opencode requests the session title through the same /chat/completions endpoint, and my fake provider was counting that as part of the turn under test. Filtering it out (the repo's own test/lib/llm-server.ts does this via isTitleRequest) gives the accurate numbers:

   Mode	Requests for the turn
   empty stop, unpatched	1 — no retry at all, gives up on the first attempt
   empty stop, with the guard	6 (1 attempt + 5 retries), then a surfaced error

   So the current behavior is worse than I described: there is no retry whatsoever, the turn is abandoned immediately. The RETRY_MAX_RETRIES = 5 bound and every test result are unaffected — the unit tests use the repo's harness, which already filters title requests.

   I also added a transient case to the repro, which is the shape that matters most in practice: first attempt empty, second attempt fine.

   	Requests	Outcome
   Unpatched	1	silent empty turn; the second attempt that would have succeeded is never made
   With the guard	2	empty turn, automatic retry, normal answer

   That is the common gateway hiccup, and today a single blip discards the turn even though the provider is already healthy again.