Skip to content

Engine: a hung LLM stream never times out — the turn and its busy registration leak forever #1739

Closed

Description

@aarontrowbridge

The episode (2026-10-09, live-diagnosed on shard 3): a session's turn started normally (streams at 09:46-09:47Z, step 3), then went silent with the engine otherwise healthy — /config 5ms, idle CPU, and crucially no provider socket left: the baseten connection died without an error ever surfacing to the stream await. The turn never unwound: the run-coordinator held the session busy for 8 hours while its work had long finished, the panel rendered eternal "thinking," and only an explicit POST /session/{id}/abort cleared it (cancel + the errored message at 17:32:25Z).

The gap: the transport work bounded the provider response (#1716's 64MB cap) and the question wait is now cancellable (#1730), but the LLM stream await itself has no idle/read timeout: a provider socket that dies without a clean FIN/EOF (tunnel drop, NAT timeout, LB idle-kill) leaves the read parked forever, and with it the turn, its coordinator registration, and the session's UI state.

Fix shape: an idle timeout on the stream read (configurable, generous — long generations legitimately pause between chunks; reasoning deltas arrive continuously, so a 60-120s idle ceiling is safe) that fails the step with a transport error — the existing error path then unwinds the turn, publishes the failure state, and frees the coordinator. This is the last member of the lifecycle family: tools (#1708), questions (#1730), and now the stream itself.

Reproduction sketch: start a turn, kill the provider connection mid-stream without a clean close (e.g. drop the TCP connection via a firewall rule or a proxy that blackholes it), observe the session stay busy indefinitely and the step never error.

Part of: the #775/#1717 lifecycle seams. Operational note: the frontdoor's liveness watchdogs catch wedged engines, not wedged turns — a session-level stuck-busy detector in the frontdoor (busy with zero log activity + zero event flow for N minutes → surface a warning card) would bridge the gap until this lands.

Activity

  1. aarontrowbridge commented on Oct 25, 2026

    @aarontrowbridge
    MemberAuthor

    Resolved on main: ffdeccd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions