Skip to content

Action budget, slice 4: make a timed-out dev-lead session diagnosable — stream the engine's turn-by-turn record to a file that survives the kill #1996

Open

Description

@don-petry

Slice of #1619. It's independent of #1658 / #1659 / #1706 and needs nothing else first.

Problem

When a Dev-Lead stage hits its budget, we can't tell what the session was doing.

_claude_chain_invoke (scripts/engine.sh, ~L1105–1160) runs:

timeout "$timeout_sec" claude --print --model "$model" --output-format json …

--output-format json emits a single envelope at the end of the session. GNU timeout kills the process at the budget, so on exit 124:

So we can't answer: how many turns ran, which tool calls took the time, whether it was stuck on one finding or working through eight, how many tokens it burned, or how far along it was. Tuning the budget, splitting issues or slimming prompts (#1619) is guesswork without this.

Recent example

#1953: Dev-Lead run 36856300906, step "Run fix-reviews", ran 11:35:46Z → 12:21:05Z on 2026-10-01, covering cubic's 8 findings. It failed with no record of what happened in those ~46 minutes. It had posted 10 comment dispositions by 11:56Z, so roughly 25 minutes of activity before the kill are unaccounted for. (The orphaned dispositions are #1992.)

Fix

Switch the Claude writer and agentic calls to --output-format stream-json --verbose. That emits one JSON event per line as the session runs: assistant turns, tool calls and results, and a final result event with the same usage / result fields as today's envelope. Tee the stream to a per-stage file, so whatever happened before the kill is on disk when timeout fires.

Acceptance criteria

  • Stream to disk. Claude stages that can hit ACTION_TIMEOUT_SEC / DEEP_TIMEOUT_SEC write a line-delimited event stream to $RUNNER_TEMP/engine-stream/<stage>-<hop>.jsonl as they run. At minimum that's the writer path; include deep/audit if the change is the same. The file must hold the events up to the moment of the kill, not only on success.
  • No behaviour change on success.
    • The text the caller receives (today's extracted .result) is identical.
    • parse_engine_usage gets the same usage / model data, read from the final result event.
    • Every existing bats test for the JSON-envelope path still passes, or is updated with an equivalent assertion.
    • ENGINE_USAGE_JSON=0 (text mode) still works.
  • A per-stage timeline, even on timeout. Emit one ::notice::[engine-timeline] … line per stage. It carries: turn count, tool-call count by tool name, the slowest N tool calls with their duration, input/output tokens where known, the last assistant action before the kill, and outcome (success / timeout / …). Also write it to the step summary. On a timeout, this is the "what was it doing" answer.
  • Artifact on failure only. Upload the stream files as engine-stream-<run_id> with short retention (≤ 14 days), only when the stage did not succeed (timeout, engine-error). Nothing extra is uploaded on success.
  • Redaction (CWE-532, same bar as fix(engine): unblock Opus 5.5 deep tier — valid Sonnet 5 id, fresh CLI cache, honest chain errors (#1957) #1958). The stream contains tool output and can contain secrets or untrusted PR text.
    • Pass every line through redact_secrets (scripts/lib/redact.sh) before it reaches the artifact or the logs.
    • Cap the size of each tool-result payload (e.g. keep the first and last ~2 KB and record the truncated length).
    • If redact_secrets is unavailable, upload nothing and only emit the timeline counts.
    • Never echo raw stream content into the Actions log.
  • Gemini/Copilot: out of scope. Note in the PR whether each CLI has an equivalent streaming mode, so a follow-up can mirror this.
  • Tests:
  • Keep it inside the engine invocation and its artifact upload. Don't change budgets (Measure and tune the dev-lead action budget: #1609 died 1s over ACTION_TIMEOUT_SEC and lost all its work — near-miss alerting + checkpoint push before tuning the number #1619: tune only on evidence) or the run loop.

Why this is worth doing before tuning

#1658/#1659 will tell us how long runs take. This slice tells us why: which part of a timed-out pass ate the budget. That decides whether the fix is a bigger budget, smaller issues, a cheaper prompt (e.g. not re-researching already-dispositioned comments, #1992), or a tool that hangs.

Rollout

These are Dev-Lead engine changes. Promote through the normal channel rings; no manual tag moves.

Activity

  1. github-actions commented on Oct 1, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and for tying it back to the concrete #1953 run, which makes the motivation easy to follow. The plan to stream events to a file that survives the timeout kill looks well scoped. One question before implementation: when a stage times out, would you rather the timeline's "slowest tool calls" and "last assistant action" be limited to short summaries (tool name, duration, truncated args), or do you want a bit of redacted tool-result context included too? That choice decides how much payload we keep per event under the redaction and size-cap requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions