You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[dev-lead timeout B/3] Catch exit-124 as reason=timeout and escalate to human (no same-budget retry) #1018
Classify (engine.sh): on a per-tier timeout (exit 124) in the deep/action tier, write reason=timeout to the /tmp/dev-lead-failure-reason sidecar (the mechanism handle_engine_failure already reads). Do NOT let is_transient_failure drive a same-budget model-call retry of a deep/action timeout.
Escalate (dev-lead-fix-issue.sh handle_engine_failure): add a reason=timeout branch before the retry/attempt < MAX_ATTEMPTS logic, mirroring the existing missing-binary immediate-escalate branch: ensure_needs_human_label + a status=needs-human ... reason=timeout comment stating the tier, elapsed seconds, and budget, with guidance ("hit the max deep budget — split this issue, or raise vars.DEEP_TIMEOUT_SEC, then re-apply dev-lead"), + the redacted session snippet; then exit 1. No retry marker (the retry cron skips dev-lead:needs-human).
Thanks for the detailed write-up and clear acceptance criteria — this is a well-scoped story! Before we dig in: is Story A (#901 parent) already merged and promoted to the larger budget, or should this implementation be structured to work independently of that landing first?
Issue:#1018 — [dev-lead timeout B/3] Catch exit-124 as reason=timeout and escalate to human (no same-budget retry)
Scope
Classify a per-tier stage timeout (exit 124) as a first-class, non-retryable reason=timeout in engine.sh, and make handle_engine_failure escalate it to a human on the FIRST occurrence (needs-human, no retry marker) with a comment stating tier + elapsed + budget + split/raise guidance — while surfacing any completed-but-unpushed work.
Implementation Checklist
scripts/engine.sh — in run_writer_with_fallback: clear stale timeout sidecars at start; time the run_writer call; on the non-fallback error path classify rc==124 as reason=timeout (distinct from engine-error) and write /tmp/dev-lead-timeout-{tier,budget,elapsed} sidecars. _claude_chain_invoke already propagates 124 without a same-budget chain retry.
scripts/dev-lead-fix-issue.sh — in handle_engine_failure: add a reason=timeout branch BEFORE the retry / attempt < MAX_ATTEMPTS logic (mirroring missing-binary): ensure_needs_human_label + status=needs-human … reason=timeout comment with tier/elapsed/budget + "split this issue, or raise vars.<TIER>_TIMEOUT_SEC, then re-apply dev-lead" guidance + redacted snippet; surface completed-but-unpushed work (branch name) rather than discarding it; exit 1, no retry marker.
Tests to Write
tests/dev-lead/unit/test_engine_fallback.bats — exit 124 → reason=timeout + tier/budget/elapsed sidecars; 124 does NOT fall through to another engine (no same-budget retry).
tests/dev-lead/unit/test_fix_issue.bats — stubbed 124 → needs-human + reason=timeout + no status=failed marker + tier/elapsed/budget/guidance in comment; non-124 transient (rate-limit) still retries (regression guard).
Test Command
bats tests/dev-lead/unit/test_engine_fallback.bats tests/dev-lead/unit/test_fix_issue.bats and bash .dev-lead/scripts/dev-lead-lint.sh
scripts/engine.sh — run_writer_with_fallback now classifies a per-tier stage timeout (exit 124) as a distinct, non-retryable reason=timeout (was engine-error) and records /tmp/dev-lead-timeout-{tier,budget,elapsed} sidecars. Stale sidecars are cleared at the start of each call. 124 still returns immediately (no same-budget cross-engine retry); _claude_chain_invoke already propagates 124 without a same-budget chain retry.
scripts/dev-lead-fix-issue.sh — handle_engine_failure gained a reason=timeout branch before the attempt-ceiling/retry logic (mirroring missing-binary): ensure_needs_human_label + a status=needs-human … reason=timeout comment stating tier + elapsed + budget with "split this issue, or raise vars.<TIER>_TIMEOUT_SEC, then re-apply dev-lead" guidance + redacted snippet, then exit 1. No retry marker (retry cron skips dev-lead:needs-human).
exit 124 in deep/action → reason=timeout (distinct from engine-error) in status marker + comment.
A timeout is NOT retried at engine or workflow layer — escalates on the FIRST occurrence.
The needs-human comment includes tier + elapsed + budget + split/raise guidance.
Genuine transient failures (non-124, e.g. rate-limit) retry exactly as today (regression test added).
Completed-but-unpushed work is surfaced, not silently discarded.
Tests: stubbed 124 → reason=timeout + escalate + no same-budget retry; non-124 transient → still retries.
Test Results
ok 1 fallback: primary exits 0 → success
ok 2 fallback: primary exits 2 → tries next engine
ok 3 fallback: all rate-limited → returns 2
ok 4 fallback: non-rate-limit exit 1 → no fallback
ok 5 fallback: engine not installed (exit 127) → tries next engine
ok 6 fallback: engine not installed (127) → remaining engines still tried
ok 7 fallback: fallback engine order is claude → copilot → gemini
ok 8 fallback: gemini skipped when no auth env vars set → falls through to copilot
ok 9 fallback: gemini used normally when GEMINI_API_KEY is set
ok 10 sidecar: all rate-limited → reason=rate-limited
ok 11 sidecar: missing binary (127) → reason=missing-binary
ok 12 sidecar: generic engine error (exit 1) → reason=engine-error
ok 13 sidecar: stage timeout (exit 124) → reason=timeout + tier/budget/elapsed sidecars
ok 14 sidecar: timeout (124) does NOT fall through to another engine (no same-budget retry)
ok 15 sidecar: stale reason cleared on success
ok 16 rate-limit path: session output is persisted + written to step summary
ok 17 fix-issue: dedup: existing open PR → exits 0 with comment
ok 18 fix-issue: dry-run: DEV_LEAD_DRY_RUN=true → logs [dry-run]
ok 19 fix-issue: missing ISSUE_NUMBER → exits 1
ok 20 fix-issue: ISSUE_TITLE and ISSUE_BODY exported to env before envsubst
ok 21 fix-issue: ORG_STANDARDS_HINT included in prompt context
ok 22 fix-issue: dry-run: check_existing_pr result does not affect dry-run path
ok 23 fix-issue: dry-run: prompt file path appears in output
ok 24 fix-issue: rate-limited: all engines rate-limited → exits 2, not 1
ok 25 fix-issue: rate-limited: posts comment on issue before exiting
ok 26 fix-issue: lint passes → proceeds to commit without error
ok 27 fix-issue: lint fails → posts issue comment and exits without committing
ok 28 fix-issue: engine-error (attempt 1) → exits 1, posts retry marker + cause, no silent exit
ok 29 fix-issue: engine-error embeds redacted session-output snippet in the comment
ok 30 fix-issue: missing-binary → escalates to needs-human, no retry marker
ok 31 fix-issue: stage timeout (exit 124) → escalates to needs-human, reason=timeout, no retry marker
ok 32 fix-issue: non-124 transient (rate-limit) still retries — no timeout regression
ok 33 fix-issue: late-phase timeout with completed work → surfaces branch, not silently discarded
ok 34 fix-issue: attempt ceiling (prior attempt=2) → escalates to needs-human
ok 35 fix-issue: lint fails → posts lint-failure comment on the issue
ok 33 fix-issue: late-phase timeout with completed work → surfaces branch, not silently discarded
ok 34 fix-issue: attempt ceiling (prior attempt=2) → escalates to needs-human
ok 35 fix-issue: lint fails → posts lint-failure comment on the issue
Full tests/dev-lead/unit/ suite: 666/667 pass. The single failure (test_engine_mcp.bats: "healthy handshake … ::notice:: surfaced") is pre-existing and unrelated — it fails identically on the unmodified base (verified via git stash); it depends on MCP-debug behavior absent in this sandbox.
Repo lint (bash .dev-lead/scripts/dev-lead-lint.sh): all checks passed (shellcheck warning-level on 85 files + agent frontmatter). shellcheck -S warning on both changed scripts: clean.
The dev-lead issue path uses the action/writer tier (ACTION_TIMEOUT_SEC), so the recorded tier is action and the guidance references vars.ACTION_TIMEOUT_SEC (accurate for this code path; the deep→DEEP_TIMEOUT_SEC mapping is retained for completeness). The issue's example guidance named DEEP_TIMEOUT_SEC generically.
Work surfacing is the issue's sanctioned "at minimum surface" option: the branch is local to the ephemeral runner and not pushed, so the comment points at the redacted session snippet rather than implying a recoverable remote branch.
Activity
github-actions commentedon Jul 2, 2026
Thanks for the detailed write-up and clear acceptance criteria — this is a well-scoped story! Before we dig in: is Story A (#901 parent) already merged and promoted to the larger budget, or should this implementation be structured to work independently of that landing first?
don-petry commentedon Jul 2, 2026
Dev-Lead Implementation Plan
Issue: #1018 — [dev-lead timeout B/3] Catch exit-124 as reason=timeout and escalate to human (no same-budget retry)
Scope
Classify a per-tier stage timeout (exit 124) as a first-class, non-retryable
reason=timeoutinengine.sh, and makehandle_engine_failureescalate it to a human on the FIRST occurrence (needs-human, no retry marker) with a comment stating tier + elapsed + budget + split/raise guidance — while surfacing any completed-but-unpushed work.Implementation Checklist
scripts/engine.sh— inrun_writer_with_fallback: clear stale timeout sidecars at start; time therun_writercall; on the non-fallback error path classifyrc==124asreason=timeout(distinct fromengine-error) and write/tmp/dev-lead-timeout-{tier,budget,elapsed}sidecars._claude_chain_invokealready propagates 124 without a same-budget chain retry.scripts/dev-lead-fix-issue.sh— inhandle_engine_failure: add areason=timeoutbranch BEFORE the retry /attempt < MAX_ATTEMPTSlogic (mirroringmissing-binary):ensure_needs_human_label+status=needs-human … reason=timeoutcomment with tier/elapsed/budget + "split this issue, or raisevars.<TIER>_TIMEOUT_SEC, then re-applydev-lead" guidance + redacted snippet; surface completed-but-unpushed work (branch name) rather than discarding it;exit 1, no retry marker.Tests to Write
tests/dev-lead/unit/test_engine_fallback.bats— exit 124 →reason=timeout+ tier/budget/elapsed sidecars; 124 does NOT fall through to another engine (no same-budget retry).tests/dev-lead/unit/test_fix_issue.bats— stubbed 124 → needs-human +reason=timeout+ nostatus=failedmarker + tier/elapsed/budget/guidance in comment; non-124 transient (rate-limit) still retries (regression guard).Test Command
bats tests/dev-lead/unit/test_engine_fallback.bats tests/dev-lead/unit/test_fix_issue.batsandbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jul 2, 2026
Dev-Lead: Implementation Complete
Plan Execution
scripts/engine.sh—run_writer_with_fallbacknow classifies a per-tier stage timeout (exit 124) as a distinct, non-retryablereason=timeout(wasengine-error) and records/tmp/dev-lead-timeout-{tier,budget,elapsed}sidecars. Stale sidecars are cleared at the start of each call. 124 stillreturns immediately (no same-budget cross-engine retry);_claude_chain_invokealready propagates 124 without a same-budget chain retry.scripts/dev-lead-fix-issue.sh—handle_engine_failuregained areason=timeoutbranch before the attempt-ceiling/retry logic (mirroringmissing-binary):ensure_needs_human_label+ astatus=needs-human … reason=timeoutcomment stating tier + elapsed + budget with "split this issue, or raisevars.<TIER>_TIMEOUT_SEC, then re-applydev-lead" guidance + redacted snippet, thenexit 1. No retry marker (retry cron skipsdev-lead:needs-human).Acceptance Criteria
reason=timeout(distinct fromengine-error) in status marker + comment.Test Results
Full
tests/dev-lead/unit/suite: 666/667 pass. The single failure (test_engine_mcp.bats: "healthy handshake … ::notice:: surfaced") is pre-existing and unrelated — it fails identically on the unmodified base (verified viagit stash); it depends on MCP-debug behavior absent in this sandbox.Repo lint (
bash .dev-lead/scripts/dev-lead-lint.sh): all checks passed (shellcheck warning-level on 85 files + agent frontmatter).shellcheck -S warningon both changed scripts: clean.Files Changed
scripts/engine.sh: 124 →reason=timeoutclassification + tier/budget/elapsed sidecars + stale-sidecar cleanup + elapsed timing aroundrun_writer.scripts/dev-lead-fix-issue.sh:reason=timeoutimmediate-escalation branch inhandle_engine_failure+ completed-work surfacing.tests/dev-lead/unit/test_engine_fallback.bats: 124 → timeout sidecars; 124 no cross-engine fallback.tests/dev-lead/unit/test_fix_issue.bats: 124 → needs-human/no-retry-marker/tier+elapsed+budget+guidance; non-124 transient still retries; late-phase timeout surfaces completed work.Notes
ACTION_TIMEOUT_SEC), so the recorded tier isactionand the guidance referencesvars.ACTION_TIMEOUT_SEC(accurate for this code path; thedeep→DEEP_TIMEOUT_SECmapping is retained for completeness). The issue's example guidance namedDEEP_TIMEOUT_SECgenerically.feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…
feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…