Skip to content

[dev-lead timeout B/3] Catch exit-124 as reason=timeout and escalate to human (no same-budget retry) #1018

Closed

Description

@don-petry

Parent: #901. Story B of 3 — the behavior fix. Best implemented after Story A lands + promotes (larger budget).

Goal

Catch a stage timeout as a first-class, non-retryable condition and raise it for human review immediately — no wasted same-budget retries.

Scope (scripts/engine.sh + scripts/dev-lead-fix-issue.sh + tests)

  1. Classify (engine.sh): on a per-tier timeout (exit 124) in the deep/action tier, write reason=timeout to the /tmp/dev-lead-failure-reason sidecar (the mechanism handle_engine_failure already reads). Do NOT let is_transient_failure drive a same-budget model-call retry of a deep/action timeout.
  2. Escalate (dev-lead-fix-issue.sh handle_engine_failure): add a reason=timeout branch before the retry/attempt < MAX_ATTEMPTS logic, mirroring the existing missing-binary immediate-escalate branch: ensure_needs_human_label + a status=needs-human ... reason=timeout comment stating the tier, elapsed seconds, and budget, with guidance ("hit the max deep budget — split this issue, or raise vars.DEEP_TIMEOUT_SEC, then re-apply dev-lead"), + the redacted session snippet; then exit 1. No retry marker (the retry cron skips dev-lead:needs-human).
  3. Don't discard completed work (the [PR-limits Phase 3] Wire the admission gate into dev-lead-fix-issue.sh (enforce the org-wide cap) #1003 mode): if the implementation phase already produced commits/staged changes but a later phase times out, push the branch / open the PR (or at minimum surface "work completed but not pushed") rather than losing it.

Acceptance criteria

  1. exit 124 in deep/action → reason=timeout (distinct from engine-error) in the status marker + comment.
  2. A timeout is NOT retried with the same budget at either the engine or the workflow layer; it escalates to human on the FIRST occurrence.
  3. The needs-human comment includes tier + elapsed + budget + split/raise guidance.
  4. Genuine transient failures (non-124, e.g. rate-limit) retry exactly as today (no regression).
  5. Completed-but-unpushed work is not silently discarded on a late-phase timeout.
  6. Tests (tests/dev-lead/unit/): stubbed 124 → reason=timeout + escalate + no same-budget retry; non-124 transient → still retries.

Activity

  1. github-actions commented on Jul 2, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and clear acceptance criteria — this is a well-scoped story! Before we dig in: is Story A (#901 parent) already merged and promoted to the larger budget, or should this implementation be structured to work independently of that landing first?

  2. don-petry commented on Jul 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1018 — [dev-lead timeout B/3] Catch exit-124 as reason=timeout and escalate to human (no same-budget retry)

    Scope

    Classify a per-tier stage timeout (exit 124) as a first-class, non-retryable reason=timeout in engine.sh, and make handle_engine_failure escalate it to a human on the FIRST occurrence (needs-human, no retry marker) with a comment stating tier + elapsed + budget + split/raise guidance — while surfacing any completed-but-unpushed work.

    Implementation Checklist

    • scripts/engine.sh — in run_writer_with_fallback: clear stale timeout sidecars at start; time the run_writer call; on the non-fallback error path classify rc==124 as reason=timeout (distinct from engine-error) and write /tmp/dev-lead-timeout-{tier,budget,elapsed} sidecars. _claude_chain_invoke already propagates 124 without a same-budget chain retry.
    • scripts/dev-lead-fix-issue.sh — in handle_engine_failure: add a reason=timeout branch BEFORE the retry / attempt < MAX_ATTEMPTS logic (mirroring missing-binary): ensure_needs_human_label + status=needs-human … reason=timeout comment with tier/elapsed/budget + "split this issue, or raise vars.<TIER>_TIMEOUT_SEC, then re-apply dev-lead" guidance + redacted snippet; surface completed-but-unpushed work (branch name) rather than discarding it; exit 1, no retry marker.

    Tests to Write

    • tests/dev-lead/unit/test_engine_fallback.bats — exit 124 → reason=timeout + tier/budget/elapsed sidecars; 124 does NOT fall through to another engine (no same-budget retry).
    • tests/dev-lead/unit/test_fix_issue.bats — stubbed 124 → needs-human + reason=timeout + no status=failed marker + tier/elapsed/budget/guidance in comment; non-124 transient (rate-limit) still retries (regression guard).

    Test Command

    bats tests/dev-lead/unit/test_engine_fallback.bats tests/dev-lead/unit/test_fix_issue.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jul 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • scripts/engine.sh — run_writer_with_fallback now classifies a per-tier stage timeout (exit 124) as a distinct, non-retryable reason=timeout (was engine-error) and records /tmp/dev-lead-timeout-{tier,budget,elapsed} sidecars. Stale sidecars are cleared at the start of each call. 124 still returns immediately (no same-budget cross-engine retry); _claude_chain_invoke already propagates 124 without a same-budget chain retry.
    • scripts/dev-lead-fix-issue.sh — handle_engine_failure gained a reason=timeout branch before the attempt-ceiling/retry logic (mirroring missing-binary): ensure_needs_human_label + a status=needs-human … reason=timeout comment stating tier + elapsed + budget with "split this issue, or raise vars.<TIER>_TIMEOUT_SEC, then re-apply dev-lead" guidance + redacted snippet, then exit 1. No retry marker (retry cron skips dev-lead:needs-human).
    • Completed-but-unpushed work ([PR-limits Phase 3] Wire the admission gate into dev-lead-fix-issue.sh (enforce the org-wide cap) #1003 mode) is surfaced (branch + session snippet) instead of being silently dropped when a late phase times out.

    Acceptance Criteria

    1. exit 124 in deep/action → reason=timeout (distinct from engine-error) in status marker + comment.
    2. A timeout is NOT retried at engine or workflow layer — escalates on the FIRST occurrence.
    3. The needs-human comment includes tier + elapsed + budget + split/raise guidance.
    4. Genuine transient failures (non-124, e.g. rate-limit) retry exactly as today (regression test added).
    5. Completed-but-unpushed work is surfaced, not silently discarded.
    6. Tests: stubbed 124 → reason=timeout + escalate + no same-budget retry; non-124 transient → still retries.

    Test Results

    ok 1 fallback: primary exits 0 → success
    ok 2 fallback: primary exits 2 → tries next engine
    ok 3 fallback: all rate-limited → returns 2
    ok 4 fallback: non-rate-limit exit 1 → no fallback
    ok 5 fallback: engine not installed (exit 127) → tries next engine
    ok 6 fallback: engine not installed (127) → remaining engines still tried
    ok 7 fallback: fallback engine order is claude → copilot → gemini
    ok 8 fallback: gemini skipped when no auth env vars set → falls through to copilot
    ok 9 fallback: gemini used normally when GEMINI_API_KEY is set
    ok 10 sidecar: all rate-limited → reason=rate-limited
    ok 11 sidecar: missing binary (127) → reason=missing-binary
    ok 12 sidecar: generic engine error (exit 1) → reason=engine-error
    ok 13 sidecar: stage timeout (exit 124) → reason=timeout + tier/budget/elapsed sidecars
    ok 14 sidecar: timeout (124) does NOT fall through to another engine (no same-budget retry)
    ok 15 sidecar: stale reason cleared on success
    ok 16 rate-limit path: session output is persisted + written to step summary
    ok 17 fix-issue: dedup: existing open PR → exits 0 with comment
    ok 18 fix-issue: dry-run: DEV_LEAD_DRY_RUN=true → logs [dry-run]
    ok 19 fix-issue: missing ISSUE_NUMBER → exits 1
    ok 20 fix-issue: ISSUE_TITLE and ISSUE_BODY exported to env before envsubst
    ok 21 fix-issue: ORG_STANDARDS_HINT included in prompt context
    ok 22 fix-issue: dry-run: check_existing_pr result does not affect dry-run path
    ok 23 fix-issue: dry-run: prompt file path appears in output
    ok 24 fix-issue: rate-limited: all engines rate-limited → exits 2, not 1
    ok 25 fix-issue: rate-limited: posts comment on issue before exiting
    ok 26 fix-issue: lint passes → proceeds to commit without error
    ok 27 fix-issue: lint fails → posts issue comment and exits without committing
    ok 28 fix-issue: engine-error (attempt 1) → exits 1, posts retry marker + cause, no silent exit
    ok 29 fix-issue: engine-error embeds redacted session-output snippet in the comment
    ok 30 fix-issue: missing-binary → escalates to needs-human, no retry marker
    ok 31 fix-issue: stage timeout (exit 124) → escalates to needs-human, reason=timeout, no retry marker
    ok 32 fix-issue: non-124 transient (rate-limit) still retries — no timeout regression
    ok 33 fix-issue: late-phase timeout with completed work → surfaces branch, not silently discarded
    ok 34 fix-issue: attempt ceiling (prior attempt=2) → escalates to needs-human
    ok 35 fix-issue: lint fails → posts lint-failure comment on the issue
    
    ok 33 fix-issue: late-phase timeout with completed work → surfaces branch, not silently discarded
    ok 34 fix-issue: attempt ceiling (prior attempt=2) → escalates to needs-human
    ok 35 fix-issue: lint fails → posts lint-failure comment on the issue
    

    Full tests/dev-lead/unit/ suite: 666/667 pass. The single failure (test_engine_mcp.bats: "healthy handshake … ::notice:: surfaced") is pre-existing and unrelated — it fails identically on the unmodified base (verified via git stash); it depends on MCP-debug behavior absent in this sandbox.

    Repo lint (bash .dev-lead/scripts/dev-lead-lint.sh): all checks passed (shellcheck warning-level on 85 files + agent frontmatter). shellcheck -S warning on both changed scripts: clean.

    Files Changed

    • scripts/engine.sh: 124 → reason=timeout classification + tier/budget/elapsed sidecars + stale-sidecar cleanup + elapsed timing around run_writer.
    • scripts/dev-lead-fix-issue.sh: reason=timeout immediate-escalation branch in handle_engine_failure + completed-work surfacing.
    • tests/dev-lead/unit/test_engine_fallback.bats: 124 → timeout sidecars; 124 no cross-engine fallback.
    • tests/dev-lead/unit/test_fix_issue.bats: 124 → needs-human/no-retry-marker/tier+elapsed+budget+guidance; non-124 transient still retries; late-phase timeout surfaces completed work.

    Notes

    • The dev-lead issue path uses the action/writer tier (ACTION_TIMEOUT_SEC), so the recorded tier is action and the guidance references vars.ACTION_TIMEOUT_SEC (accurate for this code path; the deep→DEEP_TIMEOUT_SEC mapping is retained for completeness). The issue's example guidance named DEEP_TIMEOUT_SEC generically.
    • Work surfacing is the issue's sanctioned "at minimum surface" option: the branch is local to the ephemeral runner and not pushed, so the comment points at the redacted session snippet rather than implying a recoverable remote branch.
  4. added a commit that references this issue on Jul 2, 2026
  5. added 11 commits that reference this issue on Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions