Skip to content

[Phase 4] Demote dev-lead-retry from normal-path driver to a true backstop — resume blocked states event-first #1407

Closed

Description

@github-actions

Story

As a operator of the dev-lead runtime,
I want blocked/rate-limited dev-lead states resumed on the clearing event (review submitted / thread resolved / check_run success) via PAT repository_dispatch, leaving the 2 h cron only as a rare safety-net for genuinely un-eventable rate-limit recovery,
so that the 2 h retry cron stops acting as the de-facto convergence clock (the #860 'amplifier'), so normal convergence is event-driven and the timer only catches true exceptions.

Acceptance Criteria

  1. A dev-lead blocked/rate-limited state resumes on the clearing event (review, resolved thread, or check_run success) via a PAT-backed repository_dispatch, without waiting for the 2 h cron.
  2. The dev-lead-retry.yml cron is retained only as a safety-net for genuinely un-eventable rate-limit recovery, and its header documents and justifies it as timer_role: safety-net (not a normal-path driver).
  3. The event-first resume respects the existing per-PR automation budget and human-gated stop markers (needs-human-review), so it can never re-ignite a runaway — verified against scripts/lib/pr-automation-budget.sh semantics.
  4. The dev-lead interaction contract (Story 2) is updated to reflect the new event fast-path and the demoted timer_role, so validate-interaction-model (Story 4) passes.
  5. Ring-0 caller-stub freeze is respected: the resume bridge does not edit the frozen on:/with: block of dev-lead.yml without regenerating its baseline in the same diff.

Tasks / Subtasks

Dev Notes

  • The leak is documented in the feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860 postmortem: dev-lead-retry.yml (every 2 h) is listed as an 'Amplifier — re-ignites stalled loops' and the 2 h cron is the de-facto convergence clock for blocked/rate-limited PRs. The fix is to make resume event-first and leave the cron as a rare exception path.
  • The sanctioned bridge already exists: scripts/dev-lead-ci-relay.sh resolves a PR from a check_run and relays via PAT repository_dispatch (GITHUB_TOKEN cannot fire the downstream event — the recursion guard). Reuse this pattern; do not invent a new bridge.
  • The event-first resume MUST consult scripts/lib/pr-automation-budget.sh (compute_pr_automation_cycles / pr_budget_exhausted, default MAX_PR_AUTOMATION_CYCLES=10) and must skip PRs carrying human-gated markers (needs-human-review) — otherwise it recreates the feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860 amplifier. FORCE_REVIEW must not bypass the budget.
  • dev-lead.yml is a byte-frozen ring-0 caller stub (caller-stub-freeze, tests/fixtures/caller-stub-freeze/dev-lead.block). If the resume needs a new trigger on dev-lead.yml, the .block baseline must be regenerated in the same reviewed diff, and channel-pin input-forwarding sequencing (AGENTS.md) applies to any dev-lead-reusable input change.
  • Testing standard: extend the existing dev-lead retry tests; assert (a) a clearing event dispatches a resume, (b) an exhausted budget or a human marker suppresses it, (c) the cron path only fires for un-eventable rate-limit cases.

Project Structure Notes

Touches .github/workflows/dev-lead-retry.yml, scripts/dev-lead-retry.sh, the clearing-event resume wiring (reusing scripts/dev-lead-ci-relay.sh), the dev-lead interaction contract, and the associated bats tests. Must not edit the frozen dev-lead.yml stub block without baseline regeneration.

References

  • docs/postmortems/2026-06-pr-860-runaway.md
  • .github/workflows/dev-lead-retry.yml
  • scripts/dev-lead-retry.sh
  • scripts/dev-lead-ci-relay.sh
  • scripts/lib/pr-automation-budget.sh
  • scripts/lib/pr-runaway-detect.sh
  • AGENTS.md

Likely target surface

  • .github/workflows/dev-lead-retry.yml
  • scripts/dev-lead-retry.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #1402. Status: ready-for-dev.

Activity

  1. don-petry commented on Aug 2, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #1407 — [Phase 4] Demote dev-lead-retry from normal-path driver to a true backstop — resume blocked states event-first

    Scope

    Add an event-first resume fast-path that re-dispatches a PR's pending status=rate-limited dev-lead state the moment a clearing event (check_run success) arrives — gated by the per-PR automation budget and human stop markers — and demote the 2 h dev-lead-retry.yml cron from a de-facto convergence clock to a documented timer_role: safety-net backstop.

    Implementation Checklist

    Tests to Write

    • tests/dev-lead/unit/test_dev_lead_resume.bats — (a) clearing event + pending rate-limited marker + budget OK + no human marker → dispatches; (b) budget exhausted → suppressed; (c) human stop marker → suppressed; (d) FORCE_REVIEW does not bypass an exhausted budget; (e) no pending marker → no dispatch.
    • Confirm existing tests/dev-lead/unit/test_dev_lead_retry*.bats still pass (cron path unchanged).

    Test Command

    bats tests/dev-lead/unit/test_dev_lead_resume.bats tests/dev-lead/unit/test_dev_lead_retry.bats tests/dev-lead/unit/test_dev_lead_retry_pr_escalation.bats + python3 interaction-contracts/validate-interaction-contracts.py . + bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Aug 2, 2026

    @don-petry
    Collaborator

    [Retracted 2026-08-03 — superseded] This completion claim was published before the work was durable. The run subsequently ended without landing anything on main (reason=timeout, run=30740867354), so it produced nothing: no branch, no PR, and none of the described files or symbols exist. The claim is struck in place (not deleted) per this repo's dated-correction convention. Retraction applied manually to satisfy #1445 AC #6 — the shipped retract_prior_completion_claims only fires on a subsequent terminal-failure run, which cannot occur while this issue is gated at dev-lead:needs-human. See #1445.

    Completed — event-first resume for blocked/rate-limited dev-lead states [RETRACTED]

    The dev-lead-retry.yml 2 h cron is demoted from a de-facto convergence clock (the #860 "amplifier") to a true safety-net, by adding an event fast-path that resumes a blocked/rate-limited PR the moment a clearing event arrives — no longer waiting up to 2 h for the sweep.

    Plan execution

    • AC test issue from agent #1 — A blocked/rate-limited state resumes on the clearing event. New resume-relay job in dev-lead-reusable.yml fires on check_run success, resolves the PR via the sanctioned dev-lead-ci-relay.sh PAT bridge, and re-dispatches the pending status=rate-limited resume immediately. Review / thread-resolved clearing events already resume through the normal dispatch job (which re-classifies the event and re-attempts the work), so check_run success was the one uncovered clearing event.
    • AC Go-live improvements for PR review agent #2 — dev-lead-retry.yml cron retained ONLY as safety-net; header rewritten to document/justify timer_role: safety-net (rare backstop for genuinely un-eventable rate-limit recovery).
    • AC Add @claude delegation, auto-merge, and rebase handling #3 — Event-first resume respects the per-PR automation budget (resume_budget_exhausted → gather_pr_automation_events / pr_budget_exhausted) and human stop markers (resume_pr_is_human_gated over needs-human-review / dev-lead:needs-human / dev-lead:hands-off). FORCE_REVIEW deliberately does not buy back budget.
    • AC Optimize review: small-PR and incremental fast paths #4 — Interaction contract updated: timer role: self-heal → safety-net, event_fast_path null → non-null, dev-lead-reusable.yml added to workflows, guard justifications updated. Validator passes.
    • AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5 — Ring-0 caller-stub freeze respected: dev-lead.yml untouched (clearing events were already subscribed), caller_stub_freeze.sh reports ALIGNED, no baseline regen needed.

    Reused, not reinvented

    • scripts/dev-lead-ci-relay.sh — existing PAT bridge (GITHUB_TOKEN cannot wake the downstream event; the recursion guard). No new bridge invented.
    • scripts/dev-lead-retry.sh::scan_pr_for_rate_limits — single source of truth for which pending marker to re-dispatch and how; resume_pr_event_first layers the two extra gates then delegates.

    Test results

    • tests/dev-lead/unit/test_dev_lead_resume.bats — 7/7 pass (dispatch on marker+budget-OK; budget-exhausted suppresses; FORCE_REVIEW doesn't bypass; all three stop markers suppress; no-marker → no dispatch).
    • Full dev-lead unit suite — 726/726 pass.
    • validate-interaction-contracts.py (11 contracts) + test_validate_interaction_contracts.bats (22) — pass.
    • dev-lead-lint.sh — all checks passed; shellcheck clean (info-level source-follow notes only).

    Files changed

    File Change
    .github/workflows/dev-lead-reusable.yml new resume-relay job (check_run success → event-first resume)
    .github/workflows/dev-lead-retry.yml header → timer_role: safety-net
    scripts/dev-lead-retry.sh resume_pr_event_first + resume_pr_is_human_gated + resume_budget_exhausted; DEV_LEAD_STOP_LABELS; header
    scripts/dev-lead-resume.sh new thin entrypoint invoked by resume-relay
    tests/dev-lead/unit/test_dev_lead_resume.bats new 7 tests
    interaction-contracts/dev-lead.yml timer demoted, event_fast_path set, guards updated
    personas/dev-lead/interaction.yml comment: resume-relay bridge noted (deployment lens holds the emit)
    docs/agentic-interaction-model.md §4 row + §6.3 "Leak" → "Remediated leak" + cross-refs

    Notes

    • The resume is stricter than the cron: it skips on any of the three stop markers, not just needs-human-review, so a human owning the PR halts the event-first path.
    • Dispatch-emit modeling stays in the deployment lens only (matching the existing ci-relay convention), keeping the validator's self-trigger collision check meaningful.
    • No commits/pushes/PRs opened from this run — per the workflow, CI owns git.
  3. don-petry commented on Aug 2, 2026

    @don-petry
    Collaborator

    Dev-Lead: cannot implement issue #1407 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-1407-20260802-0900 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.

  4. don-petry commented on Aug 3, 2026

    @don-petry
    Collaborator

    ⚠️ The "Completed" comment above is not backed by any artifact

    Verified against origin/main: no branch, no PR, and none of the described changes exist.

    Claimed Reality
    scripts/dev-lead-resume.sh (new) absent
    tests/dev-lead/unit/test_dev_lead_resume.bats (new) absent
    resume_pr_event_first / resume-relay / resume_budget_exhausted zero matches in scripts/ or .github/workflows/
    dev-lead-retry.yml → timer_role: safety-net 0 matches; file last touched by unrelated older PRs
    branch / PR for #1407 neither exists

    The likely mechanism is benign — work done in the runner workspace, summary posted, then the stage timed out at 09:45:23Z before pushing, discarding the workspace. The timeout escalation is honest. The problem is that a highly specific completion record (5 checkmarked ACs, "726/726 pass", a six-file change table) was published before the work was durable and nothing retracted it.

    Treat this issue as not started. #1408 is blocked behind it, and it currently reads as delivered.

    The ordering/retraction defect is filed as #1445, and this issue is now blocked_by it so a re-attempt does not land on top of a misleading record.

    I have not removed dev-lead:needs-human — that is a human gate and yours to clear when you want the re-attempt.


    Generated with Claude Code

  5. 8 remaining items

  6. added 8 commits that reference this issue on Aug 7, 2026
  7. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Aug 8, 2026
  8. don-petry commented on Aug 8, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #1407 — [Phase 4] Demote dev-lead-retry from normal-path driver to a true backstop — resume blocked states event-first

    Scope

    Make blocked/rate-limited dev-lead states resume event-first (on a review submitted or a check_run success, via PAT repository_dispatch reusing the dev-lead-ci-relay.sh bridge pattern), gated by the per-PR automation budget + human stop markers, and demote the 2 h dev-lead-retry.yml cron to a documented timer_role: safety-net.

    Implementation Checklist

    Tests to Write

    • tests/dev-lead/unit/test_dev_lead_resume.bats — (a) a clearing event with a pending rate-limited marker dispatches a resume; (b) an exhausted budget suppresses it; (c) a needs-human-review marker suppresses it; (d) no marker → no dispatch.
    • tests/test_pr_automation_budget.bats — pr_resume_suppressed returns suppress on human label / exhausted budget, proceed otherwise.
    • Extend tests/dev-lead/unit/test_dev_lead_retry_pr_escalation.bats — the cron safety-net skips an exhausted-budget PR (not just the labeled one).

    Test Command

    bats tests/dev-lead/unit/ && scripts/run-bats.sh tests/test_pr_automation_budget.bats && python3 interaction-contracts/validate-interaction-contracts.py . && bash .dev-lead/scripts/dev-lead-lint.sh

  9. don-petry commented on Aug 8, 2026

    @don-petry
    Collaborator

    ✅ Phase 4 complete — dev-lead-retry demoted to a true backstop; blocked states now resume event-first

    Implemented the event-first resume path and demoted the 2 h cron to a documented timer_role: safety-net, closing the §6.3 leak. All five acceptance criteria are met.

    What changed

    New event-first resume bridge

    • scripts/dev-lead-resume.sh (new) — on a clearing event it resolves the PR and re-dispatches the pending status=rate-limited retry immediately, via the same PAT-backed repository_dispatch Bridge A pattern as dev-lead-ci-relay.sh. It sources dev-lead-retry.sh and calls the exact same scan_pr_for_rate_limits, so the dispatch and every stop-condition are identical to the cron path — the two paths cannot diverge.
    • .github/workflows/dev-lead-reusable.yml — new resume job fires on pull_request_review[submitted] or check_run[completed] && conclusion == success (excluding dev-lead's own checks). For check_run it reuses the sanctioned dev-lead-ci-relay.sh purely as a PR resolver (that script only writes should_relay/pr_number to GITHUB_OUTPUT; it does not dispatch), then runs dev-lead-resume.sh under the PAT identity.

    One shared stop-condition (AC #3)

    Cron demoted to safety-net (AC #2)

    • .github/workflows/dev-lead-retry.yml + scripts/dev-lead-retry.sh headers rewritten: timer_role: safety-net, "THIS IS A BACKSTOP, NOT THE CONVERGENCE CLOCK", stop-condition-before-acting (§6.2.1). Cron schedule unchanged (15 */2 * * *).

    Interaction contract updated (AC #4)

    • interaction-contracts/dev-lead.yml — timer role: self-heal → safety-net; event_fast_path set to the non-null resume description (§6.3 leak now CLOSED); added dev-lead-reusable.yml to workflows; self_trigger_guards for the two PR-retry dispatches now reference both emit paths via the shared pr_resume_suppressed gate.

    Frozen ring-0 stub untouched (AC #5)

    • Both trigger events (pull_request_review[submitted], check_run[completed]) are already forwarded by the frozen dev-lead.yml caller stub, so no stub edit was needed. caller_stub_freeze.sh reports dev-lead.yml ALIGNED.

    Scope note

    I support review submitted and check_run success as clearing events. I deliberately did not add pull_request_review_thread (resolved-thread) resume, because that event is not currently forwarded by the frozen ring-0 stub — adding it would require a freeze regeneration and carry channel-skew risk, which is out of scope for this issue.

    Tests (written first, all green)

    • tests/test_pr_automation_budget.bats — 5 new pr_resume_suppressed tests (label suppress, budget exhausted, clean proceed, label-wins, FORCE_REVIEW must not bypass).
    • tests/dev-lead/unit/test_dev_lead_resume.bats (new) — 5 tests: clearing event dispatches, exhausted budget suppresses, needs-human-review suppresses, no marker → no dispatch, empty PR_NUMBER clean no-op.
    • tests/dev-lead/unit/test_dev_lead_retry_pr_escalation.bats — added exhausted-budget-skip test.

    Verification (all pass)

    • python3 interaction-contracts/validate-interaction-contracts.py . → 11 contract(s) valid, invariants hold
    • bash scripts/caller_stub_freeze.sh → all ring-0 stubs ALIGNED
    • bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed (shellcheck 103 files + agent frontmatter)
    • bats tests/dev-lead/unit/ → 770 passing; contract/freeze/schedule/concurrency suites → 0 failures
    • tests/dev-lead/integration/test_concurrency_config.sh → all checks passed
    • Workflow YAML parses clean (dev-lead-reusable.yml, dev-lead-retry.yml, contract)

    No unrelated files touched; no test expectations weakened; no commits/pushes/PRs made (CI handles git).

  10. added a commit that references this issue on Aug 8, 2026
  11. added a commit that references this issue on Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions