Skip to content

feat(budget-breaker): wake on agent workflow runs, keep the schedule as a backstop - #2293

Merged
don-petry merged 3 commits into
mainfrom
claude/issue-2259-breaker-workflow-run
Oct 11, 2026
Merged

don-petry merged 3 commits into
mainfrom
claude/issue-2259-breaker-workflow-run

Conversation

@don-petry

@don-petry don-petry commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The budget breaker is schedule-only so that only default-branch code ever runs with its write credential. But GitHub delivers almost none of this repository's scheduled ticks: in the first 13 hours after the breaker merged it ran twice (08:35 and 15:03 UTC on 2026-10-11), 6.5 hours apart, against a 10-minute cron. A control that may not run for hours can neither pause the fleet during a quota outage nor, once armed, lift its own pause on time. This blocks arming ("Auto-pause companion", #1565).

Design

The maintainer agreed in principle on 2026-10-11: keep the schedule as a backstop and add a workflow_run trigger.

Trigger. workflow_run, types: [completed], on three workflows in this repository:

Workflow Why
"Dev-Lead Agent" Runs the three scripts that post a dev-lead quota marker
"GitHub Agentic Workflows — Cross-Org Dispatcher" Runs two of those scripts directly; no runs in retained history, so no cost today
"Dev-Lead Retry (rate-limited)" Posts no quota marker; listed as one more wake-up that does not depend on agent traffic

pr-review's workflows are deliberately absent: its rate-limited marker is not quota evidence.

Why this keeps the default-branch-only property. GitHub's "Events that trigger workflows" page (read 2026-10-11) says of workflow_run: "This event will only trigger a workflow run if the workflow file exists on the default branch", with GITHUB_SHA the "Last commit on default branch". The same page warns that such a workflow "is able to access secrets and write tokens, even if the previous workflow was not". So the triggering run is treated as untrusted and is never read:

  • The workflow file contains no reference to the event context at all: not in if:, env:, with:, run: or concurrency:.
  • The checkout has no ref: and no repository:. No artifact is downloaded. The script reads the event's name from the runner for its log and nothing else of the event.
  • Permissions, the three credentials and the concurrency group are unchanged.

What an outsider gains. Anyone who can make a listed workflow's run complete here (a comment, a fork pull request, a pull request adding a workflow that carries a listed name) now chooses when the breaker runs. They choose nothing else: quota markers still count only from owners, members and collaborators, and the decision is idempotent.

Cost control: an early exit in the script, not a job-level if:. About half of today's Dev-Lead Agent completions are runs cancelled in their lane. Two reasons there is no conclusion filter:

  • A paused Dev-Lead Agent run has no notice job and ends skipped. Filtering skipped would drop exactly the runs a resume relies on.
  • The concurrency group keeps one pending run and a newer arrival replaces it. GitHub does not document whether a run whose only job is skipped takes that place. If it does, a wake-up that does nothing would replace one that carries news.

Instead, a run woken by workflow_run on a fleet that is not paused exits before the organisation scan when both hold: an earlier run of this workflow was created in the current clock slot (BUDGET_BREAKER_MIN_INTERVAL_MIN, default 2 minutes, 0 switches it off), and this repository shows no quota marker created in that slot. It never applies to a scheduled run, never while the fleet is paused, and never when an input could not be read. The rule is clock slots rather than "the previous run finished less than N minutes ago" because a run that exits early is itself a previous run; under a steady stream of wake-ups that rule would exit early for ever.

Visibility. Every run logs, and puts in its summary, which event woke it and how long ago the workflow last ran to a conclusion.

Risk

  • Run volume. Replaying today's 988 Dev-Lead Agent completions through the concurrency group gives about 47 breaker runs an hour, each a job of roughly ten seconds, most taking the early exit. One runner slot at most is in use at a time.
  • Feedback loop, checked and not found. Dev-Lead Agent also triggers on check_run. If the breaker's own job produced a check_run event that started Dev-Lead Agent, its completion would wake the breaker again. In the two real breaker runs so far (job completed 08:35:45 and 15:05:43 UTC) no check_run-triggered Dev-Lead Agent run followed within a minute. Two data points; see Monitoring.
  • Not observed, from the documentation only: that workflow_run fires when the triggering run ended skipped. It was observed here for cancelled runs (8 of 8 cancelled Auto-rebase runs were followed by their workflow_run consumer within 3 seconds). The skipped case is what wakes a resume; it is added to the arming preconditions in AGENTS.md and can be observed after merge without a pause.
  • Worst cases are improved, not bounded. A trip driven by markers in this repository takes under a minute with a free runner and a few minutes when the job cap is full (one breaker run waited 144 seconds for a runner). Markers only in other repositories, or a quiet repository, still wait for the next wake-up of any kind; today contains a gap of 2 hours 54 minutes between Dev-Lead Agent completions. A resume during a quiet pause can still be hours late. AGENTS.md states this with the measurements.
  • The repository's fork pull request approval setting was not inspected.
  • The breaker stays disarmed. BUDGET_BREAKER_ARMED is unset, and this change does not claim arming is unblocked: the live checks on "Budget breaker: close the gaps found in review before it is armed" (Budget breaker: close the gaps found in review before it is armed (race with a human pause, pagination, hold on a bad timestamp, lost notice, live API checks) #2251) are still outstanding.

Test plan

  • tests/dev-lead/unit/test_budget_breaker_writer.bats: 97 tests (83 before; 3 new static, 11 new behaviour).
    • Triggers are exactly schedule and workflow_run; every other trigger is rejected, as are types: [requested] and a branches: filter.
    • Every listed workflow name is the name: of a workflow file here, and the list is pinned.
    • No event-context reference anywhere in the workflow; no ref: or repository: on the checkout; no artifact download; the scripts read only the event's name.
    • Early exit: never while paused, never on a scheduled run, fails open on every unreadable input, cannot starve the decision.
  • test_budget_breaker.bats 39 of 39 and tests/test_fleet_pause_coverage.bats 60 of 60, unchanged.
  • Mutation-checked: 31 mutations, each caught by a named test. I re-ran four independently after rebasing (add workflow_dispatch; checkout the triggering SHA; a job if: on the triggering conclusion; early exit moved ahead of the paused check): each turned the expected test red, and the suite returned to 97 of 97.
  • actionlint and shellcheck --severity=warning -x clean.
  • A disarmed dry run of the script against the live API (GET only) printed the trigger and the previous-run line, and took the early exit with a long slot.
  • Not run: anything on GitHub itself. Nothing was dispatched.

Rollback

Revert the squash commit: the workflow goes back to schedule-only. Faster, without a merge: set the repository variable BUDGET_BREAKER_MIN_INTERVAL_MIN to a large value to make nearly every event-woken run exit early (this does not stop the runs). The breaker is disarmed either way, so a rollback changes logging only.

Monitoring

  • Within minutes of merge, "Budget Breaker" runs should appear with event workflow_run. Their summaries show "Triggered by" and "Previous run".
  • Watch the first hour for a loop: breaker runs arriving at a steady interval of well under a minute with no other Dev-Lead Agent activity would mean the breaker is waking itself. Expected rate is about 47 an hour on a busy day, in bursts that follow agent activity.
  • Record on Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 the first workflow_run breaker run that follows a Dev-Lead Agent run which ended skipped (arming precondition 6).
  • Scheduled runs should continue to appear occasionally; their absence over a day is the delivery problem this change works around, not a regression.

Closes #2259

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

…as a backstop

The breaker was schedule-only so that only default-branch code runs with its
write credential, but GitHub delivers few scheduled ticks here (two in the
first 13 hours after it merged), so it could neither trip nor resume in time.

- Triggers are now exactly schedule and workflow_run (completed) for the
  workflows in this repository that can post a dev-lead quota marker, plus the
  retry sweep as one more wake-up. workflow_run runs the default branch's
  workflow file and commit; the triggering run is never read: no reference to
  the event context anywhere in the workflow, no ref or repository on the
  checkout, and the script reads the event's name only.
- No job-level filter on the triggering run's conclusion: it would need the
  event context, and a run that only skips may take the concurrency group's
  one pending place from a run that carries news.
- Early exit before the organisation scan for a workflow_run wake-up on a
  fleet that is not paused, when an earlier run was created in the same clock
  slot (BUDGET_BREAKER_MIN_INTERVAL_MIN, default 2) and this repository shows
  no quota marker created in that slot. Never on a scheduled run, never while
  paused, and never when an input could not be read. The earlier runs are
  listed with the read credential; permissions are unchanged.
- Every run logs and summarises what woke it and how long ago the workflow
  last ran to a conclusion.
- AGENTS.md: the trigger rule, why workflow_run keeps the default-branch-only
  property, measured reaction time and resume delay, the cross-repository
  limit, and one more arming precondition. The interaction-model row is now
  Class 2 with a backstop timer.

Refs #2259

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 11, 2026 16:10
@don-petry don-petry added the dev-lead:hands-off Exclude from dev-lead agent automation label Oct 11, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 34311e8f-b133-4cb6-9a15-593bc16b24f5

📥 Commits

Reviewing files that changed from the base of the PR and between 684e4b9 and efdd945.


📒 Files selected for processing (6)
  • .github/workflows/budget-breaker.yml
  • AGENTS.md
  • docs/agentic-interaction-model.md
  • scripts/budget_breaker.sh
  • scripts/lib/budget-breaker-writer.sh
  • tests/dev-lead/unit/test_budget_breaker_writer.bats

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an 'early exit' mechanism for the budget breaker workflow to reduce unnecessary organization scans when multiple runs occur within the same time slot. It adds a 'workflow_run' trigger to the workflow, implements logic to check for recent runs and quota markers, and updates the documentation and test suite to reflect these changes. I have kept the reviewer's comments as they provide actionable feedback regarding test robustness and file extension handling.

@codeant-ai

codeant-ai Bot commented Oct 11, 2026

Copy link
Copy Markdown

CodeAnt Nitpicks

2 code suggestions

1. When another organization repository gets a quota marker after the prior scan, this local-only check can skip it; the marker may expire before another full scan.

Logic error · scripts/budget_breaker.sh:256-257


2. success or failure does not prove a scan ran: checkout failures and successful leftover-marker exits skip scanning, yet later wake-ups count them as coverage and omit the decision.

Incorrect condition logic · scripts/lib/budget-breaker-writer.sh:741-743

@don-petry

Copy link
Copy Markdown
Collaborator Author

dev-lead is withholding action on this item.

It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

To re-enable automated pickup: remove the dev-lead:hands-off label.

@don-petry

Copy link
Copy Markdown
Collaborator Author

QA Lead — test-risk advisory

Risk tier: LOW. The breaker is still disarmed, so in production this change only alters when it wakes, what it logs and how many runs it costs. The new trust boundary (an untrusted workflow_run with a write credential) has static tests and was mutation-checked 31 times.

What I'd shore up (highest leverage first):

  • A failed run should not count as covering a slot. bbw_read_previous_runs treats failure the same as success. If the first run in a slot fails during its organisation scan, every later wake-up in that slot takes the early exit and the slot gets no decision. Either count only success, or add a test that pins down the current behaviour on purpose.
  • Test the slot edges. Add a fixture where the earlier run's created_at falls exactly on the slot start, and one where NOW and created_at sit on either side of a boundary (runner clock against API clock). The >= comparison decides both, and no named test covers them.
  • Make the arming check verify itself. Whether workflow_run fires after a skipped run is the whole resume path, and today the only evidence is the documentation. Record the first observed run ID on Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 as a pass/fail condition for arming, not a monitoring note.
  • Watch for a self-trigger loop. Two data points can't rule out a loop through check_run. A one-line alert, such as more than N breaker runs in 10 minutes with no Dev-Lead Agent run in that window, would turn the "watch the first hour" step into a guard that keeps running.

Escalate? No.

Advisory only — I comment, I do not change code. Ground: BMAD Test Architecture (bmad-tea v1.19.0). Opt out on this item with the qa-lead:hands-off label.

@codeant-ai

codeant-ai Bot commented Oct 11, 2026

Copy link
Copy Markdown

CodeAnt PR Risk: Medium Risk

  • The PR needs attention before merging because the early-exit guard can treat a failed prior run as scan coverage.
  • A failed run may end before the organization scan, yet bbw_read_previous_runs counts failures toward same-slot coverage, allowing later wake-ups to skip the scan.

Assessed commit: 684c3ec0ba1b

don-petry and others added 2 commits October 11, 2026 11:59
… exit

The early exit counted any earlier run that reached a conclusion. A run that
failed may have ended before its organisation scan, so every later wake-up in
that slot exited early and the slot got no decision. Only a run that succeeded
covers a slot now. From the QA advisory and CodeAnt on the pull request.

Tests: the failed-run case end to end, and the slot's edges (created exactly on
the slot start, one second before it, the last second, and a boundary between
the runner's clock and the API's). The workflow-name test also reads .yaml.

Refs #2259

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry

Copy link
Copy Markdown
Collaborator Author

Review round 1, head efdd9452:

  • A failed run no longer covers a slot (QA advisory, CodeAnt). The early exit now counts only an earlier run that succeeded. A failed run may have ended before its scan, and counting it let every later wake-up in that slot exit with no decision taken. Tested end to end and in the pure helper; restoring the old rule turns two tests red.
  • Slot edges are tested (QA advisory): created exactly on the slot start, one second before it, the last second of the slot, and the runner's clock a second past a boundary the API stamped before. Changing >= to > turns the test red.
  • The arming check is already a pass/fail condition (QA advisory): arming precondition 6 in AGENTS.md requires the run ids on Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565, including a breaker run that starts within seconds of a Dev-Lead Agent run that ended skipped.
  • Not done here: a standing alert for a self-trigger loop (QA advisory). It is a fair point that two data points cannot rule a loop out. It belongs with the fleet's runaway detection rather than in this workflow; I will look at the first hour of runs after merge and file it with what that shows.
  • Full writer suite: 97 of 97. main is merged in.

@sonarqubecloud

Copy link
Copy Markdown

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T18:19:10Z.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-lead:hands-off Exclude from dev-lead agent automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Budget breaker depends on scheduled ticks that this repository barely receives: add a default-branch-only event trigger before arming

2 participants