Description
Story
As a platform engineer / org-lead,
I want implement the token-budget circuit breaker that pauses new agentic dispatch when the rolling 5-hour Claude subscription budget reaches the configured threshold (default 90%), prioritizing Claude-backed agents, using the telemetry source selected in the Phase-1 ADR,
so that agents stop before they exhaust the shared 5-hour Claude budget, so a runaway cannot fully block the fleet and humans — the failure mode raised in the discussion.
Acceptance Criteria
- A token-budget check, reading the org_wide token_budget block from standards/agent-rate-limits.json (threshold default 90%, Claude-priority flag), integrates into the agent-rate-limit gate so a dispatch is deferred when the rolling 5-hour Claude subscription budget is at/over the threshold.
- The check consumes the telemetry source selected in the Phase-1 ADR for the 5-hour window, behind a small adapter; if that source is unavailable at runtime it fails safe (documented default) rather than crashing the dispatch.
- When the breaker is tripped, Claude-backed agents are prioritized on the remaining budget / recovery per the config flag, and the pause is surfaced via the human-clearable marker/label used by the rest of the library.
- The breaker is exercised by tests with a mocked telemetry source (at-threshold defers, under-threshold allows, source-unavailable fails safe) — no live network in unit tests.
- Rollout is canary/dry-run first with a documented promotion path, and activation respects the Phase-2 human sign-off gate; the threshold stays config-driven (no hardcoded 90%).
- Story 5 is completable regardless of the ADR's telemetry finding: if no windowed 5-hour source exists, the gate hook still ships — either backed by a coarse proxy (the private-repo per-run token totals) or shipped disabled behind the config flag — with the chosen degraded mode documented, so no AC depends on a source that may not exist.
Tasks / Subtasks
- Add a token-budget evaluation to the gate reading the org_wide token_budget block (AC: Addressing PR comments #1)
- Consume the ADR-selected 5-hour telemetry source behind an adapter; fail safe when unavailable (AC: Add multi-agent isolation strategy using git worktrees #2)
- Implement Claude-priority-on-recovery and marker/label surfacing on trip (AC: Add coding standards & principles section to AGENTS.md #3)
- Add tests/test_agent_rate_limit_token_budget.bats with a mocked telemetry source (AC: Add workflow, environment, and orchestration guidance #4)
- Canary/dry-run rollout + promotion doc; keep the threshold config-driven (AC: Add stacked PR strategy and Epic-level workflow guidance #5)
- Guarantee a completable degraded path (coarse proxy or disabled-behind-flag) when no windowed telemetry exists, per the ADR (AC: feat: add Structured Logging and CQRS standards #6)
Dev Notes
- This implements the discussion comment's explicit ask: 'Pause agentic work when the 5 hour budget reaches 90%. Claude as priority.' The 5-hour window is the rolling Claude subscription cap shared across all Claude models (private repo scripts/engine.sh, issue Monitor and Optimize Token Usage #206). Today the engine only reacts AFTER hitting the cap (exit-2 walks the model fallback chain); this story adds the PROACTIVE pre-dispatch check.
- The telemetry source is unresolved until the Phase-1 ADR selects it — do NOT assume a native GitHub surface exists (there is none for the 5-hour Claude budget; the Actions Data Stream is a 2026 roadmap item, and the private repo's token-metrics.sh emits per-run ET/JSONL, not a windowed budget %). Build the check against the ADR's selected source behind a small adapter so the source can change without touching the gate.
- Token accounting lives in the PRIVATE repo (token-metrics.sh ET formula + JSONL, model-pricing.tsv, token_report.sh aggregation). Reading a 5-hour windowed % from it is a cross-repo concern flagged as an untracked prerequisite; the adapter boundary keeps this story shippable in the public repo while the telemetry wiring lands separately.
- Fail-safe: mirror the defensive degradation in the existing libs — a missing/malformed telemetry read must not crash the dispatch; choose a documented default (allow-with-warning vs defer) in the ADR and encode it.
- TDD (AGENTS.md): unit tests mock the telemetry source; no live network.
Project Structure Notes
Extends scripts/lib/agent-rate-limit.sh (from Phase 3) with a token-budget evaluation + adapter; new test under tests/. The actual 5-hour telemetry wiring is cross-repo (private) and tracked as an untracked prerequisite.
References
- standards/pr-limits.json (petry-projects/.github — org_wide config block pattern reused for token_budget)
- scripts/engine.sh (private repo — 5-hour subscription-cap handling / exit-2 fallback, Monitor and Optimize Token Usage #206, cross-repo)
- scripts/lib/token-metrics.sh (private repo — ET formula + JSONL token telemetry, cross-repo)
Likely target surface
scripts/lib/agent-rate-limit.sh (extend — add token-budget evaluation)tests/test_agent_rate_limit_token_budget.bats (new — mocked-telemetry tests)
Story prepared by the BMAD Scrum Master (Bob) for epic #636. Status: ready-for-dev.
⚠️ Post-ADR clarification (added 2026-08-31, after #637/PR #995, #638/PR #1017 and #639/PR #1018 all merged — read before implementing)
This story was written while the Phase-1 ADR was still open, so several ACs above hedge on questions that are now settled. Where they conflict, the merged ADR (docs/initiatives/agent-rate-limits-adr.md) and config (standards/agent-rate-limits.json) win.
1. The telemetry source is resolved — stop hedging
AC #6 and the Dev Notes say the source "is unresolved" and offer a coarse-proxy / shipped-disabled degraded path. Resolved: GET https://api.anthropic.com/api/oauth/usage, verified first-hand (HTTP 200, live window state). Build against it behind the adapter. The degraded path in AC #6 is no longer a completion hedge — keep only the adapter boundary.
Two load-bearing details from ADR §4.1:
User-Agent: claude-code/<version>is required, not cosmetic. Omitting it lands the caller in an aggressively rate-limited bucket with persistent 429s. Rate limiting is per access token; polling at ≥180 s is safe.- Prefer the
limits[]array over the flattenedfive_hour/seven_daykeys. Each entry is{ kind, group, percent, severity, resets_at, is_active }withkind ∈ session | weekly_all | weekly_scoped. Treat the flattened keys as a fallback for older response shapes.
2. "Fail safe" is now a decided direction, not a choice to make
Dev Notes say "choose a documented default (allow-with-warning vs defer) in the ADR and encode it." It is chosen: allow-with-warning. On telemetry error — non-200, malformed body, or a missing window entry — the breaker ALLOWS dispatch and logs a warning. An outage of an undocumented third-party endpoint must never stop the fleet; that is the failure mode this epic exists to prevent. The one exception, also from ADR §7: a fresh 429 with retry-after BLOCKS, because that is a hard first-hand signal the cap is already hit.
Phase 3 already implements this direction — mirror scripts/lib/agent-rate-limit.sh's arl_sanitize_int idiom (malformed degrades to an allowing value) rather than inventing a second convention. Prefer the words "allows dispatch" / "defers dispatch" over "fail-open"/"fail-closed"; ADR §7 avoids that vocabulary deliberately because it inverts confusingly.
3. The window is FIXED, and there is NO resume mark
The story says "rolling 5-hour" in the AC and Dev Notes. The session window is a FIXED window with an authoritative resets_at (ADR §4.1): it opens on first use, accumulates until resets_at, and old usage does not age out continuously. Within a window percent is monotonically non-decreasing.
Consequences that must be implemented, not just documented:
- No percentage resume mark, no half-open probe for this breaker. An earlier draft proposed resuming at 80%; it was removed because a downward dip cannot occur under a fixed window, making it unreachable.
standards/agent-rate-limits.jsondeliberately contains noresume_pctkey — do not add one. - The only resume for
sessionisresets_at. - This is not the same as the consecutive-failure breaker in Phase 3 (
#639), which legitimately keeps its half-open probe because a failure count genuinely recovers. Do not delete that one.
4. Scope guard
weekly_scoped (per-model exhaustion) is never pause-worthy — only session and weekly_all are. The config already encodes this as pause_worthy: false; assert it in a test rather than trusting the caller.
5. Read thresholds from config, and share the adapter with #994
org_wide.token_budget.limits.session.pause_threshold_pct (default 90) and claude_priority are already merged and inert — no workflow reads them yet. Never hardcode.
#994 (Phase 6, weekly glide-path) explicitly says: "If #641 lands first, extend its adapter; if this lands first, build the adapter to serve both windows from the single call." This story lands first — so build the adapter to serve both session and weekly_all from one call (ADR §4.1: "Both breakers share one call"). Do not build a session-only adapter that #994 then has to refactor.
Activity
don-petry commentedon Aug 20, 2026
Scope note — the weekly glide path was filed separately as #994, not folded in here.
A proposal came up to have the breaker read
/usageand tighten its threshold as the weekly reset approaches (pause Monday at 98%, Sunday at 96%, … reserving ~2% of the weekly budget per remaining day). That is a real and complementary control, but it is a different window (7-day vs this story's 5-hour), a time-varying rather than static threshold, and it needs calendar math plus auto-set/auto-clear of a pause switch — roughly doubling this story's surface. This one is BMAD-prepared and ready-for-dev; it stays as scoped.What does change here: the telemetry source is no longer unknown.
GET https://api.anthropic.com/api/oauth/usagewas verified first-hand (HTTP 200, live window data) and returns both windows in a single call — see the finding comment on #637 for the endpoint contract, response schema, and risks. Two consequences for this story:limits[].kind == "session"). Keep the fail-safe; the degraded-mode escape hatch is now insurance rather than the plan.Also worth encoding here: a
weekly_scoped(per-model) limit at critical must not trip a fleet pause — the right response to scoped exhaustion is the existing model-fallback chain. The probe caught exactly that case (one model's weekly bucket at 100% while the account-wide weekly window was at 85%).fix(adr): correct AC #4 — the subscription budget IS queryable [skip …
docs: resolve rolling-vs-fixed window contradiction in rate-limit ADR
feat: implement issue #637 — [Phase 1] ADR: agent rate-limit & circui…
don-petry commentedon Aug 31, 2026
Dev-Lead Implementation Plan
Issue: #641 — [Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window
Scope
Extend the Phase-3 gate library
scripts/lib/agent-rate-limit.shwith an org-wide token-budget breaker that defers new agentic dispatch when the fixed 5-hour Claudesessionwindow is at/over the config-drivenpause_threshold_pct(default 90%), reading the ADR §4.1 OAuth-usage telemetry through a small, mockable adapter seam that serves both thesessionandweekly_allwindows from one call (so #994 extends, not refactors); fail-safe allows on telemetry error, blocks on a fresh 429+retry-after; ships inert/disabled behind a flag (canary/dry-run rollout) so the provisional config stays inert until human sign-off.Implementation Checklist
scripts/lib/agent-rate-limit.sh— config accessors:arl_token_pause_threshold,arl_token_window_pause_worthy,arl_token_claude_priority,arl_token_priority_rank(Claude-priority-on-recovery, AC Add coding standards & principles section to AGENTS.md #3)scripts/lib/agent-rate-limit.sh— adapter seam:arl_token_fetch_envelope(CMD/FILE injection, default-unavailable), pure parsersarl_token_extract_percent/arl_token_window_active(preferlimits[], fall back to flattened keys) (AC Add multi-agent isolation strategy using git worktrees #2)scripts/lib/agent-rate-limit.sh— pure corearl_token_budget_decision+ orchestratorarl_token_budget_gate(fail-safe allow on error; block on fresh 429; scope-guard non-pause-worthy windows) (AC Addressing PR comments #1/Add multi-agent isolation strategy using git worktrees #2/feat: add Structured Logging and CQRS standards #6)scripts/lib/agent-rate-limit.sh— marker/label surfacing:arl_token_breaker_marker+arl_token_should_escalate, reusearl_breaker_label(AC Add coding standards & principles section to AGENTS.md #3)scripts/lib/agent-rate-limit.sh— integrate intoarl_admission_gate, gated byAGENT_TOKEN_BUDGET_ENABLED(off by default → inert; canary activation) (AC Addressing PR comments #1/Add stacked PR strategy and Epic-level workflow guidance #5)standards/agent-rate-limits.md— document the canary/dry-run rollout + promotion path and the degraded/disabled-behind-flag mode (AC Add stacked PR strategy and Epic-level workflow guidance #5/feat: add Structured Logging and CQRS standards #6).github/workflows/agent-rate-limits-tests.yml— add the new test file to the paths + bats run listTests to Write
tests/test_agent_rate_limit_token_budget.bats— at-threshold defers, under-threshold allows, source-unavailable/malformed/missing-window fail-safe allows (AC Add workflow, environment, and orchestration guidance #4)retry-afterblocks (ADR §7)weekly_scoped/ non-pause-worthy window never defers (scope guard, post-ADR Add workflow, environment, and orchestration guidance #4)limits[], falls back to flattenedfive_hour; servessessionandweekly_allfrom one envelope (post-ADR Add stacked PR strategy and Epic-level workflow guidance #5)arl_admission_gateinert whenAGENT_TOKEN_BUDGET_ENABLEDunset; defers when enabled + tripped (AC Addressing PR comments #1/Add stacked PR strategy and Epic-level workflow guidance #5)ghcall;set -euo pipefail-safeTest Command
bats --print-output-on-failure tests/test_agent_rate_limit_token_budget.bats tests/test_agent_rate_limit.bats tests/test_agent_rate_limits_config.batsdon-petry commentedon Aug 31, 2026
Dev-Lead: Implementation Complete
Implements the org-wide token-budget circuit breaker — the proactive pre-dispatch check that pauses new agentic dispatch when the shared, fixed 5-hour Claude
sessionwindow is at/over the config-drivenpause_threshold_pct(default 90%), reading the ADR §4.1 OAuth-usage telemetry through a small, mockable adapter seam. Shipped inert behind a flag so the provisional config stays inert until human sign-off.Plan Execution
arl_token_pause_threshold,arl_token_window_pause_worthy,arl_token_claude_priority,arl_token_priority_rank(AC Add coding standards & principles section to AGENTS.md #3/Add stacked PR strategy and Epic-level workflow guidance #5)arl_token_fetch_envelope(CMD/FILE injection, default-unavailable) + pure parsersarl_token_extract_percent/arl_token_window_active— preferlimits[], fall back to flattened keys; serves bothsessionandweekly_allfrom one call so [Phase 6] Weekly-budget glide-path breaker: pause at 100 − 2×days-until-reset of the 7-day Claude window #994 extends it (AC Add multi-agent isolation strategy using git worktrees #2, post-ADR Add stacked PR strategy and Epic-level workflow guidance #5)arl_token_budget_decision+ orchestratorarl_token_budget_gate: fail-safe allows on telemetry error, blocks on a fresh 429+retry-after, scope-guards non-pause-worthy windows (AC Addressing PR comments #1/Add multi-agent isolation strategy using git worktrees #2/feat: add Structured Logging and CQRS standards #6, ADR §7/§2.5)arl_token_breaker_marker+arl_token_should_escalate, reusingneeds-human-review(AC Add coding standards & principles section to AGENTS.md #3)arl_admission_gate, gated byAGENT_TOKEN_BUDGET_ENABLED(off by default → inert; canary/dry-run rollout) (AC Addressing PR comments #1/Add stacked PR strategy and Epic-level workflow guidance #5)standards/agent-rate-limits.md§4.1/§4.2; degraded/disabled-behind-flag mode documented (AC Add stacked PR strategy and Epic-level workflow guidance #5/feat: add Structured Logging and CQRS standards #6)tests/test_agent_rate_limit_token_budget.batswired into the CI workflowTest Results
Full repo bats suite: 448/448 pass, 0 failures.
.dev-lead/scripts/dev-lead-lint.sh: pass (shellcheck --severity=warning clean on 30 files; CODEOWNERS OK).Files Changed
scripts/lib/agent-rate-limit.sh: added the token-budget breaker (config accessors, telemetry adapter seam, pure decision + orchestrator, marker/label surfacing) and integrated it intoarl_admission_gatebehindAGENT_TOKEN_BUDGET_ENABLED.tests/test_agent_rate_limit_token_budget.bats(new): 42 tests with a mocked telemetry source — at-threshold defers, under-threshold allows, source-unavailable/malformed/missing-window/inactive fail safe, fresh-429 blocks, scope guard, config-driven threshold, adapter fallback, integration + inert-behind-flag, no-mutating-gh,set -euo pipefail-safe..github/workflows/agent-rate-limits-tests.yml: run the new test file; updated paths + header.standards/agent-rate-limits.md: §4.1 (where the breaker lives + adapter seam) and §4.2 (canary/dry-run → promotion path, degraded mode).Notes
AGENT_TOKEN_BUDGET_TELEMETRY_CMD/_FILE) the private poller plugs the live read into — so no unit test touches the network.standards/agent-rate-limits.jsonat run time; a test exercises a non-default (75/40) threshold to prove it..github-private#1565,#1525) and the Phase-6 weekly glide-path breaker ([Phase 6] Weekly-budget glide-path breaker: pause at 100 − 2×days-until-reset of the 7-day Claude window #994, which extends this adapter).feat: implement issue #641 — [Phase 5] Token-budget circuit breaker: …