Skip to content

[Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641

Closed

Description

@don-petry

Story

As a platform engineer / org-lead,
I want implement the token-budget circuit breaker that pauses new agentic dispatch when the rolling 5-hour Claude subscription budget reaches the configured threshold (default 90%), prioritizing Claude-backed agents, using the telemetry source selected in the Phase-1 ADR,
so that agents stop before they exhaust the shared 5-hour Claude budget, so a runaway cannot fully block the fleet and humans — the failure mode raised in the discussion.

Acceptance Criteria

  1. A token-budget check, reading the org_wide token_budget block from standards/agent-rate-limits.json (threshold default 90%, Claude-priority flag), integrates into the agent-rate-limit gate so a dispatch is deferred when the rolling 5-hour Claude subscription budget is at/over the threshold.
  2. The check consumes the telemetry source selected in the Phase-1 ADR for the 5-hour window, behind a small adapter; if that source is unavailable at runtime it fails safe (documented default) rather than crashing the dispatch.
  3. When the breaker is tripped, Claude-backed agents are prioritized on the remaining budget / recovery per the config flag, and the pause is surfaced via the human-clearable marker/label used by the rest of the library.
  4. The breaker is exercised by tests with a mocked telemetry source (at-threshold defers, under-threshold allows, source-unavailable fails safe) — no live network in unit tests.
  5. Rollout is canary/dry-run first with a documented promotion path, and activation respects the Phase-2 human sign-off gate; the threshold stays config-driven (no hardcoded 90%).
  6. Story 5 is completable regardless of the ADR's telemetry finding: if no windowed 5-hour source exists, the gate hook still ships — either backed by a coarse proxy (the private-repo per-run token totals) or shipped disabled behind the config flag — with the chosen degraded mode documented, so no AC depends on a source that may not exist.

Tasks / Subtasks

Dev Notes

  • This implements the discussion comment's explicit ask: 'Pause agentic work when the 5 hour budget reaches 90%. Claude as priority.' The 5-hour window is the rolling Claude subscription cap shared across all Claude models (private repo scripts/engine.sh, issue Monitor and Optimize Token Usage #206). Today the engine only reacts AFTER hitting the cap (exit-2 walks the model fallback chain); this story adds the PROACTIVE pre-dispatch check.
  • The telemetry source is unresolved until the Phase-1 ADR selects it — do NOT assume a native GitHub surface exists (there is none for the 5-hour Claude budget; the Actions Data Stream is a 2026 roadmap item, and the private repo's token-metrics.sh emits per-run ET/JSONL, not a windowed budget %). Build the check against the ADR's selected source behind a small adapter so the source can change without touching the gate.
  • Token accounting lives in the PRIVATE repo (token-metrics.sh ET formula + JSONL, model-pricing.tsv, token_report.sh aggregation). Reading a 5-hour windowed % from it is a cross-repo concern flagged as an untracked prerequisite; the adapter boundary keeps this story shippable in the public repo while the telemetry wiring lands separately.
  • Fail-safe: mirror the defensive degradation in the existing libs — a missing/malformed telemetry read must not crash the dispatch; choose a documented default (allow-with-warning vs defer) in the ADR and encode it.
  • TDD (AGENTS.md): unit tests mock the telemetry source; no live network.

Project Structure Notes

Extends scripts/lib/agent-rate-limit.sh (from Phase 3) with a token-budget evaluation + adapter; new test under tests/. The actual 5-hour telemetry wiring is cross-repo (private) and tracked as an untracked prerequisite.

References

  • standards/pr-limits.json (petry-projects/.github — org_wide config block pattern reused for token_budget)
  • scripts/engine.sh (private repo — 5-hour subscription-cap handling / exit-2 fallback, Monitor and Optimize Token Usage #206, cross-repo)
  • scripts/lib/token-metrics.sh (private repo — ET formula + JSONL token telemetry, cross-repo)

Likely target surface

  • scripts/lib/agent-rate-limit.sh (extend — add token-budget evaluation)
  • tests/test_agent_rate_limit_token_budget.bats (new — mocked-telemetry tests)

Story prepared by the BMAD Scrum Master (Bob) for epic #636. Status: ready-for-dev.


⚠️ Post-ADR clarification (added 2026-08-31, after #637/PR #995, #638/PR #1017 and #639/PR #1018 all merged — read before implementing)

This story was written while the Phase-1 ADR was still open, so several ACs above hedge on questions that are now settled. Where they conflict, the merged ADR (docs/initiatives/agent-rate-limits-adr.md) and config (standards/agent-rate-limits.json) win.

1. The telemetry source is resolved — stop hedging

AC #6 and the Dev Notes say the source "is unresolved" and offer a coarse-proxy / shipped-disabled degraded path. Resolved: GET https://api.anthropic.com/api/oauth/usage, verified first-hand (HTTP 200, live window state). Build against it behind the adapter. The degraded path in AC #6 is no longer a completion hedge — keep only the adapter boundary.

Two load-bearing details from ADR §4.1:

  • User-Agent: claude-code/<version> is required, not cosmetic. Omitting it lands the caller in an aggressively rate-limited bucket with persistent 429s. Rate limiting is per access token; polling at ≥180 s is safe.
  • Prefer the limits[] array over the flattened five_hour/seven_day keys. Each entry is { kind, group, percent, severity, resets_at, is_active } with kind ∈ session | weekly_all | weekly_scoped. Treat the flattened keys as a fallback for older response shapes.

2. "Fail safe" is now a decided direction, not a choice to make

Dev Notes say "choose a documented default (allow-with-warning vs defer) in the ADR and encode it." It is chosen: allow-with-warning. On telemetry error — non-200, malformed body, or a missing window entry — the breaker ALLOWS dispatch and logs a warning. An outage of an undocumented third-party endpoint must never stop the fleet; that is the failure mode this epic exists to prevent. The one exception, also from ADR §7: a fresh 429 with retry-after BLOCKS, because that is a hard first-hand signal the cap is already hit.

Phase 3 already implements this direction — mirror scripts/lib/agent-rate-limit.sh's arl_sanitize_int idiom (malformed degrades to an allowing value) rather than inventing a second convention. Prefer the words "allows dispatch" / "defers dispatch" over "fail-open"/"fail-closed"; ADR §7 avoids that vocabulary deliberately because it inverts confusingly.

3. The window is FIXED, and there is NO resume mark

The story says "rolling 5-hour" in the AC and Dev Notes. The session window is a FIXED window with an authoritative resets_at (ADR §4.1): it opens on first use, accumulates until resets_at, and old usage does not age out continuously. Within a window percent is monotonically non-decreasing.

Consequences that must be implemented, not just documented:

  • No percentage resume mark, no half-open probe for this breaker. An earlier draft proposed resuming at 80%; it was removed because a downward dip cannot occur under a fixed window, making it unreachable. standards/agent-rate-limits.json deliberately contains no resume_pct key — do not add one.
  • The only resume for session is resets_at.
  • This is not the same as the consecutive-failure breaker in Phase 3 (#639), which legitimately keeps its half-open probe because a failure count genuinely recovers. Do not delete that one.

4. Scope guard

weekly_scoped (per-model exhaustion) is never pause-worthy — only session and weekly_all are. The config already encodes this as pause_worthy: false; assert it in a test rather than trusting the caller.

5. Read thresholds from config, and share the adapter with #994

org_wide.token_budget.limits.session.pause_threshold_pct (default 90) and claude_priority are already merged and inert — no workflow reads them yet. Never hardcode.

#994 (Phase 6, weekly glide-path) explicitly says: "If #641 lands first, extend its adapter; if this lands first, build the adapter to serve both windows from the single call." This story lands first — so build the adapter to serve both session and weekly_all from one call (ADR §4.1: "Both breakers share one call"). Do not build a session-only adapter that #994 then has to refactor.

Activity

  1. don-petry commented on Aug 20, 2026

    @don-petry
    ContributorAuthor

    Scope note — the weekly glide path was filed separately as #994, not folded in here.

    A proposal came up to have the breaker read /usage and tighten its threshold as the weekly reset approaches (pause Monday at 98%, Sunday at 96%, … reserving ~2% of the weekly budget per remaining day). That is a real and complementary control, but it is a different window (7-day vs this story's 5-hour), a time-varying rather than static threshold, and it needs calendar math plus auto-set/auto-clear of a pause switch — roughly doubling this story's surface. This one is BMAD-prepared and ready-for-dev; it stays as scoped.

    What does change here: the telemetry source is no longer unknown. GET https://api.anthropic.com/api/oauth/usage was verified first-hand (HTTP 200, live window data) and returns both windows in a single call — see the finding comment on #637 for the endpoint contract, response schema, and risks. Two consequences for this story:

    Also worth encoding here: a weekly_scoped (per-model) limit at critical must not trip a fleet pause — the right response to scoped exhaustion is the existing model-fallback chain. The probe caught exactly that case (one model's weekly bucket at 100% while the account-wide weekly window was at 85%).

  2. added a commit that references this issue on Aug 21, 2026
  3. added 2 commits that reference this issue on Aug 30, 2026
  4. don-petry commented on Aug 31, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #641 — [Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window

    Scope

    Extend the Phase-3 gate library scripts/lib/agent-rate-limit.sh with an org-wide token-budget breaker that defers new agentic dispatch when the fixed 5-hour Claude session window is at/over the config-driven pause_threshold_pct (default 90%), reading the ADR §4.1 OAuth-usage telemetry through a small, mockable adapter seam that serves both the session and weekly_all windows from one call (so #994 extends, not refactors); fail-safe allows on telemetry error, blocks on a fresh 429+retry-after; ships inert/disabled behind a flag (canary/dry-run rollout) so the provisional config stays inert until human sign-off.

    Implementation Checklist

    Tests to Write

    Test Command

    bats --print-output-on-failure tests/test_agent_rate_limit_token_budget.bats tests/test_agent_rate_limit.bats tests/test_agent_rate_limits_config.bats

  5. don-petry commented on Aug 31, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Implements the org-wide token-budget circuit breaker — the proactive pre-dispatch check that pauses new agentic dispatch when the shared, fixed 5-hour Claude session window is at/over the config-driven pause_threshold_pct (default 90%), reading the ADR §4.1 OAuth-usage telemetry through a small, mockable adapter seam. Shipped inert behind a flag so the provisional config stays inert until human sign-off.

    Plan Execution

    Test Results

    ok 115 the token-budget windows are never described as 'rolling' (post-ADR #1)
    ok 116 exempt_actors is a non-empty array
    ok 117 exemption sets match pr-limits (same-set contract)
    ok 118 dependabot[bot] is present in the exempt-actor list (reused from pr-limits)
    ok 119 org-leads break-glass actor is present in the exempt-actor list
    ok 120 security is present in the exempt-label list
    

    Full repo bats suite: 448/448 pass, 0 failures. .dev-lead/scripts/dev-lead-lint.sh: pass (shellcheck --severity=warning clean on 30 files; CODEOWNERS OK).

    Files Changed

    • scripts/lib/agent-rate-limit.sh: added the token-budget breaker (config accessors, telemetry adapter seam, pure decision + orchestrator, marker/label surfacing) and integrated it into arl_admission_gate behind AGENT_TOKEN_BUDGET_ENABLED.
    • tests/test_agent_rate_limit_token_budget.bats (new): 42 tests with a mocked telemetry source — at-threshold defers, under-threshold allows, source-unavailable/malformed/missing-window/inactive fail safe, fresh-429 blocks, scope guard, config-driven threshold, adapter fallback, integration + inert-behind-flag, no-mutating-gh, set -euo pipefail-safe.
    • .github/workflows/agent-rate-limits-tests.yml: run the new test file; updated paths + header.
    • standards/agent-rate-limits.md: §4.1 (where the breaker lives + adapter seam) and §4.2 (canary/dry-run → promotion path, degraded mode).

    Notes

    • Cross-repo boundary honored (ADR §8.2): the credentialed OAuth read stays private. The public library holds thresholds/policy/decision + the adapter seam (AGENT_TOKEN_BUDGET_TELEMETRY_CMD/_FILE) the private poller plugs the live read into — so no unit test touches the network.
    • No hardcoded 90% — the threshold is read from standards/agent-rate-limits.json at run time; a test exercises a non-default (75/40) threshold to prove it.
    • Inert on merge: with the flag unset (and with no telemetry seam wired) the gate reads no telemetry and cannot change any dispatch decision, mirroring the pr-limits sign-off staging.
    • Follow-up (not in scope): the private telemetry poller/adapter + fleet-pause actuation (.github-private#1565, #1525) and the Phase-6 weekly glide-path breaker ([Phase 6] Weekly-budget glide-path breaker: pause at 100 − 2×days-until-reset of the 7-day Claude window #994, which extends this adapter).
  6. added a commit that references this issue on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions