Skip to content

Gemini trickle under a budget pause: let a bounded slice of cheap work run on Gemini while the Claude breaker has paused the fleet (default off) #2147

Open

Description

@don-petry

Carried over from the review of pull request 2150 — a precondition for this issue

The Gemini tier snapshot and history are built from token-usage artifacts collected across the organisation. Those artifacts are untrusted: a pull request run from a fork in a public repository can upload one. Today that is harmless, because the snapshot is informational only. The moment anything gates or defers dispatch on it (which is what this issue starts to do), forged gemini_attempt records or malformed token counts could mark a tier exhausted or unavailable and deny Gemini to the fleet.

Before any dispatch decision reads gq_tier_can_run or the snapshot, records must be traceable to trusted runs (default branch, not a fork), and anything else must be ignored. Treat that as the first acceptance criterion here.


Decision this implements

Maintainer decision (2026-10-09): when the Claude budget breaker trips, pausing the fleet is the primary control, and Gemini is only a small trickle for cheap work. Spend on Gemini may be increased later, so the size of the trickle must be set by data (the caps file), never by code. This supersedes the earlier note on #1565 that the fleet would "degrade to Gemini".

Why this is not a config change

AGENTS_PAUSED=true is a hard stop. It is honoured in .github/workflows/dev-lead-reusable.yml, .github/workflows/pr-review.yml, .github/workflows/engine-token-liveness.yml and scripts/validate-engines.sh, where a pause sets CLAUDE_AVAILABLE, GEMINI_AVAILABLE and COPILOT_AVAILABLE all to false and skips availability probes. A trickle therefore needs an explicit, narrow exemption to the pause, not just a different engine order.

Constraints

  1. Default off. A new Actions variable GEMINI_TRICKLE_ENABLED (default false, absent means false). Merging the code must change no behaviour until the maintainer sets it. Same arming pattern as the dry-run poller.
  2. Only a budget pause can be exempted. The exemption applies only when AGENTS_PAUSE_SOURCE marks the pause as automation-set by the budget breaker. A pause a human set by hand (no source, or a human source) is never exempted: a manual pause means stop.
  3. Cheap work only, by data. The allowed tiers come from a data file, default triage only (tier_for_intent in scripts/engine.sh: triage = human-pr, fix-bot-comment; action = fix-reviews, fix-ci, rebase; deep = fix-issue, human). Deep, audit and single work never run on Gemini under a pause. Widening the list later is a data change.
  4. Only when capacity exists. A tier runs on Gemini under a pause only if the tier view from Gemini availability per task tier: show remaining capacity per model, key and day so a capped model is visible before the work fails #2041 reports it available or degraded (never unavailable), computed from the caps file (Fill the Gemini caps file: self-imposed Free Tier limits for all four keys (Pro excluded), so the Gemini gate starts enforcing #2146). If no Gemini capacity is left, the work is deferred exactly as it is under a plain pause, with no attempt and no failed call.
  5. Gemini only. Under the exemption the engine chain is Gemini only. Claude and Copilot stay unavailable. No fallback onto Claude.
  6. No new credential and no new Gemini API calls for measurement. Capacity comes from the existing ledger-based metering.

Scope

  1. The variable, the tier allowlist data file, and the exemption logic in one shared helper (not copied into each workflow) used by validate-engines.sh, dev-lead-reusable.yml and pr-review.yml.
  2. A single log line per run stating that the pause is exempted for a named tier on Gemini, and why work was deferred when it is not, so a budget pause that is quietly doing a little work is legible.
  3. Fleet monitor: count of items processed under the exemption per day, next to the pause reason.
  4. AGENTS.md documents the variable, the allowlist, how to widen it, and how to turn it off.

Acceptance criteria

  1. With the variable unset or false, a pause behaves byte-for-byte as today (a test asserts all three engines unavailable).
  2. A pause with no AGENTS_PAUSE_SOURCE, or a human-set source, is not exempted even with the variable true.
  3. A budget-source pause with the variable true exempts only Gemini and only the allowlisted tiers; a deep-tier item is deferred.
  4. When the tier view reports unavailable, nothing is attempted and the deferral is logged.
  5. Claude and Copilot remain unavailable under the exemption (asserted).
  6. The helper is the single implementation: a test fails if the exemption logic is duplicated in a workflow.
  7. Shellcheck clean; existing tests pass unchanged; no live network in tests.

Blocked by

#2041 (tier view and rotation that drops a cap-0 key), #2146 (the caps file filled), and the budget breaker actually being armed (#1565, which itself waits for a dry-run window of the poller, #2029). Do not start until the first two have merged; the maintainer adds the dev-lead label then.

Related: #1565, #2029, #2041, #2146, #2030.

Activity

  1. github-actions commented on Oct 9, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and for spelling out the constraints so clearly. The design already looks well scoped (default off, budget-source pauses only, data-driven tier allowlist, Gemini only). One question to settle before work starts: when the exemption is active and the tier view reports degraded for an allowlisted tier, should the trickle run at the full capped rate, or should it be throttled further than the caps file implies? That would tell us whether degraded needs its own data setting or can share the available behaviour.

  2. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    dev-lead is withholding action on this item.

    It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

    To re-enable automated pickup: remove the dev-lead:hands-off label.

  3. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Status 2026-10-11: the precondition is in review; the trickle itself is stopped on three findings

    In review: the first acceptance criterion (records traceable to trusted runs) as #2305. It adds the trust rule, a provenance stamp on every collected record, and a fail-closed reader for future dispatch decisions. Nothing calls that reader yet.

    Not built: the exemption. Designing it against the code found three things that the issue as written does not account for.

    1. The Copilot credential cannot be withheld from a dev-lead job

    Constraint 5 says Claude and Copilot stay unavailable under the exemption. For Claude that can be enforced: its credential is handed to individual steps and can be withheld by expression, or never named in a separate job. For Copilot it cannot. In dev-lead-reusable.yml the Copilot credential is the acting token: GH_TOKEN is set for the whole job from the maintainer PAT (line 180), COPILOT_GITHUB_TOKEN is the same secret, copilot_chat falls back to GH_TOKEN, and the writer session inherits GH_TOKEN. A job that must push and comment as the maintainer cannot be denied that token. Copilot can only be excluded by configuration (AI_ENGINES=gemini), which is a setting, not a guarantee.

    2. The ledger the capacity check would read is almost entirely test fixtures

    About 8 of 1,819 request-counting Gemini records in four days of artifacts were real calls. The rest were written by this repository's engine tests, run by agent sessions that inherit the real ledger path. They come from trusted runs, so no provenance rule removes them. Filed as #2304. Until that is fixed at the source, "the tier view reports capacity" (constraint 4) is a statement about test fixtures.

    3. Capacity is not knowable when a job starts, and the caps file does not bound the fleet

    A job's own ledger starts empty, so a job-local tier view always reads available. The only cross-job view is the tier snapshot, written once a day. And the caps file is enforced inside one job's rotation plan: across jobs, the bound under a wrong "go" is the provider's real limits and the prepaid credit, not the caps. (The recent samples also include Gemini answering HTTP 402, "prepayment credits are depleted", four times on 2026-10-10 and 11.)

    Smaller points that change the design

    • "Triage tier" in dev-lead is in practice one intent, fix-bot-comment, and it is a writer run that can push commits, not a read-only classification. human-pr is in tier_for_intent but nothing emits it.
    • A job that starts and then defers ends success, not skipped, so it cannot look exactly like a plain pause; the canary promotion gate counts a success as a health sample.
    • A Gemini refusal under the trickle posts the same status=rate-limited marker the budget breaker counts, with no engine in it. Three just before a resume would re-trip the breaker. Preventing that means posting no marker under the trickle.
    • scripts/validate-engines.sh honours the pause only on paper: no workflow passes it the variable.
    • The gate invariant from AGENTS_PAUSED stops only dev-lead: pr-review, the persona runner and the other model-calling workflows keep running through a fleet pause #2249 can be kept checkable with a separate trickle job and a new registry mode; that part is not a blocker.

    What would unblock it

    1. The Gemini token ledger is mostly test fixtures: dev-lead agent sessions run the engine tests with the real TOKEN_LOG_FILE set, and the run uploads what the tests wrote #2304 (tests stop writing to the real ledger).
    2. A decision on Copilot: a credential for the trickle job that is not the Copilot-entitled token, or written acceptance that Copilot is excluded by configuration only.
    3. A fresh cross-job capacity signal: an hourly snapshot the job can fetch, or the fleet-state persistence deferred under Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered #2030.

    The full design (gate form, verdict table, kill switch) is ready to reuse when those are settled. The decision whether to continue is with the maintainer.

  4. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer decision, 2026-10-11: parked. The exemption is not being built. Pause stays the only control during a budget pause. What continues: the trusted-records precondition (#2305) and the ledger fix (#2304), both of which stand on their own. Reopen the design here only if the three conditions above are met and the trickle is still wanted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions