Repository navigation
Description
Decision this implements
Maintainer decision (2026-10-02, recorded on #1565): when the Claude budget breaker trips, the fleet degrades to the Gemini engine instead of pausing, and Gemini quota must be checked and respected exactly as Claude's is. Degrading onto an unmetered provider just moves the outage, so this is the prerequisite for arming anything.
Problem: Gemini spend is observed only as a binary "depleted" probe
check_provider_headroominscripts/engine.shhas aclaudebranch and acopilotbranch and nogeminibranch. Forgemini,used_pctstays 0 and the function logs "usage 0% — ok" and returns 0. Gemini can never be skipped on headroom.- The only protection is reactive: a call fails,
_rate_limit_patternmatches (429,RESOURCE_EXHAUSTED,quota exceeded), and the key rotates (per key, then model chain, then cross-provider). Rotation across the 4 configured keys (GOOGLE_API_KEY,_2,_3,_4) has no memory: a key that just returnedRESOURCE_EXHAUSTEDis retried by the next job from the start of the rotation. - A pre-flight
_gemini_billing_probe(scripts/validate-engines.sh) detects keys whose prepayment credits are depleted, and_gemini_api_keys(scripts/engine.sh) tries depleted keys last. This is the only Gemini awareness that exists. It is binary: it says nothing until a key is already at zero, so it cannot stop a fleet-wide fallback from draining the credits within one window. - Under a Claude breaker trip every job would hit Gemini at once. With no per-key ceiling, the free-tier daily request quotas are exhausted within hours, after which Gemini is dead exactly when Claude is constrained.
Design constraints
- Do not assume the provider reports remaining quota. Do not write a probe that depends on rate-limit response headers for Gemini; meter from the fleet's own ledger instead. A probe that silently returns "ok" on missing data is exactly the bug above (check_provider_headroom has never worked: probes with an unset ANTHROPIC_API_KEY, reads per-minute TPM instead of the subscription window, and pr-review never calls it #1862).
- Self-meter from the existing ledger, and add the key index to it.
emit_token_recordrecords engine and model but not which key made the call, so per-key usage cannot be computed today. Extend the record with the key index (e.g.0forGEMINI_API_KEY/GOOGLE_API_KEY,2..4for the others; never the value, and a test asserts no key value can appear in a record). With that field in place, per-key Gemini consumption (calls and tokens per window) is computed from the ledger inscripts/lib/token-metrics.shwith no new credential. - Limits are configuration, not code, keyed per key index and model. Funding differs by key (maintainer, 2026-10-02): one key is Tier 1 with prepaid credits and a $1 cap; the others are Free Tier. Per Google's rate-limits documentation, limits apply per project, not per API key, and vary by model; the maintainer states each key is on a separate account, so each key index is an independent pool. Rows carry
model_glob(a*row expresses a shared pool; per-model rows express separate pools), the tier, and rpm / tpm / rpd, all read from the AI Studio rate-limit viewer by the maintainer. The daily quota resets at midnight Pacific; the daily window must use the configured reset time and timezone, not UTC days. Estimating dollar credit burn is deliberately out of scope: the paid key's $1 balance is covered by the existing depleted-credits probe (_gemini_billing_probe) and the rotation-last ordering of depleted keys, and a burn estimator would add machinery for a balance that small. - Fail direction. Unknown or unreadable usage for Gemini degrades conservatively (treat the key as constrained and prefer the next key), and the log says so. This is the opposite of the Claude probe's fail-open, deliberately, because Gemini is the fallback of last resort.
Scope (slice 1: observe and respect; no actuation)
- The caps data file ships unfilled. One row per configured key index and model glob, every limit
unknown. A key whose limits areunknownis treated as constrained (constraint 4), so nothing is trusted until a maintainer fills real numbers. Do not attempt to read limits from the provider, call the Gemini API from this work, or read any key. The numbers are maintainer-supplied, tracked as a checklist in this issue's comments. - A
geminibranch incheck_provider_headroomthat computes usage from the ledger against the configured caps for each key and returns 1 (skip) when at or aboveDEV_LEAD_USAGE_THRESHOLD, with the same one-line[headroom]log format as Claude and Copilot. Unknown limits or an unreadable ledger return a distinct constrained result (2) that is logged and ordered last, not silently ok. - Per-key cooldown memory. When a key returns a rate-limit or
RESOURCE_EXHAUSTED, record the key (by index, never the value) with a cooldown deadline derived from the error's retry hint, else the configured default, and skip it during rotation until the deadline. The state must survive across jobs in a run window (use the same mechanism the other fleet state uses; do not invent a new store without saying why). - Gemini usage appears in the fleet monitor report next to Claude's: per key index, used against cap for each window, the reset window used, and cooldown state.
AGENTS.mddocuments the Gemini metering and the cooldown.
Acceptance criteria
check_provider_headroom geminino longer always returns ok: a test with a ledger at or above the threshold returns 1 and logs the percent; below returns 0; an unreadable ledger returns the documented conservative result and logs why.- A test shows a key placed in cooldown is skipped by rotation until its deadline and used again after it. No key value appears in any log or test output.
- Caps are read from the data file; a test fails if a numeric limit is hard-coded in
engine.sh. The daily window honours the configured reset time and timezone: a test places calls just before and after Pacific midnight on different UTC dates. - A test asserts that with every limit
unknownthe gemini headroom result is the conservative one and the log names "limits unknown", so shipping the unfilled file cannot read as healthy. A record with no key_index is unattributed and is never charged to key 1. - No code path added by this work makes a network call to the Gemini API for measurement purposes.
- Shellcheck clean; existing engine tests pass unchanged.
Out of scope (tracked separately, do not build here)
Actuation: flipping AI_ENGINES to put Gemini first when the Claude breaker trips, restoring it after reset, and refusing to degrade when Gemini itself is constrained. That needs the dry-run poller (#2029) to exist first. Order: this issue, then #2029, then the actuation issue.
Also out of scope: changing the Gemini authentication mode. The fleet continues to use API keys. Do not add Google-login OAuth or Compute-ADC auth, a cached oauth_creds.json, or any use of the Gemini CLI /quota command.
Related: #1565, #2029, #1862 (the Claude headroom probe that never worked; do not repeat its failure mode), #1777 (key rotation).
Activity
don-petry commentedon Oct 2, 2026
Maintainer-supplied inputs (not dev-lead work; needed before the gemini gate can trust any key):
Fill these into the caps data file in a follow-up commit once it exists.
don-petry commentedon Oct 2, 2026
Dev-Lead Implementation Plan
Issue: #2030 — Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered
Scope
Meter Gemini per key index from the existing token ledger against a maintainer-filled caps file (shipped all-
unknown), add ageminiheadroom branch, add per-key cooldown memory to rotation, and surface per-key usage-vs-cap in the fleet token report. No actuation, no Gemini API calls, and no key values read into logs.Design decisions (flagging for maintainer review)
scripts/lib/gemini-quota-caps.tsv, next tomodel-pricing.tsv. One row per key index (1–4) × chain model (gemini-3.8-flash,gemini-3.1-pro-preview) with columnstier rpm tpm rpd. Every row ships asunknown. Settings rows holddaily_reset(unknown; while it is unknown the RPD window falls back to a rolling 24h, which is the conservative choice) anddefault_cooldown_sec.GEMINI_API_KEY/GOOGLE_API_KEY= 1 (the primary slot),GOOGLE_API_KEY_N= N. Only the index is ever logged or recorded.emit_token_recordgains an optionalkey_indexfield, which gemini rotation fills in. Usage is computed fromGEMINI_LEDGER_FILE(defaultTOKEN_LOG_FILE).kind:"gemini_key_cooldown"record on the same Token Observatory JSONL channel, the same oneemit_verification_recordandemit_lsp_coldstart_recordalready use. The cost report already ignores non-token_usagekinds, and the record rides the existing ledger persistence (per-run file plus artifact). The record is keyed by key index and model, because Gemini quotas are per model. Its deadline is the error's retry hint (retryDelay/ "retry in Ns"), ordefault_cooldown_secif the error gives none.0means a key has measured headroom.1means every metered key is at or aboveDEV_LEAD_USAGE_THRESHOLD, or every key is cooling down.2is new: constrained, meaning limits are unknown or the ledger is unreadable. For2, the log is a::warning::that nameslimits unknown/ledger unreadableand never prints "ok". Keys in that state are rotated last.run_writer_with_fallbackstill proceeds on2, because refusing to degrade onto a constrained Gemini is the actuation slice (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029 → follow-up), which is out of scope here. The distinct code lets that slice consume the answer without parsing logs.headroom: gemini returns 0 unconditionally (no headroom API)asserts the bug this issue fixes (AC1, "no longer always returns ok"). I will replace it with the documented conservative-result assertion. Every other existing engine test stays unchanged.Implementation Checklist
scripts/lib/gemini-quota-caps.tsv: caps data file, shipped unfilledscripts/lib/gemini-quota.sh: caps lookup, ledger window usage, key status, cooldown record/read, retry-hint parse (pure, no network)scripts/lib/token-metrics.sh: optionalkey_indexonemit_token_recordscripts/engine.sh:geminibranch incheck_provider_headroom; key-name/index helper; rotation skips cooling keys, orders constrained keys last, records cooldowns; key index flows to the token record; the caller treats2as proceed-with-warningscripts/token_report.sh: "Gemini quota (per key index)" section with peak RPM/TPM/RPD vs cap and cooldown counts.github/workflows/actions-fleet-monitor.yml: inline per-workflow aggregate counts onlytoken_usagerecordsAGENTS.md: Gemini metering + cooldown sectionTests to Write
tests/dev-lead/unit/test_gemini_quota.bats: ledger at/above threshold → 1 with the percent logged; below → 0; unreadable ledger → 2 and logs why; all-unknowncaps → 2 and logs "limits unknown"; the shipped file is all-unknown; no numeric limit literal inengine.sh; a cooling key is skipped until its deadline and used after it; retry hint vs default deadline; no key value in any output; no Gemini endpoint referenced by the new codetests/token_report.bats: Gemini per-key sectiontests/dev-lead/unit/test_token_metrics.bats:key_indexfieldTest Command
bats tests/dev-lead/unit/ tests/token_report.bats tests/model_pricing.bats+bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Oct 2, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
Files touched (this workspace, not yet pushed)
scripts/lib/gemini-quota-caps.tsv(new): 4 key indexes × the 2 default chain models, every limitunknown;daily_reset_*isunknown,default_cooldown_secis 60 (one RPM window)scripts/lib/gemini-quota.sh(new): ledger metering, verdict, cooldown record/read, retry-hint parse, rotation plan. No network calls.scripts/engine.sh:geminiheadroom branch (0 ok / 1 skip / 2 constrained);_gemini_api_key_names; rotation skips cooling keys and orders constrained keys last; key index flows into the token record;run_writer_with_fallbackproceeds on 2scripts/lib/token-metrics.sh: optionalkey_index(11th arg, numeric only; the field is absent otherwise)scripts/token_report.sh: "Gemini quota (per key index)" section, placed after the per-model cost drivers.github/workflows/actions-fleet-monitor.yml: the inline per-workflow aggregate counts onlytoken_usagerecords.github/workflows/lint.yml: new bats file added to the always-run list (per AGENTS.md)AGENTS.md: "Gemini quota metering & per-key cooldown" sectiontests/dev-lead/unit/test_gemini_quota.bats; additions totest_token_metrics.batsandtests/token_report.batsNotes
headroom: gemini returns 0 unconditionally (no headroom API)asserted the exact bug in AC1. It now asserts the documented conservative result (2, logs "limits unknown", never "ok", no curl call).GEMINI_LEDGER_FILE, default the run'sTOKEN_LOG_FILE). A per-run ledger only sees its own run. The org-wide view is in the fleet report, which aggregates every repo's artifacts. Feeding a merged ledger to the gate is a natural fit for the Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029 poller.don-petry commentedon Oct 2, 2026
Dev-Lead: Implementation Complete — PR #2033
The implementation for issue #2030 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
fe303295992e7069d6ddbe897a88d7d419e0b533main: 12 file(s), +1201/-27 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
don-petry commentedon Oct 2, 2026
Updated per maintainer input (2026-10-02): (1) the quota is one shared pool per key across all Gemini models, so the caps file is one row per key, and usage is summed over every model; the earlier checklist item about which models the chain uses is no longer needed for caps. (2) The metering must record the key index in the ledger, since it is not recorded today. (3) Note for implementers: the Gemini CLI
/quotacommand reads the Code Assist backend, which exists only for Google-login or Compute-ADC auth. The fleet authenticates with API keys, for which that endpoint is unavailable, so do not build on/quotaorretrieveUserQuota. Self-metering stands unless the maintainer decides to move the fallback to a different auth mode.don-petry commentedon Oct 3, 2026
Revised maintainer checklist (supersedes the earlier one). The Gemini keys are funded by prepaid credits, so the budget is credit balance, not request rate:
Correction to the issue text as first filed: it said Gemini was "completely unobserved". There is a pre-flight depleted-credits probe (
_gemini_billing_probe); it is binary and only fires at zero. The body now says so.don-petry commentedon Oct 3, 2026
Corrections and maintainer inputs (2026-10-02):
*row expresses a shared pool if AI Studio shows one.Do not claim or encode any Tier 1 vs Free limit numbers from memory.
don-petry commentedon Oct 3, 2026
Scope note (2026-10-02): the earlier request for prepaid-credit dollar-burn estimation is withdrawn. Only one key is paid, with a $1 balance, which the existing depleted-credits probe and rotation-last ordering already cover. The caps file stays rate-limit based. New in-flight requirements for PR 2033: the daily window follows the Pacific-midnight reset, and a record without a key_index is unattributed, never charged to key 1.
don-petry commentedon Oct 3, 2026
Maintainer-supplied data, 2026-10-02 (from the AI Studio rate-limit viewer for the Tier 1 key; format used / limit). Reference only; effective caps are decided separately below.
Models the fleet chain actually calls (engine-models.sh): gemini-3.8-flash (triage, action) and gemini-3.1-pro-preview (deep, audit, single). Google limits for the Tier 1 key:
(The viewer lists it as "Gemini 3.1 Pro"; confirm it is the same quota bucket as the
gemini-3.1-pro-previewid the fleet sends.)Findings that bind this work:
model_globrows; do not collapse to one row per key.don-petry commentedon Oct 3, 2026
Maintainer-supplied data, 2026-10-02: AI Studio rate-limit viewer for a Free Tier key (format used / limit; the maintainer states each key is its own account; whether the three free keys share identical limits is assumed, to be confirmed).
Models relevant to the fleet chain (limits are RPM / TPM / RPD):
Implementation requirements that follow from this data:
gemini-3.1-pro-previewfirst, which can never succeed on a Free Tier key.feat: implement issue #2030 — Gemini quota is unobserved: check_provi…
don-petry commentedon Oct 9, 2026
Verified on main after PR 2033 merged (2026-10-06): per-key per-model metering, key_index in the ledger (call-unique sidecar, unattributed when absent), Pacific-midnight daily window from the settings rows, cap 0 computes as 100% and usage above cap exceeds 100%, constrained result 2, cooldown memory, report section, docs. The caps file ships unfilled by design.
One requirement from this issue is NOT met and is carried into the follow-up (2041): a cap of 0 or an exhausted daily window still lands in the "tried after the healthy keys" bucket of gq_rotation_plan and is called, instead of being skipped. Also outstanding and not code: the maintainer must supply which key slot is the Tier 1 key and its effective caps. Closing this as delivered; enforcement starts when real numbers are filled in.