Skip to content

Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered #2030

Closed

Description

@don-petry

Decision this implements

Maintainer decision (2026-10-02, recorded on #1565): when the Claude budget breaker trips, the fleet degrades to the Gemini engine instead of pausing, and Gemini quota must be checked and respected exactly as Claude's is. Degrading onto an unmetered provider just moves the outage, so this is the prerequisite for arming anything.

Problem: Gemini spend is observed only as a binary "depleted" probe

  • check_provider_headroom in scripts/engine.sh has a claude branch and a copilot branch and no gemini branch. For gemini, used_pct stays 0 and the function logs "usage 0% — ok" and returns 0. Gemini can never be skipped on headroom.
  • The only protection is reactive: a call fails, _rate_limit_pattern matches (429, RESOURCE_EXHAUSTED, quota exceeded), and the key rotates (per key, then model chain, then cross-provider). Rotation across the 4 configured keys (GOOGLE_API_KEY, _2, _3, _4) has no memory: a key that just returned RESOURCE_EXHAUSTED is retried by the next job from the start of the rotation.
  • A pre-flight _gemini_billing_probe (scripts/validate-engines.sh) detects keys whose prepayment credits are depleted, and _gemini_api_keys (scripts/engine.sh) tries depleted keys last. This is the only Gemini awareness that exists. It is binary: it says nothing until a key is already at zero, so it cannot stop a fleet-wide fallback from draining the credits within one window.
  • Under a Claude breaker trip every job would hit Gemini at once. With no per-key ceiling, the free-tier daily request quotas are exhausted within hours, after which Gemini is dead exactly when Claude is constrained.

Design constraints

  1. Do not assume the provider reports remaining quota. Do not write a probe that depends on rate-limit response headers for Gemini; meter from the fleet's own ledger instead. A probe that silently returns "ok" on missing data is exactly the bug above (check_provider_headroom has never worked: probes with an unset ANTHROPIC_API_KEY, reads per-minute TPM instead of the subscription window, and pr-review never calls it #1862).
  2. Self-meter from the existing ledger, and add the key index to it. emit_token_record records engine and model but not which key made the call, so per-key usage cannot be computed today. Extend the record with the key index (e.g. 0 for GEMINI_API_KEY/GOOGLE_API_KEY, 2..4 for the others; never the value, and a test asserts no key value can appear in a record). With that field in place, per-key Gemini consumption (calls and tokens per window) is computed from the ledger in scripts/lib/token-metrics.sh with no new credential.
  3. Limits are configuration, not code, keyed per key index and model. Funding differs by key (maintainer, 2026-10-02): one key is Tier 1 with prepaid credits and a $1 cap; the others are Free Tier. Per Google's rate-limits documentation, limits apply per project, not per API key, and vary by model; the maintainer states each key is on a separate account, so each key index is an independent pool. Rows carry model_glob (a * row expresses a shared pool; per-model rows express separate pools), the tier, and rpm / tpm / rpd, all read from the AI Studio rate-limit viewer by the maintainer. The daily quota resets at midnight Pacific; the daily window must use the configured reset time and timezone, not UTC days. Estimating dollar credit burn is deliberately out of scope: the paid key's $1 balance is covered by the existing depleted-credits probe (_gemini_billing_probe) and the rotation-last ordering of depleted keys, and a burn estimator would add machinery for a balance that small.
  4. Fail direction. Unknown or unreadable usage for Gemini degrades conservatively (treat the key as constrained and prefer the next key), and the log says so. This is the opposite of the Claude probe's fail-open, deliberately, because Gemini is the fallback of last resort.

Scope (slice 1: observe and respect; no actuation)

  1. The caps data file ships unfilled. One row per configured key index and model glob, every limit unknown. A key whose limits are unknown is treated as constrained (constraint 4), so nothing is trusted until a maintainer fills real numbers. Do not attempt to read limits from the provider, call the Gemini API from this work, or read any key. The numbers are maintainer-supplied, tracked as a checklist in this issue's comments.
  2. A gemini branch in check_provider_headroom that computes usage from the ledger against the configured caps for each key and returns 1 (skip) when at or above DEV_LEAD_USAGE_THRESHOLD, with the same one-line [headroom] log format as Claude and Copilot. Unknown limits or an unreadable ledger return a distinct constrained result (2) that is logged and ordered last, not silently ok.
  3. Per-key cooldown memory. When a key returns a rate-limit or RESOURCE_EXHAUSTED, record the key (by index, never the value) with a cooldown deadline derived from the error's retry hint, else the configured default, and skip it during rotation until the deadline. The state must survive across jobs in a run window (use the same mechanism the other fleet state uses; do not invent a new store without saying why).
  4. Gemini usage appears in the fleet monitor report next to Claude's: per key index, used against cap for each window, the reset window used, and cooldown state.
  5. AGENTS.md documents the Gemini metering and the cooldown.

Acceptance criteria

  1. check_provider_headroom gemini no longer always returns ok: a test with a ledger at or above the threshold returns 1 and logs the percent; below returns 0; an unreadable ledger returns the documented conservative result and logs why.
  2. A test shows a key placed in cooldown is skipped by rotation until its deadline and used again after it. No key value appears in any log or test output.
  3. Caps are read from the data file; a test fails if a numeric limit is hard-coded in engine.sh. The daily window honours the configured reset time and timezone: a test places calls just before and after Pacific midnight on different UTC dates.
  4. A test asserts that with every limit unknown the gemini headroom result is the conservative one and the log names "limits unknown", so shipping the unfilled file cannot read as healthy. A record with no key_index is unattributed and is never charged to key 1.
  5. No code path added by this work makes a network call to the Gemini API for measurement purposes.
  6. Shellcheck clean; existing engine tests pass unchanged.

Out of scope (tracked separately, do not build here)

Actuation: flipping AI_ENGINES to put Gemini first when the Claude breaker trips, restoring it after reset, and refusing to degrade when Gemini itself is constrained. That needs the dry-run poller (#2029) to exist first. Order: this issue, then #2029, then the actuation issue.

Also out of scope: changing the Gemini authentication mode. The fleet continues to use API keys. Do not add Google-login OAuth or Compute-ADC auth, a cached oauth_creds.json, or any use of the Gemini CLI /quota command.

Related: #1565, #2029, #1862 (the Claude headroom probe that never worked; do not repeat its failure mode), #1777 (key rotation).

Activity

  1. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer-supplied inputs (not dev-lead work; needed before the gemini gate can trust any key):

    • Tier (free or paid) for each of the 4 configured keys, by index only, never the value
    • Per-model limits for each key: requests/minute, tokens/minute, requests/day
    • The daily quota reset time and timezone
    • Which Gemini models the fleet chain actually uses (engine-models.sh), so caps are filled for the right rows
      Fill these into the caps data file in a follow-up commit once it exists.
  2. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2030 — Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered

    Scope

    Meter Gemini per key index from the existing token ledger against a maintainer-filled caps file (shipped all-unknown), add a gemini headroom branch, add per-key cooldown memory to rotation, and surface per-key usage-vs-cap in the fleet token report. No actuation, no Gemini API calls, and no key values read into logs.

    Design decisions (flagging for maintainer review)

    • Caps file: scripts/lib/gemini-quota-caps.tsv, next to model-pricing.tsv. One row per key index (1–4) × chain model (gemini-3.8-flash, gemini-3.1-pro-preview) with columns tier rpm tpm rpd. Every row ships as unknown. Settings rows hold daily_reset (unknown; while it is unknown the RPD window falls back to a rolling 24h, which is the conservative choice) and default_cooldown_sec.
    • Key index: GEMINI_API_KEY/GOOGLE_API_KEY = 1 (the primary slot), GOOGLE_API_KEY_N = N. Only the index is ever logged or recorded.
    • Ledger: emit_token_record gains an optional key_index field, which gemini rotation fills in. Usage is computed from GEMINI_LEDGER_FILE (default TOKEN_LOG_FILE).
    • Cooldown store: the cooldown is not a new store. It is a kind:"gemini_key_cooldown" record on the same Token Observatory JSONL channel, the same one emit_verification_record and emit_lsp_coldstart_record already use. The cost report already ignores non-token_usage kinds, and the record rides the existing ledger persistence (per-run file plus artifact). The record is keyed by key index and model, because Gemini quotas are per model. Its deadline is the error's retry hint (retryDelay / "retry in Ns"), or default_cooldown_sec if the error gives none.
    • Headroom return codes: 0 means a key has measured headroom. 1 means every metered key is at or above DEV_LEAD_USAGE_THRESHOLD, or every key is cooling down. 2 is new: constrained, meaning limits are unknown or the ledger is unreadable. For 2, the log is a ::warning:: that names limits unknown / ledger unreadable and never prints "ok". Keys in that state are rotated last. run_writer_with_fallback still proceeds on 2, because refusing to degrade onto a constrained Gemini is the actuation slice (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029 → follow-up), which is out of scope here. The distinct code lets that slice consume the answer without parsing logs.
    • Test change: the existing test headroom: gemini returns 0 unconditionally (no headroom API) asserts the bug this issue fixes (AC1, "no longer always returns ok"). I will replace it with the documented conservative-result assertion. Every other existing engine test stays unchanged.

    Implementation Checklist

    • scripts/lib/gemini-quota-caps.tsv: caps data file, shipped unfilled
    • scripts/lib/gemini-quota.sh: caps lookup, ledger window usage, key status, cooldown record/read, retry-hint parse (pure, no network)
    • scripts/lib/token-metrics.sh: optional key_index on emit_token_record
    • scripts/engine.sh: gemini branch in check_provider_headroom; key-name/index helper; rotation skips cooling keys, orders constrained keys last, records cooldowns; key index flows to the token record; the caller treats 2 as proceed-with-warning
    • scripts/token_report.sh: "Gemini quota (per key index)" section with peak RPM/TPM/RPD vs cap and cooldown counts
    • .github/workflows/actions-fleet-monitor.yml: inline per-workflow aggregate counts only token_usage records
    • AGENTS.md: Gemini metering + cooldown section

    Tests to Write

    • tests/dev-lead/unit/test_gemini_quota.bats: ledger at/above threshold → 1 with the percent logged; below → 0; unreadable ledger → 2 and logs why; all-unknown caps → 2 and logs "limits unknown"; the shipped file is all-unknown; no numeric limit literal in engine.sh; a cooling key is skipped until its deadline and used after it; retry hint vs default deadline; no key value in any output; no Gemini endpoint referenced by the new code
    • tests/token_report.bats: Gemini per-key section
    • tests/dev-lead/unit/test_token_metrics.bats: key_index field

    Test Command

    bats tests/dev-lead/unit/ tests/token_report.bats tests/model_pricing.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet. The automation posts the authoritative completion record (with PR number + head SHA) after the push. Treat the notes below as intent, not delivered results.

    Plan Execution (intended)

    • Followed the plan posted above: caps file, quota library, headroom branch, cooldown-aware rotation, ledger key attribution, report section, AGENTS.md.

    Files touched (this workspace, not yet pushed)

    • scripts/lib/gemini-quota-caps.tsv (new): 4 key indexes × the 2 default chain models, every limit unknown; daily_reset_* is unknown, default_cooldown_sec is 60 (one RPM window)
    • scripts/lib/gemini-quota.sh (new): ledger metering, verdict, cooldown record/read, retry-hint parse, rotation plan. No network calls.
    • scripts/engine.sh: gemini headroom branch (0 ok / 1 skip / 2 constrained); _gemini_api_key_names; rotation skips cooling keys and orders constrained keys last; key index flows into the token record; run_writer_with_fallback proceeds on 2
    • scripts/lib/token-metrics.sh: optional key_index (11th arg, numeric only; the field is absent otherwise)
    • scripts/token_report.sh: "Gemini quota (per key index)" section, placed after the per-model cost drivers
    • .github/workflows/actions-fleet-monitor.yml: the inline per-workflow aggregate counts only token_usage records
    • .github/workflows/lint.yml: new bats file added to the always-run list (per AGENTS.md)
    • AGENTS.md: "Gemini quota metering & per-key cooldown" section
    • Tests: new tests/dev-lead/unit/test_gemini_quota.bats; additions to test_token_metrics.bats and tests/token_report.bats

    Notes

    • One existing test was replaced, not just added to: headroom: gemini returns 0 unconditionally (no headroom API) asserted the exact bug in AC1. It now asserts the documented conservative result (2, logs "limits unknown", never "ok", no curl call).
    • Cross-job scope: the gate meters and remembers cooldowns from whatever ledger it is pointed at (GEMINI_LEDGER_FILE, default the run's TOKEN_LOG_FILE). A per-run ledger only sees its own run. The org-wide view is in the fleet report, which aggregates every repo's artifacts. Feeding a merged ledger to the gate is a natural fit for the Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029 poller.
    • Before trusting any key, maintainer inputs are still needed (see the checklist comment above). Until then every key reads as constrained, and Gemini still runs as the fallback with a warning.
  4. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2033

    The implementation for issue #2030 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  5. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Updated per maintainer input (2026-10-02): (1) the quota is one shared pool per key across all Gemini models, so the caps file is one row per key, and usage is summed over every model; the earlier checklist item about which models the chain uses is no longer needed for caps. (2) The metering must record the key index in the ledger, since it is not recorded today. (3) Note for implementers: the Gemini CLI /quota command reads the Code Assist backend, which exists only for Google-login or Compute-ADC auth. The fleet authenticates with API keys, for which that endpoint is unavailable, so do not build on /quota or retrieveUserQuota. Self-metering stands unless the maintainer decides to move the fallback to a different auth mode.

  6. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Revised maintainer checklist (supersedes the earlier one). The Gemini keys are funded by prepaid credits, so the budget is credit balance, not request rate:

    • Confirm the 4 keys are prepaid-credit funded (not pay-as-you-go billed in arrears)
    • Current credit balance in USD for each key index, and the date read (index only, never the key value)
    • Optional rate caps per key if known: requests/minute, tokens/minute, requests/day (one shared pool across models)
    • Whether any two key indexes draw on the same project or billing account (they would share one balance)

    Correction to the issue text as first filed: it said Gemini was "completely unobserved". There is a pre-flight depleted-credits probe (_gemini_billing_probe); it is binary and only fires at zero. The body now says so.

  7. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Corrections and maintainer inputs (2026-10-02):

    1. RETRACTION: an earlier comment said the Gemini quota is "one shared pool per key across all models, maintainer-confirmed". That observation came from the CLI /quota command, which reads the Code Assist pool, not the API-key pool. Google documents API limits as per project, not per key, varying by model. Keep per-model rows (model_glob); a * row expresses a shared pool if AI Studio shows one.
    2. Funding per key: one key is Tier 1 with prepaid credits and a $1 cap; the others are Free Tier. Each key is on a separate account, so each key index is an independent pool. The tier column must drive which limit binds (free: rate caps, chiefly requests/day; paid: credit burn).
    3. Daily quota resets at midnight Pacific.
    4. Still maintainer-supplied, never derived by dev-lead: the actual rpm/tpm/rpd per key and model from the AI Studio rate-limit viewer, and the paid key remaining credit with the date read.
      Do not claim or encode any Tier 1 vs Free limit numbers from memory.
  8. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Scope note (2026-10-02): the earlier request for prepaid-credit dollar-burn estimation is withdrawn. Only one key is paid, with a $1 balance, which the existing depleted-credits probe and rotation-last ordering already cover. The caps file stays rate-limit based. New in-flight requirements for PR 2033: the daily window follows the Pacific-midnight reset, and a record without a key_index is unattributed, never charged to key 1.

  9. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer-supplied data, 2026-10-02 (from the AI Studio rate-limit viewer for the Tier 1 key; format used / limit). Reference only; effective caps are decided separately below.

    Models the fleet chain actually calls (engine-models.sh): gemini-3.8-flash (triage, action) and gemini-3.1-pro-preview (deep, audit, single). Google limits for the Tier 1 key:

    • Gemini 3.8 Flash: 1,000 RPM, 2,000,000 TPM, 10,000 RPD
    • Gemini 3.1 Pro: 25 RPM, 2,000,000 TPM, 250 RPD
      (The viewer lists it as "Gemini 3.1 Pro"; confirm it is the same quota bucket as the gemini-3.1-pro-preview id the fleet sends.)

    Findings that bind this work:

    1. Quotas are per model (each model has its own RPM/TPM/RPD and its own usage counter), not one pool per key. Keep model_glob rows; do not collapse to one row per key.
    2. The Tier 1 key is paid with a $1 balance/cap. Google limits 10,000 RPD would make this key look like the healthiest and rotate it FIRST, burning the $1 in roughly 70 Flash or 18 Pro calls and then stalling each call on depleted credits. So the caps file must carry the effective cap: the lower of the Google limit and what the maintainer will spend, never Google limit alone. A row whose cap exceeds the key funded capacity is a bug.
    3. Pending from the maintainer: which key index is the Tier 1 key, its effective caps, and the same table for each Free Tier key. Do not fill these from memory or from the numbers above for any key other than the Tier 1 key.
  10. don-petry commented on Oct 3, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer-supplied data, 2026-10-02: AI Studio rate-limit viewer for a Free Tier key (format used / limit; the maintainer states each key is its own account; whether the three free keys share identical limits is assumed, to be confirmed).

    Models relevant to the fleet chain (limits are RPM / TPM / RPD):

    • Gemini 3.8 Flash: 5 / 250K / 20
    • Gemini 3.5 Flash: 5 / 250K / 20
    • Gemini 3 Flash: 5 / 250K / 20
    • Gemini 3.1 Flash Lite: 15 / 250K / 500
    • Gemini 3.5 Flash Lite: 15 / 250K / 500
    • Gemini 3.1 Pro: 0 / 0 / 0
    • Gemini 2.5 Pro: 0 / 0 / 0

    Implementation requirements that follow from this data:

    1. A cap of 0 means the model is unavailable on that key. The gate and rotation must skip that key for that model without calling it, and must not divide by zero. The fleet deep/audit/single tiers send gemini-3.1-pro-preview first, which can never succeed on a Free Tier key.
    2. Observed usage exceeds the limit (Gemini 3.8 Flash showed 81 requests against 20 RPD, 338K against 250K TPM). Usage over the cap must compute as over 100 percent and be reported as over, never as an error or a negative headroom. It also shows rejected attempts are being retried against an exhausted key, which the cooldown memory exists to stop.
    3. Quotas are per model, so total daily capacity depends on which models the chain spans. Do not change the chain in this issue; record that the Flash Lite models carry 500 RPD against 20 for the Flash models, so the choice of overflow model is a maintainer decision.
    4. Still pending from the maintainer: which key index is the Tier 1 key and which are free, and its effective caps. Fill nothing for a specific index until stated.
  11. added a commit that references this issue on Oct 6, 2026
  12. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Verified on main after PR 2033 merged (2026-10-06): per-key per-model metering, key_index in the ledger (call-unique sidecar, unattributed when absent), Pacific-midnight daily window from the settings rows, cap 0 computes as 100% and usage above cap exceeds 100%, constrained result 2, cooldown memory, report section, docs. The caps file ships unfilled by design.

    One requirement from this issue is NOT met and is carried into the follow-up (2041): a cap of 0 or an exhausted daily window still lands in the "tried after the healthy keys" bucket of gq_rotation_plan and is called, instead of being skipped. Also outstanding and not code: the maintainer must supply which key slot is the Tier 1 key and its effective caps. Closing this as delivered; enforcement starts when real numbers are filled in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions