Skip to content

Gemini availability per task tier: show remaining capacity per model, key and day so a capped model is visible before the work fails #2041

Closed

Description

@don-petry

Story

As the fleet maintainer, I want to see, per task tier, per model, per key, per day, how much Gemini capacity is left and whether the work that tier does can still run, so that a capped model is visible before work is started and fails, not after.

Slice 2 of the Gemini metering. Slice 1 is #2030 (PR #2033): it meters usage per key index and model against caps and reports peaks over a lookback. This issue builds on it and must not start until #2033 has merged.

Problem

Quotas are per model, and each fleet task tier is bound to specific models (scripts/lib/engine-models.sh: for Gemini, triage and action use gemini-3.8-flash then gemini-3.1-pro-preview; deep, audit and single use gemini-3.1-pro-preview then gemini-3.8-flash). When the model a tier needs is capped on every key, that tier's work fails. Slice 1 reports what was used; nothing answers "can tier T run on Gemini right now, and how much headroom does it have left today".

Effective limits (maintainer, 2026-10-09): every Gemini key has $1 prepaid, one-time, and the fleet is to assume the Free Tier limits for all four keys regardless of what Google would allow. The caps file therefore carries self-imposed caps: for gemini-3.8-flash 5 RPM, 250K TPM, 20 RPD; for gemini-3.1-pro-preview 0 (unavailable). So the common case is a model that is unavailable (Pro) or exhausted for the day (Flash at 20 requests), and the deep, audit and single tiers (Pro first, Flash fallback) will usually report degraded, never available. A spend estimator was considered and dropped: at 20 requests/day the $1 lasts 3 to 4 days and the existing depleted-credits probe then drops the key.

Scope

  1. Remaining capacity, forward-looking. A pure function that, for each (key index, model), returns remaining requests, tokens/minute and requests for the current window: daily windows run from the configured reset (Pacific midnight, from the caps file settings) to the next reset; minute windows are rolling. Result states: available(n remaining), cooling(until T), unavailable (cap 0), exhausted (used at or above cap), unknown (limits not filled).
  2. Tier view. For each tier (triage, action, deep, audit, single, duck) resolve its Gemini model chain with the existing ai_models_gemini_chain (do not duplicate the chain) and report the tier status:
    • available: its first-choice model has a usable key;
    • degraded: only a later model in its chain has a usable key (name it, because it is a quality change);
    • unavailable: no model in its chain has a usable key today;
    • unknown: caps not filled.
      Include which (model, key index) pairs are usable and the total remaining calls.
  3. Per-day history. Keep a per-Pacific-day matrix of key index x model: calls, peak per minute, number of rate-limit/cooldown events, and whether the model hit its cap that day. The report shows today and the previous N days (configurable, default 7).
  4. Machine-readable snapshot. Write the tier view as JSON (one object per tier, fields as above, plus a timestamp and the window boundaries used) every fleet-monitor run, and expose it to the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029) so a later actuation change can ask "can tier T run on Gemini now?". This issue does not gate or defer any dispatch. Delivery (maintainer decision 2026-10-09): the fleet-monitor step writes the snapshot to one documented file path AND uploads it as a workflow artifact under a fixed, documented name. Do not use step outputs (they do not cross workflows, and the poller is a separate workflow). Do not modify the poller or its workflow in this issue: PR 2031 (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029) is still unmerged. Document the path, the artifact name and the schema in AGENTS.md so the poller can consume it in a later change.
  5. Notice on loss. When a tier transitions to unavailable or degraded, the fleet monitor emits one notice per tier per Pacific day, not one per run.
  6. Count rejected attempts. When a Gemini call is rejected as rate-limited or quota-exceeded, record an attempt record in the ledger (key index, model, timestamp, rejected: true; no key value, no response body) so the per-window counts approach what Google counts. Remaining-capacity calculations use attempts, not only successes.
  7. Escalating cooldown to the reset. In feat: implement issue #2030 — Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered #2033 a cooldown lasts for the provider's retryDelay hint or a default, and nothing distinguishes a per-minute rejection from a daily one, so a key that exhausted its daily requests can be retried all day. Do not depend on a documented error-body format (none is documented, and no real example has been captured). Instead, each consecutive rejection of the same key and model doubles that cooldown (capped at the next daily reset in Pacific time), and a success resets it. A rejection recorded when the ledger already shows the daily cap reached sets the cooldown straight to the next reset.
  8. If the 429 text matches Google's documented wording for a daily quota (quota_exceeded, "daily quota"), treat it as daily; otherwise treat it as per-minute. Capture a redacted sample of each distinct rejection message in the report so a maintainer can see what Google actually sent. Fixtures in tests must come from a real captured message, never an invented one; if none is available, test only the escalation and the ledger-derived rule.
  9. Rotation must not call a key/model that cannot run. In the merged slice 1, gq_rotation_plan (scripts/lib/gemini-quota.sh) puts any key at or above the threshold into a "tried after the healthy keys" bucket instead of removing it. A cap of 0 computes as 100% and an exhausted daily window computes at or above 100%, so both are still called, last. Change the plan so that, for the model being called, a key whose state is unavailable (cap 0) or exhausted (used at or above its cap) is dropped from the plan and not called, reported as skip with the reason, like cool. Keys that are merely over the 75% threshold, and unknown keys, keep their current "tried last" behaviour: unknown must not become a hard block while the caps file is unfilled. If every key is dropped for a model, the existing path for "all keys cooling down" applies (classified as a rate limit so the chain moves to the next model).
  10. AGENTS.md documents the tier view, the states, the escalation, and the limitation below.

Acceptance criteria

  1. Tests for the remaining-capacity function covering each state, a window that straddles Pacific midnight, a DST change day, cap 0, usage above cap (reports exhausted, never negative remaining), and unknown.
  2. Tests for the tier view using a mocked chain and ledger: one model capped but a later one free gives degraded and names the model; all capped gives unavailable; the chain is read from ai_models_gemini_chain, not hard-coded (a test fails if a model id is hard-coded in the tier logic).
  3. The per-day history survives across runs and is bounded by the retention setting.
  4. The JSON snapshot validates against a documented schema and a test asserts it contains no key value, only indexes.
  5. The once-per-tier-per-day notice is tested, including that a second run the same Pacific day does not repeat it.
  6. Tests for scope 6 and 7: a rejection is recorded as an attempt and counted; consecutive rejections double the cooldown up to the next Pacific reset; a success resets it; a rejection at an already-reached daily cap jumps to the reset.
  7. Tests for scope 9: a cap-0 key is not called for that model but is still used for another model it has capacity for; an exhausted key is not called until the Pacific reset; an unknown key is still tried (no new hard block); with every key dropped the engine moves to the next model.
  8. Shellcheck clean; existing tests pass unchanged.

Known limitation to document, not to hide

All of this is metered from the fleet's own ledger. It cannot see usage of the same project by anything else (manual use, AI Studio, other tools), and Google counts attempts it rejected against the limits, which a success-only ledger does not. Google's viewer can therefore show more usage than the ledger. The report must say so and show the ledger count, so a maintainer can compare it with the AI Studio viewer.

No reconciliation or probe calls. Google's rate-limits documentation names the AI Studio web viewer as the way to see limits and describes no API that reports usage or remaining quota for a key, and a probe call would itself spend one request out of a daily cap as small as 20 (and a rejected probe still counts). Do not add any call to the Gemini API to measure or reconcile quota. Calibration against the AI Studio viewer is a manual maintainer step.

Out of scope

Deferring or gating dispatch on the snapshot, changing any model chain, and changing the engine order. Those are the actuation work, which depends on this.

Related: #2030 (PR #2033), #2029, #1565.


Review round 2 — required changes before PR 2150 can merge (maintainer, 2026-10-09)

PR 2150 has been through several passes that chased individual bot nitpicks, replied "Fixed" for changes that never landed (the harness retracted two), and tripped the tamper and regression guards. Stop chasing nitpicks. The maintainer has already resolved nine threads with reasons. Work ONLY this list, in at most TWO commits, and run bats on the touched test files (with date/sed compatible tooling as CI has) before every push. Never push a red suite. For any listed item you do not do, reply in its thread with the reason and resolve it. Do not rewrite history.

STATUS after the push to head 6044d9f (maintainer, 2026-10-09) — read this first

Verified by running the expressions, not by reading commit titles. Two things in that push are wrong and must be corrected; three MUST items are not started.

  • MUST 1 (gitleaks) is NOT fixed: the scan is still red. The new allowlist regex AIza[0-9A-Za-z_-]{39} requires 43 characters; the gcp-api-key finding is AIza + 35 = 39 characters, so it never matches (tested: it does not match a 39-character key; {35} and the narrow form below do). Replace the two regexes in that [[allowlists]] entry with exactly ONE: AIza0123456789[0-9A-Za-z_-]{25} (4 + 10 + 25 = 39). Remove the fake-secret regex unless a test proves a gitleaks finding needs it. Keep the path regex and condition = "AND". Still also build the fake key at runtime in the test so the literal is not in source. The broader {35} form is NOT acceptable: it would silently suppress a real Google key pasted into that file.
  • MUST 4 (pipe escape) is now WORSE, not fixed. The new line gsub("|"; "\|") treats | as a regex alternation of two empty patterns, which matches at every position: it turns gemini-3.8-flash into \|g\|e\|m\|i\|n\|i\|-... (tested). The correct expression is gsub("\\|"; "\\|") in the jq source as written inside the shell script (a regex for a literal pipe, replaced by backslash-pipe); verify by running it on gemini-3.8-flash (must be unchanged) and on weird|name (must become weird\|name). The tests MUST cover both a plain model name (unchanged) and a model name containing a pipe, because the missing plain-name test is how this shipped.
  • MUST 2 (re-redact untrusted artifact samples), MUST 3 (cooling must be skipped in rotation) and MUST 5 (only genuine quota rejections recorded as attempts) are not started. Do them.

MUST (real bugs or security; none optional)

  1. Secret-scan false positive (thread on tests/dev-lead/unit/test_gemini_tier_view.bats line ~289). Build the synthetic Google key at runtime so no AIza... literal exists in source (keep every assertion). Add exactly ONE [[allowlists]] entry to .gitleaks.toml in the existing style (condition = "AND", path ^tests/dev-lead/unit/test_gemini_tier_view\.bats$, regex AIza0123456789[A-Za-z0-9_-]{25}). No .gitleaksignore entry. The commit must carry a Test-Change-Justification: trailer (the live tamper guard still treats this PR's own new test file as existing). Confirm the gitleaks job is green on the pushed head.
  2. Untrusted artifact content must be re-redacted when rendered (scripts/gemini_tier_report.sh ~line 136 and scripts/token_report.sh ~line 328). Collected artifacts are untrusted JSONL: a forged gemini_rejection_sample can carry a credential-shaped string. At render time, pass every sample through the shared redaction (_gq_redact_line / redact_secrets), cap its length, and drop any record that does not match the expected shape (kind, numeric key_index, timestamp). Test: a forged record containing a fake credential renders redacted, in both reports.
  3. A key whose state is cooling must be skipped (scripts/lib/gemini-quota.sh, gq_rotation_plan ~line 436). When RPM or TPM is full gq_remaining returns cooling with an until, but the plan falls through to use, so the engine calls a key predicted to be rate-limited. Emit skip with the reason and until, as for unavailable and exhausted. unknown keys must still be tried. Test it.
  4. Markdown pipe escape is wrong (scripts/gemini_tier_report.sh ~line 120): it emits two backslashes before each pipe, so the pipe still splits the table row. Emit exactly one backslash. Test the rendered cell for a model name containing a pipe.
  5. Only genuine quota rejections may be recorded as attempts (scripts/engine.sh ~line 1067). Today any match of the shared rate-limit pattern is recorded, including 413 body-too-large and 402 payment failures, which falsely deplete local headroom and make the gate skip healthy keys. Record a gemini_attempt only for Gemini responses that indicate a quota rejection (429, RESOURCE_EXHAUSTED, quota exceeded). Tests: a 413 and a 402 record nothing; a 429 records one.

SHOULD (do in the same pass if they fit, else reply with the reason and resolve)

  1. Fleet-monitor cache key (.github/workflows/actions-fleet-monitor.yml ~line 660): include the org input and github.run_attempt in the exact key, keep the restore prefix broad enough to find the prior attempt.
  2. Do not upload the previous run's snapshot when the report step fails (~line 691): upload only on success, or remove the restored snapshot before the report runs.
  3. History store (scripts/gemini_tier_report.sh ~line 55): replace the retained store only if the merge completed; on any jq or sort error keep the old store.
  4. scripts/lib/gemini-quota.sh ~line 246: validate gemini_attempt records (valid timestamp) the same way token_usage records are validated, so a malformed one is not silently dropped from the count.
  5. AGENTS.md ~line 844: qualify the unavailable bullet; when an unknown pair precedes any usable model the tier is unknown, which is not confirmed unavailability.
  6. tests/dev-lead/unit/test_gemini_tier_view.bats ~lines 558 and 578 import jsonschema. Check whether the unit job has it. If it does not, use a validator already available there, or install the dependency in that job, and say which.
  7. tests/canary_report.bats ~line 586: assert the candidate invocation count stays 5 and the verdict stays PASS, as the neighbouring audit-record test does.

Out of scope for this PR

Anything not listed above. Do not add features.

Activity

  1. github-actions commented on Oct 3, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and for laying out the slice boundaries so clearly. The scope and acceptance criteria are well defined. One question before this is picked up: since it depends on #2033 merging first, should the dry-run poller integration (#2029) be consumed from the JSON snapshot file written by the fleet monitor, or exposed some other way, such as a step output or an artifact?

  2. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Answer: write the JSON snapshot to a single documented file path and also upload it as a workflow artifact under a fixed name. Not step outputs, since the poller (#2029) is a separate workflow and they do not cross workflows. Do not touch the poller or its workflow here: PR 2031 is unmerged. #2033 has merged, so you are unblocked. Also note the extra requirement added to the body (scope 9): rotation must drop a key/model that is unavailable (cap 0) or exhausted instead of calling it last.

  3. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2041 — Gemini availability per task tier: show remaining capacity per model, key and day so a capped model is visible before the work fails

    Scope

    Build on the merged slice 1 (#2033, scripts/lib/gemini-quota.sh): add a forward-looking remaining-capacity function, a per-tier view built on ai_models_gemini_chain, a per-Pacific-day history kept across fleet-monitor runs, a JSON snapshot with a documented schema, one notice per tier per day, ledger records for rejected attempts, escalating cooldowns, and rotation that drops cap-0 and exhausted keys. Nothing gates dispatch, and nothing calls the Gemini API.

    Implementation Checklist

    • scripts/lib/gemini-quota.sh
      • gq_day_window (current reset window start/end, DST-safe) and gq_day_bounds <n> (today plus the previous n days)
      • gq_remaining <key> <model>: JSON with state available|cooling|unavailable|exhausted|unknown, remaining rpd/rpm/tpm (never negative), ledger usage, until
      • gq_tier_view <tier> [key...]: chain from ai_models_gemini_chain, status available|degraded|unavailable|unknown, degraded_to, usable (model, key) pairs, total remaining calls
      • gq_tier_snapshot, gq_day_history <n>, gq_tier_notices (once per tier per reset-day)
      • gq_record_attempt (kind:"gemini_attempt", rejected:true, index only), gq_record_rejection_sample (redacted, one per distinct message), gq_rejection_scope (Google's documented daily wording → daily, otherwise per-minute), gq_escalated_cooldown (doubles per consecutive rejection, a success resets the streak, capped at the next reset; jumps to the reset when the ledger already shows the daily cap reached)
      • Metering counts rejected attempts as requests
      • gq_rotation_plan: keys that are unavailable or exhausted for the model are emitted as skip and not called. Keys that are over the threshold or unknown are still tried last.
    • scripts/engine.sh: record the attempt and sample on a rejection, use the escalated cooldown, handle skip; if every key is dropped, use the existing "all keys cooling" path
    • scripts/gemini_tier_report.sh (new): merge collected records into the persisted history store (deduplicated, pruned to retention), write the snapshot JSON, render the markdown report (tiers, per-pair remaining with the ledger count, per-day history, rejection samples, the known limitation), emit notices
    • scripts/lib/gemini-tier-snapshot.schema.json (new): documented schema
    • scripts/token_report.sh: optional GEMINI_RECORDS_OUT to hand Gemini records to the tier report
    • .github/workflows/actions-fleet-monitor.yml: restore and save the history with actions/cache (existing pinned SHA), run the tier report, upload the gemini-tier-snapshot artifact for the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029)
    • scripts/canary_report.sh: recognise the new audit kinds (it rejects unknown kinds)
    • AGENTS.md: document the tier view, states, escalation, rotation drop, snapshot, and the ledger-only limitation (no probe calls)

    Tests to Write

    • tests/dev-lead/unit/test_gemini_tier_view.bats: remaining-capacity states, Pacific-midnight straddle, DST day, cap 0, usage above cap; tier view with a mocked chain (degraded names the model, unavailable, unknown) plus a static no-hard-coded-model check; history persistence and retention; snapshot schema validation and a no-key-value check; once-per-day notice; attempt counting; escalation, reset on success, jump to reset
    • Rotation: a cap-0 key is skipped for that model but used for another; an exhausted key is skipped until the reset; an unknown key is still tried; when every key is dropped, the chain moves to the next model
    • Register the new file in lint.yml; canary_report.bats gets a case for the new kinds

    Test Command

    bats tests/dev-lead/unit/test_gemini_quota.bats tests/dev-lead/unit/test_gemini_tier_view.bats tests/token_report.bats tests/canary_report.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  4. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Funding update (maintainer, 2026-10-09): all four Gemini keys are now Tier 1 with a limited prepaid budget, and are to be treated identically. This supersedes the Free Tier data posted earlier on #2030 (free keys: 20 RPD per Flash model, Pro cap 0); do not use those numbers. The Tier 1 limits posted earlier apply to every key slot. Because the budget, not the rate limit, is now the binding constraint, a budget guard is tracked separately; keep this issue to the rate/availability states it describes.

  5. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet. The automation posts the authoritative completion record, with the PR number and head SHA, after the push. Treat the notes below as intent, not delivered results.

    Plan Execution (intended)

    • Remaining capacity (gq_remaining): states available, cooling, unavailable, exhausted, unknown. The day window runs from Pacific reset to Pacific reset and is DST-safe; the minute window is rolling.
    • Tier view (gq_tier_view): the chain comes from ai_models_gemini_chain. Statuses are available, degraded (with degraded_to), unavailable and unknown, plus the usable pairs and the total remaining calls.
    • Per-day history store, kept across fleet-monitor runs with actions/cache and pruned to today plus history_days.
    • JSON snapshot with a documented schema, uploaded as the gemini-tier-snapshot artifact; gq_tier_can_run reads it for the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029). It gates nothing.
    • One ::warning:: per degraded or unavailable tier per Pacific day.
    • Rejected calls are recorded as gemini_attempt records and counted as requests. Each distinct rejection message is stored once, redacted.
    • Escalating cooldown: doubles per consecutive rejection, capped at the next Pacific reset. A success resets it. If the ledger already shows the daily cap reached, the cooldown runs to the reset.
    • Rotation drops keys that are unavailable or exhausted for the model being called, with a skip reason. Keys that are unknown or only over the threshold are still tried last.
    • AGENTS.md documents all of this, plus the ledger-only limitation and the no-probe rule.

    Local check output (provisional — not the pushed run)

    New tests are in tests/dev-lead/unit/test_gemini_tier_view.bats. Shellcheck via dev-lead-lint.sh passes locally. In this sandbox, several engine-fallback suites (fix-issue, duck chain, writer fallback) also fail on unmodified main, because a real claude CLI is on PATH here. Compared one by one, they give the same results with and without this change.

    Files touched (this workspace, not yet pushed)

    • scripts/lib/gemini-quota.sh, scripts/engine.sh, scripts/gemini_tier_report.sh (new), scripts/lib/gemini-tier-snapshot.schema.json (new), scripts/lib/gemini-quota-caps.tsv, scripts/token_report.sh, scripts/canary_report.sh (new audit kinds), .github/workflows/actions-fleet-monitor.yml, .github/workflows/lint.yml, AGENTS.md, plus tests.

    Notes

  6. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2150

    The implementation for issue #2041 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  7. added 2 commits that reference this issue on Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions