Repository navigation
Description
Story
As the fleet maintainer, I want to see, per task tier, per model, per key, per day, how much Gemini capacity is left and whether the work that tier does can still run, so that a capped model is visible before work is started and fails, not after.
Slice 2 of the Gemini metering. Slice 1 is #2030 (PR #2033): it meters usage per key index and model against caps and reports peaks over a lookback. This issue builds on it and must not start until #2033 has merged.
Problem
Quotas are per model, and each fleet task tier is bound to specific models (scripts/lib/engine-models.sh: for Gemini, triage and action use gemini-3.8-flash then gemini-3.1-pro-preview; deep, audit and single use gemini-3.1-pro-preview then gemini-3.8-flash). When the model a tier needs is capped on every key, that tier's work fails. Slice 1 reports what was used; nothing answers "can tier T run on Gemini right now, and how much headroom does it have left today".
Effective limits (maintainer, 2026-10-09): every Gemini key has $1 prepaid, one-time, and the fleet is to assume the Free Tier limits for all four keys regardless of what Google would allow. The caps file therefore carries self-imposed caps: for gemini-3.8-flash 5 RPM, 250K TPM, 20 RPD; for gemini-3.1-pro-preview 0 (unavailable). So the common case is a model that is unavailable (Pro) or exhausted for the day (Flash at 20 requests), and the deep, audit and single tiers (Pro first, Flash fallback) will usually report degraded, never available. A spend estimator was considered and dropped: at 20 requests/day the $1 lasts 3 to 4 days and the existing depleted-credits probe then drops the key.
Scope
- Remaining capacity, forward-looking. A pure function that, for each (key index, model), returns remaining requests, tokens/minute and requests for the current window: daily windows run from the configured reset (Pacific midnight, from the caps file settings) to the next reset; minute windows are rolling. Result states:
available(n remaining),cooling(until T),unavailable(cap 0),exhausted(used at or above cap),unknown(limits not filled). - Tier view. For each tier (triage, action, deep, audit, single, duck) resolve its Gemini model chain with the existing
ai_models_gemini_chain(do not duplicate the chain) and report the tier status:available: its first-choice model has a usable key;degraded: only a later model in its chain has a usable key (name it, because it is a quality change);unavailable: no model in its chain has a usable key today;unknown: caps not filled.
Include which (model, key index) pairs are usable and the total remaining calls.
- Per-day history. Keep a per-Pacific-day matrix of key index x model: calls, peak per minute, number of rate-limit/cooldown events, and whether the model hit its cap that day. The report shows today and the previous N days (configurable, default 7).
- Machine-readable snapshot. Write the tier view as JSON (one object per tier, fields as above, plus a timestamp and the window boundaries used) every fleet-monitor run, and expose it to the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029) so a later actuation change can ask "can tier T run on Gemini now?". This issue does not gate or defer any dispatch. Delivery (maintainer decision 2026-10-09): the fleet-monitor step writes the snapshot to one documented file path AND uploads it as a workflow artifact under a fixed, documented name. Do not use step outputs (they do not cross workflows, and the poller is a separate workflow). Do not modify the poller or its workflow in this issue: PR 2031 (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029) is still unmerged. Document the path, the artifact name and the schema in
AGENTS.mdso the poller can consume it in a later change. - Notice on loss. When a tier transitions to
unavailableordegraded, the fleet monitor emits one notice per tier per Pacific day, not one per run. - Count rejected attempts. When a Gemini call is rejected as rate-limited or quota-exceeded, record an attempt record in the ledger (key index, model, timestamp,
rejected: true; no key value, no response body) so the per-window counts approach what Google counts. Remaining-capacity calculations use attempts, not only successes. - Escalating cooldown to the reset. In feat: implement issue #2030 — Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered #2033 a cooldown lasts for the provider's
retryDelayhint or a default, and nothing distinguishes a per-minute rejection from a daily one, so a key that exhausted its daily requests can be retried all day. Do not depend on a documented error-body format (none is documented, and no real example has been captured). Instead, each consecutive rejection of the same key and model doubles that cooldown (capped at the next daily reset in Pacific time), and a success resets it. A rejection recorded when the ledger already shows the daily cap reached sets the cooldown straight to the next reset. - If the 429 text matches Google's documented wording for a daily quota (
quota_exceeded, "daily quota"), treat it as daily; otherwise treat it as per-minute. Capture a redacted sample of each distinct rejection message in the report so a maintainer can see what Google actually sent. Fixtures in tests must come from a real captured message, never an invented one; if none is available, test only the escalation and the ledger-derived rule. - Rotation must not call a key/model that cannot run. In the merged slice 1,
gq_rotation_plan(scripts/lib/gemini-quota.sh) puts any key at or above the threshold into a "tried after the healthy keys" bucket instead of removing it. A cap of 0 computes as 100% and an exhausted daily window computes at or above 100%, so both are still called, last. Change the plan so that, for the model being called, a key whose state isunavailable(cap 0) orexhausted(used at or above its cap) is dropped from the plan and not called, reported asskipwith the reason, likecool. Keys that are merely over the 75% threshold, andunknownkeys, keep their current "tried last" behaviour:unknownmust not become a hard block while the caps file is unfilled. If every key is dropped for a model, the existing path for "all keys cooling down" applies (classified as a rate limit so the chain moves to the next model). AGENTS.mddocuments the tier view, the states, the escalation, and the limitation below.
Acceptance criteria
- Tests for the remaining-capacity function covering each state, a window that straddles Pacific midnight, a DST change day, cap 0, usage above cap (reports
exhausted, never negative remaining), andunknown. - Tests for the tier view using a mocked chain and ledger: one model capped but a later one free gives
degradedand names the model; all capped givesunavailable; the chain is read fromai_models_gemini_chain, not hard-coded (a test fails if a model id is hard-coded in the tier logic). - The per-day history survives across runs and is bounded by the retention setting.
- The JSON snapshot validates against a documented schema and a test asserts it contains no key value, only indexes.
- The once-per-tier-per-day notice is tested, including that a second run the same Pacific day does not repeat it.
- Tests for scope 6 and 7: a rejection is recorded as an attempt and counted; consecutive rejections double the cooldown up to the next Pacific reset; a success resets it; a rejection at an already-reached daily cap jumps to the reset.
- Tests for scope 9: a cap-0 key is not called for that model but is still used for another model it has capacity for; an exhausted key is not called until the Pacific reset; an
unknownkey is still tried (no new hard block); with every key dropped the engine moves to the next model. - Shellcheck clean; existing tests pass unchanged.
Known limitation to document, not to hide
All of this is metered from the fleet's own ledger. It cannot see usage of the same project by anything else (manual use, AI Studio, other tools), and Google counts attempts it rejected against the limits, which a success-only ledger does not. Google's viewer can therefore show more usage than the ledger. The report must say so and show the ledger count, so a maintainer can compare it with the AI Studio viewer.
No reconciliation or probe calls. Google's rate-limits documentation names the AI Studio web viewer as the way to see limits and describes no API that reports usage or remaining quota for a key, and a probe call would itself spend one request out of a daily cap as small as 20 (and a rejected probe still counts). Do not add any call to the Gemini API to measure or reconcile quota. Calibration against the AI Studio viewer is a manual maintainer step.
Out of scope
Deferring or gating dispatch on the snapshot, changing any model chain, and changing the engine order. Those are the actuation work, which depends on this.
Related: #2030 (PR #2033), #2029, #1565.
Review round 2 — required changes before PR 2150 can merge (maintainer, 2026-10-09)
PR 2150 has been through several passes that chased individual bot nitpicks, replied "Fixed" for changes that never landed (the harness retracted two), and tripped the tamper and regression guards. Stop chasing nitpicks. The maintainer has already resolved nine threads with reasons. Work ONLY this list, in at most TWO commits, and run bats on the touched test files (with date/sed compatible tooling as CI has) before every push. Never push a red suite. For any listed item you do not do, reply in its thread with the reason and resolve it. Do not rewrite history.
STATUS after the push to head 6044d9f (maintainer, 2026-10-09) — read this first
Verified by running the expressions, not by reading commit titles. Two things in that push are wrong and must be corrected; three MUST items are not started.
- MUST 1 (gitleaks) is NOT fixed: the scan is still red. The new allowlist regex
AIza[0-9A-Za-z_-]{39}requires 43 characters; thegcp-api-keyfinding isAIza+ 35 = 39 characters, so it never matches (tested: it does not match a 39-character key;{35}and the narrow form below do). Replace the two regexes in that[[allowlists]]entry with exactly ONE:AIza0123456789[0-9A-Za-z_-]{25}(4 + 10 + 25 = 39). Remove thefake-secretregex unless a test proves a gitleaks finding needs it. Keep the path regex andcondition = "AND". Still also build the fake key at runtime in the test so the literal is not in source. The broader{35}form is NOT acceptable: it would silently suppress a real Google key pasted into that file. - MUST 4 (pipe escape) is now WORSE, not fixed. The new line
gsub("|"; "\|")treats|as a regex alternation of two empty patterns, which matches at every position: it turnsgemini-3.8-flashinto\|g\|e\|m\|i\|n\|i\|-...(tested). The correct expression isgsub("\\|"; "\\|")in the jq source as written inside the shell script (a regex for a literal pipe, replaced by backslash-pipe); verify by running it ongemini-3.8-flash(must be unchanged) and onweird|name(must becomeweird\|name). The tests MUST cover both a plain model name (unchanged) and a model name containing a pipe, because the missing plain-name test is how this shipped. - MUST 2 (re-redact untrusted artifact samples), MUST 3 (
coolingmust be skipped in rotation) and MUST 5 (only genuine quota rejections recorded as attempts) are not started. Do them.
MUST (real bugs or security; none optional)
- Secret-scan false positive (thread on
tests/dev-lead/unit/test_gemini_tier_view.batsline ~289). Build the synthetic Google key at runtime so noAIza...literal exists in source (keep every assertion). Add exactly ONE[[allowlists]]entry to.gitleaks.tomlin the existing style (condition = "AND", path^tests/dev-lead/unit/test_gemini_tier_view\.bats$, regexAIza0123456789[A-Za-z0-9_-]{25}). No.gitleaksignoreentry. The commit must carry aTest-Change-Justification:trailer (the live tamper guard still treats this PR's own new test file as existing). Confirm the gitleaks job is green on the pushed head. - Untrusted artifact content must be re-redacted when rendered (
scripts/gemini_tier_report.sh~line 136 andscripts/token_report.sh~line 328). Collected artifacts are untrusted JSONL: a forgedgemini_rejection_samplecan carry a credential-shaped string. At render time, pass every sample through the shared redaction (_gq_redact_line/redact_secrets), cap its length, and drop any record that does not match the expected shape (kind, numeric key_index, timestamp). Test: a forged record containing a fake credential renders redacted, in both reports. - A key whose state is
coolingmust be skipped (scripts/lib/gemini-quota.sh,gq_rotation_plan~line 436). When RPM or TPM is fullgq_remainingreturnscoolingwith anuntil, but the plan falls through touse, so the engine calls a key predicted to be rate-limited. Emitskipwith the reason anduntil, as forunavailableandexhausted.unknownkeys must still be tried. Test it. - Markdown pipe escape is wrong (
scripts/gemini_tier_report.sh~line 120): it emits two backslashes before each pipe, so the pipe still splits the table row. Emit exactly one backslash. Test the rendered cell for a model name containing a pipe. - Only genuine quota rejections may be recorded as attempts (
scripts/engine.sh~line 1067). Today any match of the shared rate-limit pattern is recorded, including 413 body-too-large and 402 payment failures, which falsely deplete local headroom and make the gate skip healthy keys. Record agemini_attemptonly for Gemini responses that indicate a quota rejection (429, RESOURCE_EXHAUSTED, quota exceeded). Tests: a 413 and a 402 record nothing; a 429 records one.
SHOULD (do in the same pass if they fit, else reply with the reason and resolve)
- Fleet-monitor cache key (
.github/workflows/actions-fleet-monitor.yml~line 660): include theorginput andgithub.run_attemptin the exact key, keep the restore prefix broad enough to find the prior attempt. - Do not upload the previous run's snapshot when the report step fails (~line 691): upload only on success, or remove the restored snapshot before the report runs.
- History store (
scripts/gemini_tier_report.sh~line 55): replace the retained store only if the merge completed; on anyjqorsorterror keep the old store. scripts/lib/gemini-quota.sh~line 246: validategemini_attemptrecords (valid timestamp) the same waytoken_usagerecords are validated, so a malformed one is not silently dropped from the count.AGENTS.md~line 844: qualify theunavailablebullet; when an unknown pair precedes any usable model the tier isunknown, which is not confirmed unavailability.tests/dev-lead/unit/test_gemini_tier_view.bats~lines 558 and 578 importjsonschema. Check whether theunitjob has it. If it does not, use a validator already available there, or install the dependency in that job, and say which.tests/canary_report.bats~line 586: assert the candidate invocation count stays 5 and the verdict stays PASS, as the neighbouring audit-record test does.
Out of scope for this PR
Anything not listed above. Do not add features.
Activity
github-actions commentedon Oct 3, 2026
Thanks for the detailed write-up and for laying out the slice boundaries so clearly. The scope and acceptance criteria are well defined. One question before this is picked up: since it depends on #2033 merging first, should the dry-run poller integration (#2029) be consumed from the JSON snapshot file written by the fleet monitor, or exposed some other way, such as a step output or an artifact?
don-petry commentedon Oct 9, 2026
Answer: write the JSON snapshot to a single documented file path and also upload it as a workflow artifact under a fixed name. Not step outputs, since the poller (#2029) is a separate workflow and they do not cross workflows. Do not touch the poller or its workflow here: PR 2031 is unmerged. #2033 has merged, so you are unblocked. Also note the extra requirement added to the body (scope 9): rotation must drop a key/model that is unavailable (cap 0) or exhausted instead of calling it last.
don-petry commentedon Oct 9, 2026
Dev-Lead Implementation Plan
Issue: #2041 — Gemini availability per task tier: show remaining capacity per model, key and day so a capped model is visible before the work fails
Scope
Build on the merged slice 1 (#2033,
scripts/lib/gemini-quota.sh): add a forward-looking remaining-capacity function, a per-tier view built onai_models_gemini_chain, a per-Pacific-day history kept across fleet-monitor runs, a JSON snapshot with a documented schema, one notice per tier per day, ledger records for rejected attempts, escalating cooldowns, and rotation that drops cap-0 and exhausted keys. Nothing gates dispatch, and nothing calls the Gemini API.Implementation Checklist
scripts/lib/gemini-quota.shgq_day_window(current reset window start/end, DST-safe) andgq_day_bounds <n>(today plus the previous n days)gq_remaining <key> <model>: JSON with stateavailable|cooling|unavailable|exhausted|unknown, remaining rpd/rpm/tpm (never negative), ledger usage,untilgq_tier_view <tier> [key...]: chain fromai_models_gemini_chain, statusavailable|degraded|unavailable|unknown,degraded_to, usable (model, key) pairs, total remaining callsgq_tier_snapshot,gq_day_history <n>,gq_tier_notices(once per tier per reset-day)gq_record_attempt(kind:"gemini_attempt",rejected:true, index only),gq_record_rejection_sample(redacted, one per distinct message),gq_rejection_scope(Google's documented daily wording → daily, otherwise per-minute),gq_escalated_cooldown(doubles per consecutive rejection, a success resets the streak, capped at the next reset; jumps to the reset when the ledger already shows the daily cap reached)gq_rotation_plan: keys that areunavailableorexhaustedfor the model are emitted asskipand not called. Keys that are over the threshold orunknownare still tried last.scripts/engine.sh: record the attempt and sample on a rejection, use the escalated cooldown, handleskip; if every key is dropped, use the existing "all keys cooling" pathscripts/gemini_tier_report.sh(new): merge collected records into the persisted history store (deduplicated, pruned to retention), write the snapshot JSON, render the markdown report (tiers, per-pair remaining with the ledger count, per-day history, rejection samples, the known limitation), emit noticesscripts/lib/gemini-tier-snapshot.schema.json(new): documented schemascripts/token_report.sh: optionalGEMINI_RECORDS_OUTto hand Gemini records to the tier report.github/workflows/actions-fleet-monitor.yml: restore and save the history withactions/cache(existing pinned SHA), run the tier report, upload thegemini-tier-snapshotartifact for the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029)scripts/canary_report.sh: recognise the new audit kinds (it rejects unknown kinds)AGENTS.md: document the tier view, states, escalation, rotation drop, snapshot, and the ledger-only limitation (no probe calls)Tests to Write
tests/dev-lead/unit/test_gemini_tier_view.bats: remaining-capacity states, Pacific-midnight straddle, DST day, cap 0, usage above cap; tier view with a mocked chain (degraded names the model, unavailable, unknown) plus a static no-hard-coded-model check; history persistence and retention; snapshot schema validation and a no-key-value check; once-per-day notice; attempt counting; escalation, reset on success, jump to resetunknownkey is still tried; when every key is dropped, the chain moves to the next modellint.yml;canary_report.batsgets a case for the new kindsTest Command
bats tests/dev-lead/unit/test_gemini_quota.bats tests/dev-lead/unit/test_gemini_tier_view.bats tests/token_report.bats tests/canary_report.batsandbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Oct 9, 2026
Funding update (maintainer, 2026-10-09): all four Gemini keys are now Tier 1 with a limited prepaid budget, and are to be treated identically. This supersedes the Free Tier data posted earlier on #2030 (free keys: 20 RPD per Flash model, Pro cap 0); do not use those numbers. The Tier 1 limits posted earlier apply to every key slot. Because the budget, not the rate limit, is now the binding constraint, a budget guard is tracked separately; keep this issue to the rate/availability states it describes.
don-petry commentedon Oct 9, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
gq_remaining): statesavailable,cooling,unavailable,exhausted,unknown. The day window runs from Pacific reset to Pacific reset and is DST-safe; the minute window is rolling.gq_tier_view): the chain comes fromai_models_gemini_chain. Statuses areavailable,degraded(withdegraded_to),unavailableandunknown, plus the usable pairs and the total remaining calls.actions/cacheand pruned to today plushistory_days.gemini-tier-snapshotartifact;gq_tier_can_runreads it for the dry-run poller (Dry-run budget poller: read live Claude usage, log what the breaker would do, prove it is alive (slice 2 of the auto-pause work) #2029). It gates nothing.::warning::per degraded or unavailable tier per Pacific day.gemini_attemptrecords and counted as requests. Each distinct rejection message is stored once, redacted.unavailableorexhaustedfor the model being called, with askipreason. Keys that areunknownor only over the threshold are still tried last.Local check output (provisional — not the pushed run)
New tests are in
tests/dev-lead/unit/test_gemini_tier_view.bats. Shellcheck viadev-lead-lint.shpasses locally. In this sandbox, several engine-fallback suites (fix-issue, duck chain, writer fallback) also fail on unmodifiedmain, because a realclaudeCLI is on PATH here. Compared one by one, they give the same results with and without this change.Files touched (this workspace, not yet pushed)
scripts/lib/gemini-quota.sh,scripts/engine.sh,scripts/gemini_tier_report.sh(new),scripts/lib/gemini-tier-snapshot.schema.json(new),scripts/lib/gemini-quota-caps.tsv,scripts/token_report.sh,scripts/canary_report.sh(new audit kinds),.github/workflows/actions-fleet-monitor.yml,.github/workflows/lint.yml,AGENTS.md, plus tests.Notes
gq_tier_can_runreader, but does not wire them into the poller.don-petry commentedon Oct 9, 2026
Dev-Lead: Implementation Complete — PR #2150
The implementation for issue #2041 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
145b59ce109a61c4dcde8118a264f1972532379bmain: 13 file(s), +1811/-37 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat: implement issue #2041 — Gemini availability per task tier: show…
fix(reviews): ignore the three historical fixture findings in the Gem…
fix(reviews): drop malformed key indexes at merge; warn on a failed t…
feat: implement issue #2041 — Gemini availability per task tier: show…
feat(gemini): self-imposed Free Tier caps for all four keys, and stop…