Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions .github/workflows/actions-fleet-monitor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -651,8 +651,11 @@ jobs:
GH_TOKEN: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS }}
ORG: ${{ env.ORG }}
LOOKBACK_DAYS: ${{ env.LOOKBACK_DAYS }}
# Collected Gemini ledger records for the per-tier availability report (#2041).
# Collected Gemini ledger records for the per-tier availability report (#2041):
# only those traceable to a trusted run (#2147), with a report of what was
# ignored and what could not be read.
GEMINI_RECORDS_OUT: ${{ runner.temp }}/gemini-records.jsonl
GEMINI_COLLECTION_OUT: ${{ runner.temp }}/gemini-collection.json
run: bash scripts/token_report.sh >> "$GITHUB_STEP_SUMMARY"

# Gemini availability per task tier (#2041). The per-day history store survives
Expand All @@ -662,18 +665,21 @@ jobs:
# it, and every step is non-blocking. The cache and artifact steps use
# continue-on-error (a storage hiccup must not fail the fleet monitor); the
# report step warns on failure instead.
# The cache name carries `v2` (#2147): the store now holds only records from
# trusted runs, so a store filled before that filter existed is not restored.
- name: Restore Gemini tier history
continue-on-error: true
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: .gemini-tier-state
key: gemini-tier-history-${{ env.ORG }}-${{ github.run_id }}-${{ github.run_attempt }}
key: gemini-tier-history-v2-${{ env.ORG }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
gemini-tier-history-${{ env.ORG }}-
gemini-tier-history-v2-${{ env.ORG }}-

- name: Gemini availability per task tier
env:
GEMINI_RECORDS_IN: ${{ runner.temp }}/gemini-records.jsonl
GEMINI_COLLECTION_IN: ${{ runner.temp }}/gemini-collection.json
GEMINI_TIER_STATE_DIR: .gemini-tier-state
# The tier chains come from ai_models_gemini_chain: the same variables the
# engine callers export (dev-lead-reusable.yml, pr-review.yml).
Expand All @@ -692,7 +698,7 @@ jobs:
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: .gemini-tier-state
key: gemini-tier-history-${{ env.ORG }}-${{ github.run_id }}-${{ github.run_attempt }}
key: gemini-tier-history-v2-${{ env.ORG }}-${{ github.run_id }}-${{ github.run_attempt }}

- name: Upload Gemini tier snapshot
if: always()
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@
tests/test_spec_drift.bats \
tests/test_spec_drift_eval.bats \
tests/token_report.bats \
tests/token_report_trust.bats \
tests/canary_report.bats \
tests/reviewer_report.bats \
tests/test_pr_review_miss_rate.bats \
Expand Down Expand Up @@ -362,7 +363,7 @@
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# NOTE: persist-credentials is intentionally left at the default (true) —

Check warning on line 366 in .github/workflows/lint.yml

View workflow job for this annotation

GitHub Actions / Lint

366:9 [comments-indentation] comment not indented like content
# vci_resolve_reusable needs `git fetch origin` to authenticate against this
# private repo when resolving same-repo channel tags. Setting
# persist-credentials: false would silently push every such caller into the
Expand Down
40 changes: 40 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1466,6 +1466,46 @@ on it is the follow-up actuation work.
hit. The report ends with the redacted rejection messages.
- It emits one `::warning::` per tier that is `degraded` or `unavailable`, at most once
per tier per Pacific day. A `gemini_tier_notice` record in the store prevents repeats.
- **Only records from trusted runs (#2147).** A token-usage artifact is untrusted input: a
pull request from a fork runs the fork's copy of a workflow and can upload an artifact of
any name and content, and a record says nothing about its origin but a run id it wrote
itself. `scripts/token_report.sh` decides trust from the Actions API, never from the
artifact:
- **trusted:** the uploading run's head repository is the repository itself, and either
the run was on the default branch (read from the artifact listing's `workflow_run`, no
extra call), or the run, looked up, confirms the head repository and was not started by
Dependabot. These are the runs GitHub gives the repository's secrets to.
- **not trusted:** a fork run, a Dependabot run, an artifact with no `workflow_run`, and a
run that cannot be looked up or does not match. A look-up that *failed* (timeout, 403,
rate limit) is also counted as a failed read (`run_lookups_failed`), because that run
may have been a trusted one.

Every collected record is stamped `provenance: {run_id, trusted, basis}`, replacing
anything the artifact wrote under that name. Only trusted records are handed to the tier
report. The ignored ones are counted in the log, in a `::warning::` and in the job summary
("Gemini ledger provenance"). The cost report is not filtered. The history cache was
renamed (`gemini-tier-history-v2-`), so nothing collected before the filter is restored.
- **The snapshot is not yet a basis for a dispatch decision (#2147).** A forged record can
only make a tier look less available. Usage that is missing, or records that are not real
calls, are the other half, and the snapshot records both in `collection`:
- **Missing usage** makes a tier look *more* available: a repository listing, an
artifact, a run look-up or a record file that could not be read (each is counted, and
any of them makes `collection.complete` false), or simply the time since the snapshot
(it is written once a day, at 06:27 UTC, shortly before the Pacific reset).
- **Test fixtures are in the real ledger.** An agent session that runs this repository's
engine tests passes them the job's `TOKEN_LOG_FILE`, so the tests append their fixture
calls, rejections and cooldowns to the ledger the job then uploads. They arrive from a
trusted run and some look exactly like real calls. The collector counts the visible
part (`gemini_records_unlisted`: records for a key index and model the caps file does
not list) and raises a `::warning::`. The fix belongs at the source (tests must not
inherit the ledger path); until then the tier view over-counts Gemini usage.

A dispatch decision must read the snapshot with
`gq_tier_decision <snapshot> <tier> [max_age_sec]`, not `gq_tier_can_run`. It answers
`unknown` (return 2: do not start the work) when the snapshot is unreadable, older than
the limit (default 3600 s, `GEMINI_DECISION_MAX_AGE_SEC`), for a quota day that is over,
not marked `trusted-runs-only`, or built from a collection that was incomplete or held
unlisted records. Nothing calls it yet. Tests: `tests/token_report_trust.bats`.
- **Rejected attempts count.** Google counts rejected calls against the limits. So a call
rejected as rate-limited or quota-exceeded appends a `kind:"gemini_attempt"`
(`rejected:true`) record, which holds the key index, model and timestamp only. Metering
Expand Down
79 changes: 76 additions & 3 deletions scripts/gemini_tier_report.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@
# 2. writes the per-tier snapshot (snapshot.json, schema:
# scripts/lib/gemini-tier-snapshot.schema.json) — uploaded as the
# gemini-tier-snapshot artifact so the dry-run poller (#2029) can ask "can tier T
# run on Gemini now?" (gq_tier_can_run). Nothing gates or defers dispatch on it;
# run on Gemini now?" (gq_tier_can_run). Nothing gates or defers dispatch on it.
# The snapshot also records what it was built from (`collection`, #2147): whether
# the records come only from trusted runs and whether the collection was complete,
# which is what gq_tier_decision refuses to answer without;
# 3. emits one ::warning:: per degraded/unavailable tier per reset day (stderr, so
# it is an annotation, not summary text);
# 4. prints the Markdown report (stdout → the job summary).
Expand All @@ -20,7 +23,12 @@
# Environment (all optional):
# GEMINI_TIER_STATE_DIR history store + snapshot directory (default .gemini-tier-state)
# GEMINI_RECORDS_IN JSONL of newly collected records (token_report.sh
# GEMINI_RECORDS_OUT); absent → nothing new to merge
# GEMINI_RECORDS_OUT); absent → nothing new to merge.
# That hand-off is where untrusted records are dropped
# (#2147): this script believes its input file.
# GEMINI_COLLECTION_IN the collection report (token_report.sh
# GEMINI_COLLECTION_OUT); absent or malformed → the snapshot
# says its records are unverified and the collection incomplete
# GEMINI_HISTORY_DAYS previous days shown and retained (default: caps setting
# history_days, else 7)
# GEMINI_QUOTA_CAPS / GEMINI_QUOTA_NOW / AI_MODELS_GEMINI / GEMINI_*_MODEL[_CHAIN]
Expand Down Expand Up @@ -76,6 +84,52 @@ gtr_merge_store() {
mv "$tmp" "$store"
}

# gtr_collection_block [collection_report_file]
# The snapshot's `collection` object (#2147), from token_report.sh's collection
# report. `records` is "trusted-runs-only" only when the report says the filtered
# hand-off was written; `complete` only when, in addition, every repository listing,
# listed artifact, needed run look-up and record file was read (a failed read is
# usage that is not counted)
# and no handed-over record is for a key index and model the caps file does not
# list (`gemini_records_unlisted`: a ledger that tests also wrote to).
# No report, an unreadable one, or one with a field of the wrong type gives
# "unverified" / not complete: the snapshot is still written and shown, and
# gq_tier_decision answers `unknown` on it. Pure. Always prints one JSON object.
gtr_collection_block() {
local f="${1:-}" out=""
if [ -n "$f" ] && [ -r "$f" ] && [ -f "$f" ]; then
out="$(jq -c '
def n: if type == "number" and . >= 0 and . == floor then . else null end;
if type != "object" then error("not an object") else . end
| (.handed_over == true) as $h
| (.repo_listings_failed | n) as $lf | (.artifacts_failed | n) as $af
| (.run_lookups_failed | n) as $rl | (.record_files_malformed | n) as $mf
| (.gemini_records_unlisted | n) as $ul
| (if ($h | not) then "the trusted records were not handed over"
elif $lf == null or $af == null or $rl == null or $mf == null then "the collection report does not say what could not be read"
elif $lf > 0 or $af > 0 or $rl > 0 or $mf > 0 then "\($lf) repository listing(s), \($af) artifact(s), \($rl) run look-up(s) and \($mf) record file(s) could not be read"
elif $ul == null then "the collection report does not say whether the ledger holds unlisted records"
elif $ul > 0 then "\($ul) record(s) are for a key index and model the caps file does not list, so the ledger holds records that are not real calls"
else "" end) as $why
| { complete: ($why == ""),
records: (if $h then "trusted-runs-only" else "unverified" end),
reason: $why,
collected_at: (.collected_at
| if type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$")
then . else null end),
repo_listings_failed: $lf, artifacts_failed: $af,
run_lookups_failed: $rl, record_files_malformed: $mf,
gemini_records_trusted: (.gemini_records_trusted | n),
gemini_records_ignored: (.gemini_records_ignored | n),
artifacts_ignored: (.artifacts_ignored | n),
gemini_records_unlisted: $ul }' "$f" 2>/dev/null | head -1)" || out=""
fi
if [ -z "$out" ]; then
out='{"complete":false,"records":"unverified","reason":"no collection report","collected_at":null,"repo_listings_failed":null,"artifacts_failed":null,"run_lookups_failed":null,"record_files_malformed":null,"gemini_records_trusted":null,"gemini_records_ignored":null,"artifacts_ignored":null,"gemini_records_unlisted":null}'
fi
printf '%s\n' "$out"
}

# _gtr_md <text> — text safe inside a Markdown table cell.
_gtr_md() {
local s="${1//\\/\\\\}"
Expand All @@ -99,6 +153,15 @@ gtr_render() {
+ (if .window.rolling then " (daily reset not configured: rolling 24h)"
else " (\(.window.daily_reset_time) \(.window.daily_reset_tz) reset)" end)
+ " · key indexes: \(.key_indexes | join(", ") | if . == "" then "none" else . end)_\n"' "$snap"
jq -r '(.collection // { complete: false, records: "unverified", reason: "no collection report" })
| def c: if . == null then "?" else tostring end;
"_Records: "
+ (if .records == "trusted-runs-only"
then "only from trusted runs (\(.gemini_records_trusted | c) handed over, \(.gemini_records_ignored | c) ignored in \(.artifacts_ignored | c) artifact(s))"
else "provenance not verified" end)
+ " · collection "
+ (if .complete == true then "complete" else "INCOMPLETE (\(.reason // "no reason given")): usage may be missing, so this is not a basis for a dispatch decision" end)
+ "_\n"' "$snap"
printf '> **Ledger only — compare with the AI Studio viewer.** Everything here is metered from the fleet'"'"'s own ledger '
printf '(successful calls plus the rejected attempts the engine recorded). It cannot see usage of the same Google project '
printf 'by anything else (manual use, AI Studio, other tools), and a rejection the ledger missed is not counted, so Google'"'"'s '
Expand Down Expand Up @@ -176,7 +239,17 @@ main() {
[ -f "$store" ] || : > "$store"
gtr_merge_store "$store" "${GEMINI_RECORDS_IN:-}"
export GEMINI_LEDGER_FILE="$store"
gq_tier_snapshot > "$snap"
# Built beside the snapshot and moved into place whole, so a failure leaves no
# half-written snapshot; the EXIT trap removes the two working files on every way
# out (the state directory is cached and uploaded even when this step fails).
_GTR_SNAP_TMP=("$snap.tmp" "$snap.tmp.raw")
trap 'rm -f "${_GTR_SNAP_TMP[@]}"' EXIT
rm -f "${_GTR_SNAP_TMP[@]}"
gq_tier_snapshot > "$snap.tmp.raw"
jq -c --argjson c "$(gtr_collection_block "${GEMINI_COLLECTION_IN:-}")" '. + { collection: $c }' \
"$snap.tmp.raw" > "$snap.tmp"
mv "$snap.tmp" "$snap"
rm -f "${_GTR_SNAP_TMP[@]}"
gq_tier_notices "$snap" "$store" >&2
gq_day_history "$(gq_history_days)" > "$hist"
gtr_render "$snap" "$hist" "$store"
Expand Down
65 changes: 64 additions & 1 deletion scripts/lib/gemini-quota.sh
Original file line number Diff line number Diff line change
Expand Up @@ -852,7 +852,9 @@ gq_tier_snapshot() {
# gq_tier_can_run <snapshot_file> <tier> — the question a later actuation change asks
# ("can tier T run on Gemini now?"), answered from a snapshot. Prints the status
# (degraded: "degraded <model>") and returns 0 available/degraded, 1 unavailable,
# 2 unknown or no readable snapshot. Reads only; gates nothing.
# 2 unknown or no readable snapshot. Reads only; gates nothing. It reports what the
# snapshot says, however old or incomplete: a dispatch decision must call
# gq_tier_decision below instead.
gq_tier_can_run() {
local snap="$1" tier="$2" line st m
line="$(jq -r --arg t "$tier" '.tiers[] | select(.tier == $t)
Expand All @@ -866,6 +868,67 @@ gq_tier_can_run() {
esac
}

# gq_tier_decision <snapshot_file> <tier> [max_age_sec]
# The question a DISPATCH decision asks (#2147): "on this snapshot's word, may tier T
# be started on Gemini now?". gq_tier_can_run above only reads the status; this also
# asks whether the snapshot can vouch for itself, and answers `unknown` when it cannot.
# The direction matters: a forged record can only make a tier look LESS available, and
# those are filtered at collection (token_report.sh, "Trusted runs"); MISSING usage
# makes it look MORE available, so everything that can hide usage is `unknown`:
# - the snapshot is unreadable, or not a schema_version 1 object;
# - it does not say its records come only from trusted runs, or the collection
# behind it was incomplete (`collection`, written by gemini_tier_report.sh: a
# repository listing or an artifact that could not be read is usage not counted);
# - it is older than <max_age_sec> (default GEMINI_DECISION_MAX_AGE_SEC, else 3600:
# usage since the snapshot is not in it), or claims to come from the future;
# - the quota day it describes has ended, or has not started.
# Prints "<status>" or "<status> <detail>" (degraded: the model that would serve;
# unknown: the reason) and returns 0 available/degraded, 1 unavailable, 2 unknown.
# A caller must treat 2 exactly like 1: do not start the work. Pure: reads the file
# and the clock (gq_now); no network. Gates nothing by itself.
gq_tier_decision() {
local snap="$1" tier="$2" max_age="${3:-${GEMINI_DECISION_MAX_AGE_SEC:-3600}}" now out st detail
now="$(gq_now)"
if ! [[ "$max_age" =~ ^[0-9]{1,9}$ ]]; then
printf 'unknown the age limit is not a number of seconds\n'
return 2
fi
max_age=$(( 10#$max_age ))
out="$(jq -r --arg t "$tier" --argjson now "$now" --argjson max "$max_age" '
def int: type == "number" and . == floor;
if type != "object" or .schema_version != 1 then
"unknown\tthe snapshot is not a schema_version 1 object"
elif (.generated_epoch | int | not) or .generated_epoch > $now + 300 then
"unknown\tthe snapshot has no usable generation time, or one in the future"
elif ($now - .generated_epoch) > $max then
"unknown\tthe snapshot is \($now - .generated_epoch)s old (limit \($max)s): usage since then is not in it"
elif (.window.day_start | int | not) or $now < .window.day_start
or (.window.rolling != true
and ((.window.day_end | int | not) or $now >= .window.day_end)) then
"unknown\tthe quota day the snapshot describes is not the current one"
elif (.collection | type) != "object" or .collection.records != "trusted-runs-only" then
"unknown\tthe snapshot does not say its records come only from trusted runs"
elif .collection.complete != true then
"unknown\tthe collection behind the snapshot was incomplete: \(.collection.reason // "no reason given")"
Comment thread
don-petry marked this conversation as resolved.
Resolved
else
([ .tiers[]? | select(.tier == $t) ] | first) as $x
| if $x == null then "unknown\tthe snapshot has no tier \($t)"
elif $x.status == "available" then "available\t"
elif $x.status == "degraded" and ($x.degraded_to | type) == "string" then "degraded\t\($x.degraded_to)"
elif $x.status == "unavailable" then "unavailable\t"
else "unknown\tthe tier status is \($x.status // "missing")" end
end' "$snap" 2>/dev/null | head -1)" || out=""
[ -n "$out" ] || out=$'unknown\tthe snapshot could not be read'
st="${out%%$'\t'*}"
detail="${out#*$'\t'}"
case "$st" in
available) printf 'available\n'; return 0 ;;
degraded) printf 'degraded %s\n' "$detail"; return 0 ;;
unavailable) printf 'unavailable\n'; return 1 ;;
*) printf 'unknown %s\n' "$detail"; return 2 ;;
esac
}

# gq_tier_notices <snapshot_file> <state_file>
# Prints one ::warning:: per tier whose snapshot status is degraded or unavailable,
# at most ONCE per tier per reset day: each notice appends a kind:"gemini_tier_notice"
Expand Down
Loading
Loading