You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The Gemini token ledger is mostly test fixtures: dev-lead agent sessions run the engine tests with the real TOKEN_LOG_FILE set, and the run uploads what the tests wrote #2304
The org-wide Gemini usage ledger, which the per-tier availability report is built from, is almost entirely test fixture data.
Measured on 2026-10-11 over about four days of token-usage-* artifacts (1,353 artifacts, 6,637 Gemini records, 156 artifacts with any Gemini record):
Artifacts
Request-counting Gemini records (token_usage + gemini_attempt)
Clean
76
8
Contaminated by test fixtures
80
1,811
of which visibly fixture (a model or key index with no row in the caps file: gemini-same, gemini-test, gemini-u1, gemini-pinned, key 1b, …)
1,102
of which indistinguishable from a real call (gemini-3.8-flash / gemini-3.1-pro-preview, keys 1 to 4)
709
So about 8 of 1,819 counted Gemini requests were real. The tier view over-counts Gemini use today, and its "Unparsed rate-limit messages" section is mostly fixture strings ("hit your limit", "quota exceeded", "429 too many requests"; see the read on "A weekly Claude cap parses as 'unknown reset'", #1863).
Cause
No test job uploads these. They come from trusted Dev-Lead Agent runs:
The dispatch job of dev-lead-reusable.yml sets TOKEN_LOG_FILE for the whole job (line 184).
The agent session inherits it. When the agent runs this repository's engine tests as part of its work, tests that do not isolate the variable append their fixture calls, rejections and cooldowns to the real ledger. Seven engine suites never mention the variable: test_engine_chain, test_engine_chain_config, test_engine_duck_copilot_model, test_engine_duck_model, test_engine_gemini_35, test_engine_models, test_engine_new_models.
The "Upload token usage log" step uploads the file.
Because the uploader is dev-lead itself and one artifact mixes real and fixture records, no provenance rule can filter them out afterwards. It has to be fixed where the records are written.
Scope
Tests never see the ambient ledger. In tests/dev-lead/unit/setup_suite.bash, unset TOKEN_LOG_FILE for the whole suite (CI's own bats job runs with it unset, so this is the behaviour the tests are already written for); a test that wants a ledger sets its own under $BATS_TEST_TMPDIR. Do the same for any suite under tests/ that reaches engine.sh outside that directory.
The model subprocess does not inherit it.engine.sh records tokens in the parent, around the CLI call; the child session has no use for the variable. Removing it from the child's environment stops whatever the session runs from writing to the ledger, tests or not. The path is also predictable (/tmp/token-usage-<run id>.jsonl), so consider whether the session should be able to write there at all.
A static test: no suite that sources engine.sh runs with an inherited TOKEN_LOG_FILE.
Acceptance criteria
With TOKEN_LOG_FILE exported to a file, running bats tests/dev-lead/unit/ leaves that file empty or absent.
A stub engine child process started by run_writer does not see TOKEN_LOG_FILE in its environment, and the parent still records the call.
Thanks for the thorough write-up and the measurements; the breakdown of clean versus contaminated artifacts makes the cause easy to follow. Your proposed fix looks clear, so my one question is about acceptance criterion 1. Should the static test in scope item 3 also cover suites outside tests/dev-lead/ that source engine.sh, or only the ones under tests/dev-lead/unit/?
It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.
To re-enable automated pickup: remove the dev-lead:hands-off label.
Activity
github-actions commentedon Oct 11, 2026
Thanks for the thorough write-up and the measurements; the breakdown of clean versus contaminated artifacts makes the cause easy to follow. Your proposed fix looks clear, so my one question is about acceptance criterion 1. Should the static test in scope item 3 also cover suites outside
tests/dev-lead/that sourceengine.sh, or only the ones undertests/dev-lead/unit/?don-petry commentedon Oct 11, 2026
dev-lead is withholding action on this item.
It is labeled
dev-lead:hands-off(deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.To re-enable automated pickup: remove the
dev-lead:hands-offlabel.test(engine): pin every model CLI start to the ledger-free launcher
fix(engine): keep the token ledger out of test suites and out of the …