Skip to content

The Gemini token ledger is mostly test fixtures: dev-lead agent sessions run the engine tests with the real TOKEN_LOG_FILE set, and the run uploads what the tests wrote #2304

Closed

Description

@don-petry

Problem

The org-wide Gemini usage ledger, which the per-tier availability report is built from, is almost entirely test fixture data.

Measured on 2026-10-11 over about four days of token-usage-* artifacts (1,353 artifacts, 6,637 Gemini records, 156 artifacts with any Gemini record):

Artifacts Request-counting Gemini records (token_usage + gemini_attempt)
Clean 76 8
Contaminated by test fixtures 80 1,811
of which visibly fixture (a model or key index with no row in the caps file: gemini-same, gemini-test, gemini-u1, gemini-pinned, key 1b, …) 1,102
of which indistinguishable from a real call (gemini-3.8-flash / gemini-3.1-pro-preview, keys 1 to 4) 709

So about 8 of 1,819 counted Gemini requests were real. The tier view over-counts Gemini use today, and its "Unparsed rate-limit messages" section is mostly fixture strings ("hit your limit", "quota exceeded", "429 too many requests"; see the read on "A weekly Claude cap parses as 'unknown reset'", #1863).

Cause

No test job uploads these. They come from trusted Dev-Lead Agent runs:

  1. The dispatch job of dev-lead-reusable.yml sets TOKEN_LOG_FILE for the whole job (line 184).
  2. The agent session inherits it. When the agent runs this repository's engine tests as part of its work, tests that do not isolate the variable append their fixture calls, rejections and cooldowns to the real ledger. Seven engine suites never mention the variable: test_engine_chain, test_engine_chain_config, test_engine_duck_copilot_model, test_engine_duck_model, test_engine_gemini_35, test_engine_models, test_engine_new_models.
  3. The "Upload token usage log" step uploads the file.

Because the uploader is dev-lead itself and one artifact mixes real and fixture records, no provenance rule can filter them out afterwards. It has to be fixed where the records are written.

Scope

  1. Tests never see the ambient ledger. In tests/dev-lead/unit/setup_suite.bash, unset TOKEN_LOG_FILE for the whole suite (CI's own bats job runs with it unset, so this is the behaviour the tests are already written for); a test that wants a ledger sets its own under $BATS_TEST_TMPDIR. Do the same for any suite under tests/ that reaches engine.sh outside that directory.
  2. The model subprocess does not inherit it. engine.sh records tokens in the parent, around the CLI call; the child session has no use for the variable. Removing it from the child's environment stops whatever the session runs from writing to the ledger, tests or not. The path is also predictable (/tmp/token-usage-<run id>.jsonl), so consider whether the session should be able to write there at all.
  3. A static test: no suite that sources engine.sh runs with an inherited TOKEN_LOG_FILE.

Acceptance criteria

  1. With TOKEN_LOG_FILE exported to a file, running bats tests/dev-lead/unit/ leaves that file empty or absent.
  2. A stub engine child process started by run_writer does not see TOKEN_LOG_FILE in its environment, and the parent still records the call.
  3. After the fix has reached the ring this repository pins, a day of artifacts contains no Gemini record whose model or key index has no caps row (the detector added under "Gemini trickle under a budget pause", Gemini trickle under a budget pause: let a bounded slice of cheap work run on Gemini while the Claude breaker has paused the fleet (default off) #2147, counts these).

Blocks any use of the tier view for a dispatch decision (#2147).

Activity

  1. github-actions commented on Oct 11, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up and the measurements; the breakdown of clean versus contaminated artifacts makes the cause easy to follow. Your proposed fix looks clear, so my one question is about acceptance criterion 1. Should the static test in scope item 3 also cover suites outside tests/dev-lead/ that source engine.sh, or only the ones under tests/dev-lead/unit/?

  2. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    dev-lead is withholding action on this item.

    It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

    To re-enable automated pickup: remove the dev-lead:hands-off label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-lead:hands-offExclude from dev-lead agent automation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions