Repository navigation
Description
Summary
The weekly Token Cost Observatory report (#464) shows a recurring "dev-lead path: records with no model and zero usage.
This is a cosmetic/instrumentation bug, not a cost-accuracy bug — the affected records are already $0 and 0 ET, so no dollar figure is understated. But they inflate the "no price" warning and add a junk - / - row to the "Top cost drivers" table, which is misleading (it reads like real spend is being excluded when it isn't).
Evidence
From the 2026-06-29 report (comment on #464), the only unpriced rows are:
| Workflow | Tier | Model | Calls | Input | Cache | Output | Cost | % of $ | ET |
| dev-lead | - | - | 58 | 0 | 0 | 0 | $0.00* | 0% | 0 |
Model is - (absent) and all token counts are zero. Compare with a genuinely-unpriced model, which would surface with its real model string and known=0/cost=-1 — here the model field itself is empty, so the record was emitted with no model and no usage.
For contrast, every real model in the same report is priced correctly: claude-fable-5, claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5-*, gemini-2.5-pro, gemini-2.0-flash, gemini-3.5-flash, o4-mini. So the pricing table (scripts/lib/model-pricing.tsv) is not the problem and does not need new rows.
Root cause (to confirm during implementation)
Something in the dev-lead path calls emit_token_record (scripts/lib/token-metrics.sh) with an empty model and all-zero token counts — most likely an error/fallback branch that logs a record before (or without) usage being captured (e.g. an engine call that failed to parse a usage block and also had no model resolved). emit_token_record currently writes whatever it is given; it has no guard against an all-zero, model-less record.
Proposed fix
Pick whichever is cleaner; (A) is preferred because it stops the junk at the source:
A. Suppress empty records at emission. In emit_token_record, no-op when the record carries no usable signal — i.e. model is empty/- AND input+cache+cache_write+output are all zero. A record with a real model but zero tokens should still be emitted (that is a legitimate empty call); the guard is specifically for the model-less all-zero case. Add a unit test in tests/dev-lead/unit/test_token_metrics.bats.
B. (Defense in depth) Exclude them from the report's unpriced count. In scripts/token_report.sh annotate_records/totals, don't count a record toward the "no price" warning when it has no model AND zero tokens (it isn't a priced call at all). Keep the existing behavior for real models missing a price (those SHOULD still warn). Cover with a case in tests/token_report.bats.
Ideally do both: A prevents new junk records; B keeps historical/in-window junk from showing up until it ages out.
Out of scope
- No changes to
model-pricing.tsv— the table already covers every model in use. (An earlier hypothesis that new models were unpriced was disproven by the data above.)
Acceptance criteria
dev-leadno longer emits token records that have neither a model nor any token usage.- The report's "N call(s) had no price" warning counts only genuine unpriced-but-real calls (real model, no matching price row), not empty model-less records; the
- / -row no longer appears in Top cost drivers when the only such records are empty. - A record with a real model but a legitimately empty (all-zero) usage block is still emitted and handled as before.
- New/updated unit tests cover the emission guard and (if implemented) the report-side exclusion; existing tests stay green;
shellcheck scripts/*.shclean.
References
- Emission:
scripts/lib/token-metrics.sh(emit_token_record) - Report:
scripts/token_report.sh(annotate_records, totals/unpricedhandling, "no price" warning) - Tracking issue with the evidence: 📊 Token Cost Observatory — Weekly Report #464 (2026-06-27 and 2026-06-29 report comments)
- Related: Token Cost Report weekly run times out at 30m and silently stops posting #954 / feat: implement issue #954 — Token Cost Report weekly run times out at 30m and silently stops posting #955 (the collection-timeout fix that restored these reports)
Activity
github-actions commentedon Jul 2, 2026
Thanks for the detailed write-up and investigation — the evidence table and proposed fix options make this really easy to pick up! To help confirm the root cause before implementation: can you share the specific
dev-leadcode path (or a stack trace / log snippet) that triggers theemit_token_recordcall with an empty model, so we can verify option A's guard condition covers all emission sites?don-petry commentedon Jul 2, 2026
Dev-Lead Implementation Plan
Issue: #1009 — dev-lead emits empty, model-less token records that pollute the "no price" count in the cost report
Scope
Stop empty, model-less/all-zero token records at the source in
emit_token_record, and (defense in depth) drop them in the cost report'sannotate_recordsso they never reach the "no price" count or the- / -cost-driver row — while still emitting/handling a real model with a legitimately all-zero usage block.Implementation Checklist
scripts/lib/token-metrics.sh—emit_token_record: no-op whenmodelis empty/-and input+cache+cache_write+output are all zero. Real model + zero tokens still emits.scripts/token_report.sh—annotate_records: filter out records where model is empty/-and all token counts are zero, so they don't inflate the unpriced/"no price" count or the drivers table. Real-model-missing-price still warns.Tests to Write
tests/dev-lead/unit/test_token_metrics.bats— model-less all-zero record is NOT written;model="-"all-zero NOT written; real model + all-zero IS written; empty model + non-zero tokens IS written.tests/token_report.bats— a model-less all-zero record is excluded fromannotate_recordsoutput and does not add to the "had no price" count; a real unknown-model record still warns.Test Command
bats tests/dev-lead/unit/test_token_metrics.batsandbats tests/token_report.bats; full:scripts/run-bats.sh <lint.yml list>+bats tests/dev-lead/unit/. Lint:bash .dev-lead/scripts/dev-lead-lint.sh.don-petry commentedon Jul 2, 2026
Dev-Lead: Implementation Complete
Plan Execution
scripts/lib/token-metrics.sh—emit_token_recordnow no-ops whenmodelis empty/-and input + cache-read + output + cache-write are all zero. A real model with an all-zero usage block is still emitted.scripts/token_report.sh—annotate_recordsfilters out the same empty model-less/all-zero records so they don't inflate the "no price" count or render a junk- / -cost-driver row. Real models missing a price still warn (known=0).Test Results
Lint:
bash .dev-lead/scripts/dev-lead-lint.sh→ all checks passed (shellcheck on 83 files + agent-profile validation). No new shellcheck findings introduced (only pre-existing info-level notes remain).Files Changed
scripts/lib/token-metrics.sh: guard inemit_token_recordsuppressing empty, model-less, all-zero records at the source (Fix A).scripts/token_report.sh:annotate_recordsjqselectdropping the same records so the report never counts/renders them (Fix B, defense in depth for in-window historical junk).tests/dev-lead/unit/test_token_metrics.bats: 5 cases covering suppression + the legitimate-empty-call and non-zero-usage carve-outs.tests/token_report.bats: 3 cases covering annotate exclusion, the "no price" count, and the real-unknown-model warning regression guard.Notes
model-pricing.tsv(per issue: out of scope).tests/dev-lead/unit/test_engine_mcp.bats("REVIEW_MCP_DEBUG healthy handshake → ::notice::") fails on a clean tree independent of this change (MCP handshake, not token metrics) — left untouched.model=""vsmodel="-"; the guard includescache_write(9th arg), so a record whose only non-zero count is cache-creation is still emitted.feat: implement issue #1009 — dev-lead emits empty, model-less token …
test: fix misleading test name and comment per review (#1009)
feat: implement issue #1009 — dev-lead emits empty, model-less token …