Skip to content

dev-lead emits empty, model-less token records that pollute the "no price" count in the cost report #1009

Closed

Description

@don-petry

Summary

The weekly Token Cost Observatory report (#464) shows a recurring "⚠️ N call(s) had no price" warning (58 on the 2026-06-29 run, 121 on 2026-06-27). Investigation shows these are not a pricing-table gap — every real model in use is already priced. They are empty, model-less token records emitted by the dev-lead path: records with no model and zero usage.

This is a cosmetic/instrumentation bug, not a cost-accuracy bug — the affected records are already $0 and 0 ET, so no dollar figure is understated. But they inflate the "no price" warning and add a junk - / - row to the "Top cost drivers" table, which is misleading (it reads like real spend is being excluded when it isn't).

Evidence

From the 2026-06-29 report (comment on #464), the only unpriced rows are:

| Workflow | Tier | Model | Calls | Input | Cache | Output | Cost | % of $ | ET |
| dev-lead |  -   |  -    |  58   |   0   |   0   |   0    | $0.00* |  0%  | 0 |

Model is - (absent) and all token counts are zero. Compare with a genuinely-unpriced model, which would surface with its real model string and known=0/cost=-1 — here the model field itself is empty, so the record was emitted with no model and no usage.

For contrast, every real model in the same report is priced correctly: claude-fable-5, claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5-*, gemini-2.5-pro, gemini-2.0-flash, gemini-3.5-flash, o4-mini. So the pricing table (scripts/lib/model-pricing.tsv) is not the problem and does not need new rows.

Root cause (to confirm during implementation)

Something in the dev-lead path calls emit_token_record (scripts/lib/token-metrics.sh) with an empty model and all-zero token counts — most likely an error/fallback branch that logs a record before (or without) usage being captured (e.g. an engine call that failed to parse a usage block and also had no model resolved). emit_token_record currently writes whatever it is given; it has no guard against an all-zero, model-less record.

Proposed fix

Pick whichever is cleaner; (A) is preferred because it stops the junk at the source:

A. Suppress empty records at emission. In emit_token_record, no-op when the record carries no usable signal — i.e. model is empty/- AND input+cache+cache_write+output are all zero. A record with a real model but zero tokens should still be emitted (that is a legitimate empty call); the guard is specifically for the model-less all-zero case. Add a unit test in tests/dev-lead/unit/test_token_metrics.bats.

B. (Defense in depth) Exclude them from the report's unpriced count. In scripts/token_report.sh annotate_records/totals, don't count a record toward the "no price" warning when it has no model AND zero tokens (it isn't a priced call at all). Keep the existing behavior for real models missing a price (those SHOULD still warn). Cover with a case in tests/token_report.bats.

Ideally do both: A prevents new junk records; B keeps historical/in-window junk from showing up until it ages out.

Out of scope

  • No changes to model-pricing.tsv — the table already covers every model in use. (An earlier hypothesis that new models were unpriced was disproven by the data above.)

Acceptance criteria

  • dev-lead no longer emits token records that have neither a model nor any token usage.
  • The report's "N call(s) had no price" warning counts only genuine unpriced-but-real calls (real model, no matching price row), not empty model-less records; the - / - row no longer appears in Top cost drivers when the only such records are empty.
  • A record with a real model but a legitimately empty (all-zero) usage block is still emitted and handled as before.
  • New/updated unit tests cover the emission guard and (if implemented) the report-side exclusion; existing tests stay green; shellcheck scripts/*.sh clean.

References

Activity

  1. github-actions commented on Jul 2, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and investigation — the evidence table and proposed fix options make this really easy to pick up! To help confirm the root cause before implementation: can you share the specific dev-lead code path (or a stack trace / log snippet) that triggers the emit_token_record call with an empty model, so we can verify option A's guard condition covers all emission sites?

  2. don-petry commented on Jul 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1009 — dev-lead emits empty, model-less token records that pollute the "no price" count in the cost report

    Scope

    Stop empty, model-less/all-zero token records at the source in emit_token_record, and (defense in depth) drop them in the cost report's annotate_records so they never reach the "no price" count or the - / - cost-driver row — while still emitting/handling a real model with a legitimately all-zero usage block.

    Implementation Checklist

    • A. scripts/lib/token-metrics.sh — emit_token_record: no-op when model is empty/- and input+cache+cache_write+output are all zero. Real model + zero tokens still emits.
    • B. scripts/token_report.sh — annotate_records: filter out records where model is empty/- and all token counts are zero, so they don't inflate the unpriced/"no price" count or the drivers table. Real-model-missing-price still warns.

    Tests to Write

    • tests/dev-lead/unit/test_token_metrics.bats — model-less all-zero record is NOT written; model="-" all-zero NOT written; real model + all-zero IS written; empty model + non-zero tokens IS written.
    • tests/token_report.bats — a model-less all-zero record is excluded from annotate_records output and does not add to the "had no price" count; a real unknown-model record still warns.

    Test Command

    bats tests/dev-lead/unit/test_token_metrics.bats and bats tests/token_report.bats; full: scripts/run-bats.sh <lint.yml list> + bats tests/dev-lead/unit/. Lint: bash .dev-lead/scripts/dev-lead-lint.sh.

  3. don-petry commented on Jul 2, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • A. scripts/lib/token-metrics.sh — emit_token_record now no-ops when model is empty/- and input + cache-read + output + cache-write are all zero. A real model with an all-zero usage block is still emitted.
    • B. scripts/token_report.sh — annotate_records filters out the same empty model-less/all-zero records so they don't inflate the "no price" count or render a junk - / - cost-driver row. Real models missing a price still warn (known=0).
    • Tests written first (red), then implementation (green).

    Test Results

    ### tests/dev-lead/unit/test_token_metrics.bats  (47 tests, 0 failures)
    ok 26 emit_token_record: suppresses an empty model-less all-zero record
    ok 27 emit_token_record: suppresses a '-' model all-zero record (incl. cache_write)
    ok 28 emit_token_record: still emits a real model with an all-zero usage block
    ok 30 emit_token_record: suppresses model-less record whose only tokens are cache_write
    
    ### tests/token_report.bats  (31 tests, 0 failures)
    ok 10 annotate_records: excludes empty model-less all-zero records (#1009)
    ok 11 render_token_report: empty model-less junk does not add to the 'no price' count
    ok 12 render_token_report: a real unknown-model call still warns (#1009 regression guard)
    

    Lint: bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed (shellcheck on 83 files + agent-profile validation). No new shellcheck findings introduced (only pre-existing info-level notes remain).

    Files Changed

    • scripts/lib/token-metrics.sh: guard in emit_token_record suppressing empty, model-less, all-zero records at the source (Fix A).
    • scripts/token_report.sh: annotate_records jq select dropping the same records so the report never counts/renders them (Fix B, defense in depth for in-window historical junk).
    • tests/dev-lead/unit/test_token_metrics.bats: 5 cases covering suppression + the legitimate-empty-call and non-zero-usage carve-outs.
    • tests/token_report.bats: 3 cases covering annotate exclusion, the "no price" count, and the real-unknown-model warning regression guard.

    Notes

    • No changes to model-pricing.tsv (per issue: out of scope).
    • Pre-existing, unrelated failure in tests/dev-lead/unit/test_engine_mcp.bats ("REVIEW_MCP_DEBUG healthy handshake → ::notice::") fails on a clean tree independent of this change (MCP handshake, not token metrics) — left untouched.
    • Edge cases handled: model="" vs model="-"; the guard includes cache_write (9th arg), so a record whose only non-zero count is cache-creation is still emitted.
  4. added 3 commits that reference this issue on Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions