feat(bench): measure the token economics instead of asserting them

[wenzowski]

Closes CLOUD-119.

The adoption claim is token economics, and the differentiator is not the number —
it is that ours is reproducible. This ships the harness that produces it from
committed inputs, with no credential and no network:

git clone https://github.com/button-inc/batten && cd batten
git submodule update --init && mise install
mise run token-bench          # rewrites bench/tokens/RESULTS.md
mise run token-bench-check    # proves the committed table is what a fresh run produces

Named token-bench, not bench

CLOUD-207's mise run bench measures batten's invocation cost and landed under
that name while this was in flight. Two benchmarks with two subjects need two
names, so the later and more specific one moved, and bench/tokens/ leaves room
for the sibling's own inputs beside it.

What is here

• bench/tokens/method.toml — the token divisor and the three price tiers, each with its
  source URL and the date it was read. The divisor is the one budget.rs already
  uses, so the repository has one estimator rather than two.
• bench/tokens/workloads.toml — per capability: the question, the step sequence a typical
  agent runs without Batten, and the one it runs with. The baseline is a sequence
  so the re-run is countable rather than a coefficient somebody chose.
• bench/tokens/fixtures/ — committed, hermetic inputs, carrying the .in inertness
  suffix crates/batten/tests/fixtures/repos/ already uses.
• mise-tasks/token-bench — executes what those two files declare and bakes in no price,
  divisor, re-run count or workload of its own.
• mise-tasks/token-bench-check + tests/token-bench.bats + an hk.pkl entry — the mechanism,
  since a rule without a runnable gate is half a change.

What it measures, and what it refuses to

Every arm runs three times and is compared byte-for-byte. An arm that is not
byte-stable is reported as such and never averaged — byte-stability is the
property the cross-session cache claim rests on, so losing it loses the mechanism
rather than just the precision.

token-bench-check carries two predicates. token-bench-drift is the byte-for-byte diff
against a fresh run. token-bench-unmethodical is the honesty gate, and it re-reads the
published bytes rather than trusting the generator: a figure without workload,
baseline, run count and method is refused, and so is a capability reporting "not
measured" with no reason.

Two capabilities carry a measured figure. log-handle reports "not measured"
because the capability is not shipped (CLOUD-121), and the cross-session cache hit
rate reports "not measured" because it needs a live provider credential — a
property of the world, not of this commit. Neither borrows a magnitude from its
neighbour. No aggregate is published: averaging across capabilities needs a
workload mix nobody here has measured, which would be exactly the unmethodical
figure this exists to beat.

build-warning is the one worth reading. Batten wins it at 1.47x, and the saving
is exactly the discarded tail window — not the log, which exec tees in full by
design (CLOUD-162). Whether the log itself gets cheaper is CLOUD-121's question,
and the table will move when it lands.

---

Generated by Claude Code

[linear-code]

CLOUD-119 Build the token-economics benchmark harness

Backs the Batten adoption proof design doc (top priority). No existing ticket.

Why. Adoption turns on a token-economics proof we can defend, not borrow. Batten's deterministic output contract (pointer-not-payload, byte-stable) is a measurable token-reduction mechanism; this harness measures it honestly. The market is full of unmethodical "cheaper" claims — the differentiator is that ours is reproducible.

Scope. A reproducible in-repo benchmark. Per capability, measure cost per completed task (the headline unit) and tokens-per-answer for the raw tool + typical agent handling (tail/re-run/dump) vs. the Batten-wrapped path, priced across cache-read / fresh / output tiers.

Acceptance.

• Named workloads with baselines: build/test output → warning line; scan → pointer; large log → handle; re-run → cache.
• Every figure states its workload, baseline, run count, and method; report per-capability; publish an aggregate headline only if the aggregate is itself defensible.
• Cross-session prefix-cache hit rate measured against a provider that documents caching (hit rate + time-to-first-token).
• Honesty gate: a capability with no defensible number reports "not measured," never a borrowed figure.
• Runnable by anyone from the repo.

Sources (public). Cost-of-Pass: An Economic Framework for Evaluating Language Models (arXiv:2504.13359); provider prompt-caching documentation.

Positioning input (CLOUD-211 note): the benchmark proves the token-economics story that supports the completion-gate headline (CLOUD-120 owns the headline); it is not a category proof. The note's evidence-base corrections — borrowed magnitudes that drifted from their sources — are the concrete precedent this issue's honesty gate exists for: every published figure stays quote-exact against its primary.

Review in Linear

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (f34f99a) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (4e4c6b6).

Target branch (main):

commit f34f99a3a2c842b064087c3deb6e2476f412ed29 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 18:59:53 2026 +0000

    test(findings): hold the no-escalation law over the bytes, not over a struct
    
    CLOUD-80's residual acceptance clause — "escalation on re-touch is absent" —
    was carried by an in-memory test that drives `FindingRecord::upsert` on a
    struct. That test cannot see either way a stored tier could actually move: a
    write path that re-derives it, and a later scan that re-rates the rule. Both
    need a real store dir.
    
    Two store-level tests, asserted over the record file:
    
    - `an_nth_observation_never_re_tiers_the_stored_record` runs the same identity
      through `record` five times, total over `RuleSeverity::ALL` so every
      `AdvisoryTier` is exercised, and compares the persisted `"tier"` line
      byte-for-byte against the first observation's — while asserting the
      occurrence count did move, so it cannot pass on a store that recorded
      nothing.
    - `a_re_rated_rule_never_re_tiers_a_settled_finding` records one identity at
      `warn`, then again at `deny`, and pins that the settled record keeps tier
      `caution` and severity `warn`. Severity is not an identity input, so the
      re-rating routes to the same record; `record` reusing it rather than
      refreshing from the rule now firing was documented and untested.
    
    Negative control run before landing: re-deriving the tier in `record`'s update
    path, plus a count-keyed escalation, fails both new tests and leaves the
    in-memory sibling green.
    
    Refs: CLOUD-80

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit 4e4c6b6ac7684fbc0942533fb52bc5f2e425fe64 (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run bench` executes what `bench/workloads.toml` declares and prices it with
    what `bench/method.toml` declares — no price, divisor, re-run coefficient or
    workload is baked into the program, because a constant written into a benchmark's
    code is a number nobody re-derives. Every arm runs three times and is compared
    byte-for-byte; an arm that is not byte-stable is reported as such rather than
    averaged, since byte-stability is the property the cache claim rests on.
    
    `mise run bench-check` is the mechanism that makes "reproducible" checkable: it
    regenerates the table into scratch and diffs it byte-for-byte, and it re-reads
    the published bytes for the honesty predicate — a figure without workload,
    baseline, run count and method is refused, and so is a capability reporting "not
    measured" with no reason. The gate re-reads the artifact rather than trusting the
    generator, which is what a program checking its own arithmetic would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Refs: CLOUD-119

Can't fast forward main (f34f99a) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (4e4c6b6). main (f34f99a) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (4e4c6b6). Branches appear to have diverged at ec3639a:

* 4e4c6b6ac7684fbc0942533fb52bc5f2e425fe64 feat(bench): measure the token economics instead of asserting them
| * f34f99a3a2c842b064087c3deb6e2476f412ed29 test(findings): hold the no-escalation law over the bytes, not over a struct
|/  
* ec3639a5f87a043c350ff85159db768a3fe3cfcf fix(session-start): run doctor inside the hook's synchronous window

commit ec3639a5f87a043c350ff85159db768a3fe3cfcf
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 18:51:35 2026 +0000

    fix(session-start): run doctor inside the hook's synchronous window
    
    `mise install` returning is not "the toolchain is settled". doctor provisions
    what mise does not own — the rustup cross targets above all — and left outside
    the SessionStart window it runs for the first time inside the first
    `mise run verify`, concurrently with whatever the sandbox is still laying down.
    Measured on a cold container 2026-08-07: that first verify died in doctor with a
    `detected conflict` rollback on rust-std-aarch64-apple-darwin, and a second
    doctor, nothing changed but time, exited 0. The message names cross-compilation,
    so it costs the reader a debugging session on a machine where nothing is broken.
    
    This is not a retry. It does not make the race survivable, it empties the window
    the race needs: by the time any task runs, the targets are already installed.
    CLOUD-220's per-toolchain mutex in target-ensure serializes concurrent writers
    and still stands — it covers the warm case (the verify graph racing itself,
    CLOUD-201), this covers the cold one. doctor itself is unchanged.
    
    The step goes through the hook's existing `step` helper, so it inherits the
    loud-failure contract, and runs after mise install (its rustup half needs the
    provisioned toolchain) and before container-preflight (a halt, which wants
    provisioning finished). tests/session-start.bats records every mise invocation
    through its stub and asserts that ordering, rather than grepping the source;
    doctor joins container-preflight in being neutralised there, because its rustup
    half reaches the network and test:bats already depends on `doctor --no-targets`.
    
    Refs: CLOUD-218

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (42ef663) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (04270a1).

Target branch (main):

commit 42ef66342352cf2219fd94ab2503dc228e1dd8b5 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:09:46 2026 +0000

    test(session): compare the whole store across a restart, not a subset
    
    The predicate is that a kill-and-restart preserves the findings store
    byte-for-byte, and the assertion was over the fields the test helper
    happens to parse — fingerprint, rule and instance counts. It would not
    have noticed the tier, the presentation or a disposition changing across
    the restart, which are exactly the fields no verb can vary yet and so the
    ones nothing else is watching.
    
    Refs: CLOUD-83

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit 04270a189c63ddb608bd550835d071d39e1e5180 (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run bench` executes what `bench/workloads.toml` declares and prices it with
    what `bench/method.toml` declares — no price, divisor, re-run coefficient or
    workload is baked into the program, because a constant written into a benchmark's
    code is a number nobody re-derives. Every arm runs three times and is compared
    byte-for-byte; an arm that is not byte-stable is reported as such rather than
    averaged, since byte-stability is the property the cache claim rests on.
    
    `mise run bench-check` is the mechanism that makes "reproducible" checkable: it
    regenerates the table into scratch and diffs it byte-for-byte, and it re-reads
    the published bytes for the honesty predicate — a figure without workload,
    baseline, run count and method is refused, and so is a capability reporting "not
    measured" with no reason. The gate re-reads the artifact rather than trusting the
    generator, which is what a program checking its own arithmetic would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Refs: CLOUD-119

Can't fast forward main (42ef663) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (04270a1). main (42ef663) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (04270a1). Branches appear to have diverged at 78d313a:

* 04270a189c63ddb608bd550835d071d39e1e5180 feat(bench): measure the token economics instead of asserting them
| * 42ef66342352cf2219fd94ab2503dc228e1dd8b5 test(session): compare the whole store across a restart, not a subset
| * 497f6d5b98b533c525d6da0dc678604082dfa83a test(session): pin what a warm fork must keep
| * e7165259b0afb55ca82b7d23a72b18bb6987cb88 feat(state): record where a session left the journal
| * 708d12c6e7e58b4653f18373c94798ec9eb33233 feat(session): give a warm fork somewhere to resume from
|/  
* 78d313aea99ff609bf6e24360538bd8ba4543354 fix(lock-complete): name lock-check as the remedy, not the raw regeneration command

commit 78d313aea99ff609bf6e24360538bd8ba4543354
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 18:56:11 2026 +0000

    fix(lock-complete): name lock-check as the remedy, not the raw regeneration command
    
    The unlocked-tool pointer spelled out the regeneration command, which put the
    literal into an executable line — where tests/lock-complete.bats's "makes no
    network call" assertion reads it as this gate fetching. That assertion is a
    coarse string match on purpose and is worth keeping coarse, so the message
    moves instead.
    
    `mise run lock-check` is also the better pointer on its own terms: it is the
    single sanctioned lockfile writer, since `[settings] lockfile = false` denies
    the write to everything else (CLOUD-223).
    
    Refs: CLOUD-333

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (3fde7a8) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (afc3e27).

Target branch (main):

commit 3fde7a81593c6a60d838b6006f8db6a638b4738f (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:30:27 2026 +0000

    fix(bench): keep the workflow's summary block off shellcheck's SC2016
    
    A backtick pair inside single quotes reads as legacy command substitution
    to shellcheck, which actionlint runs over every `run:` block, so the
    summary line failed the gate. Double-quoted with escaped backticks keeps
    the markdown code span and says nothing shellcheck has to guess at.
    
    Prettier's table realignment in README rides along — hk staged it.
    
    Refs: CLOUD-207

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit afc3e27f543089f55c5b7ab33c581b0f9826efda (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run bench` executes what `bench/workloads.toml` declares and prices it with
    what `bench/method.toml` declares — no price, divisor, re-run coefficient or
    workload is baked into the program, because a constant written into a benchmark's
    code is a number nobody re-derives. Every arm runs three times and is compared
    byte-for-byte; an arm that is not byte-stable is reported as such rather than
    averaged, since byte-stability is the property the cache claim rests on.
    
    `mise run bench-check` is the mechanism that makes "reproducible" checkable: it
    regenerates the table into scratch and diffs it byte-for-byte, and it re-reads
    the published bytes for the honesty predicate — a figure without workload,
    baseline, run count and method is refused, and so is a capability reporting "not
    measured" with no reason. The gate re-reads the artifact rather than trusting the
    generator, which is what a program checking its own arithmetic would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Refs: CLOUD-119

Can't fast forward main (3fde7a8) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (afc3e27). main (3fde7a8) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (afc3e27). Branches appear to have diverged at 040fa56:

* afc3e27f543089f55c5b7ab33c581b0f9826efda feat(bench): measure the token economics instead of asserting them
| * 3fde7a81593c6a60d838b6006f8db6a638b4738f fix(bench): keep the workflow's summary block off shellcheck's SC2016
| * 8854916d55b1126fcae162c601404edfc5cc6f9e feat(bench): measure the invocation cost, publish it, and gate it
|/  
* 040fa566b81b0c89387ac7bdb833e0e43d028e1c docs(resolve): mark §5's max_effect as specified, not implemented

commit 040fa566b81b0c89387ac7bdb833e0e43d028e1c
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:24:16 2026 +0000

    docs(resolve): mark §5's max_effect as specified, not implemented
    
    Two module headers asserted `max_effect` as an invariant already in the tree:
    `resolve.rs` said the raise-only override rule "extends §5's `max_effect`
    invariant to the config layer", and `verbs.rs` said sharing the effect
    vocabulary keeps "the raise-only `max_effect` rule" meaning one thing across
    the tool. Neither is true today. `max_effect` is per-flag effect annotations
    plus a monotone maximum over them; `Effect` is declared per command and carries
    no ordering, so there is nothing to take a maximum of.
    
    That is the CLOUD-198 class, and `resolve.rs` was the costly instance: it
    presented this layer's raise-only rule as a corollary of a mechanism that
    exists, so a reader checking whether config-layer monotonicity was load-bearing
    on its own found a sentence telling them it was not.
    
    Both now name the invariant as specified-not-implemented and point at CLOUD-27,
    which is where the implementation rides. No test: the Ready block puts the test
    obligation on the implementation path, and a doc comment stating a gap has
    nothing to assert until the gap closes.
    
    Refs: CLOUD-217

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (a2ed37d) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (d9759b2).

Target branch (main):

commit a2ed37d212a129bfc423cf34a378d890ce19a0d2 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:23:26 2026 +0000

    fix(lint): add the lint rows to the two committed surface lists, and fix the census's clippy denial
    
    `verify` found three things local iteration had not:
    
    - `spec.rs` pins the emitted row set and the derived read-only allowlist as two
      committed lists. Both are hand-maintained on purpose — a verb added, renamed
      or re-parented has to be stated — so `lint`/`lint brief` join them, the noun
      with its kind because the whole subtree is `read`.
    - The census's positional lookup used `map(..).unwrap_or_else(panic!)`, which is
      `clippy::map_unwrap_or`; a `let ... else` says the same thing.
    - rustfmt over the new module, the surface row and the tests.
    
    Refs: CLOUD-84

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit d9759b20395cf1bafe34bb445436247b017df02b (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
    prices it with what `bench/tokens/method.toml` declares — no price, divisor,
    re-run coefficient or workload is baked into the program, because a constant
    written into a benchmark's code is a number nobody re-derives. Every arm runs
    three times and is compared byte-for-byte; an arm that is not byte-stable is
    reported as such rather than averaged, since byte-stability is the property the
    cache claim rests on.
    
    `mise run token-bench-check` is the mechanism that makes "reproducible"
    checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
    it re-reads the published bytes for the honesty predicate — a figure without
    workload, baseline, run count and method is refused, and so is a capability
    reporting "not measured" with no reason. The gate re-reads the artifact rather
    than trusting the generator, which is what a program checking its own arithmetic
    would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
    batten's invocation cost and landed under that name first. Two benchmarks with
    two subjects need two names, so the later and more specific one moves, and
    `bench/tokens/` leaves room for the sibling's own inputs beside it.
    
    Refs: CLOUD-119

Can't fast forward main (a2ed37d) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (d9759b2). main (a2ed37d) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (d9759b2). Branches appear to have diverged at 3fde7a8:

* d9759b20395cf1bafe34bb445436247b017df02b feat(bench): measure the token economics instead of asserting them
| * a2ed37d212a129bfc423cf34a378d890ce19a0d2 fix(lint): add the lint rows to the two committed surface lists, and fix the census's clippy denial
| * bb83902973a8652f1db3dbcc68820996559d5310 feat(lint): gate a delegation brief on the facts that do not inherit
|/  
* 3fde7a81593c6a60d838b6006f8db6a638b4738f fix(bench): keep the workflow's summary block off shellcheck's SC2016

commit 3fde7a81593c6a60d838b6006f8db6a638b4738f
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:30:27 2026 +0000

    fix(bench): keep the workflow's summary block off shellcheck's SC2016
    
    A backtick pair inside single quotes reads as legacy command substitution
    to shellcheck, which actionlint runs over every `run:` block, so the
    summary line failed the gate. Double-quoted with escaped backticks keeps
    the markdown code span and says nothing shellcheck has to guess at.
    
    Prettier's table realignment in README rides along — hk staged it.
    
    Refs: CLOUD-207

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (18bb416) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (5d60ebf).

Target branch (main):

commit 18bb41610d628560c9babe07224ae6c30478c81c (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:29:23 2026 +0000

    ci(hooks): remind once when the instruction surface moves under a running session
    
    A session loads its hook wiring and reads its instruction files once, at start.
    When the contract then changes on `main`, the running session keeps executing the
    old one with no signal that it drifted: `issue-guard` landed mid-session and a
    live session readied a PR whose branch and commits carried no `CLOUD-<n>` key,
    because that hook was never in that session's wiring.
    
    `contract-drift` hashes the tracked contract surface (`AGENTS.md`,
    `.claude/rules`, `.claude/settings.json`, `hk.pkl`, `mise-tasks`) in one
    `git hash-object` pass — 63 files, 8ms — keyed per session under `.git/`, and
    names the paths that moved. Silence is the default; a change-set is reported once,
    because reporting overwrites the snapshot. Pointer-only: paths and a count, never
    a byte of the file, since a reminder carrying the new text is a mirror and a
    mirror is cleared by reading the hook instead of the file.
    
    Not `PreToolUse`, which the Ready block specified: that event's only model-facing
    channel is exit 2, which blocks the call, and CLOUD-97 and CLOUD-219 each ruled a
    deny out independently. `PostToolBatch` documents `additionalContext` firing
    "before the next model request", which is the same instant as "before the next
    lifecycle step"; `SessionStart` seeds the snapshot before any tool runs, since an
    autonomous session's first batch is routinely fetch+rebase and a snapshot written
    after it would record the drift as its baseline.
    
    Measured while building it: the `PostToolBatch` entry took effect within the
    session that added it, so the channel works and the premise is narrower than the
    title reads. The reminder therefore tells the agent to check whether a new hook is
    loaded rather than asserting it cannot be.
    
    Refs: CLOUD-187

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit 5d60ebfcd13018ef85eef1202ce87ad910980062 (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
    prices it with what `bench/tokens/method.toml` declares — no price, divisor,
    re-run coefficient or workload is baked into the program, because a constant
    written into a benchmark's code is a number nobody re-derives. Every arm runs
    three times and is compared byte-for-byte; an arm that is not byte-stable is
    reported as such rather than averaged, since byte-stability is the property the
    cache claim rests on.
    
    `mise run token-bench-check` is the mechanism that makes "reproducible"
    checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
    it re-reads the published bytes for the honesty predicate — a figure without
    workload, baseline, run count and method is refused, and so is a capability
    reporting "not measured" with no reason. The gate re-reads the artifact rather
    than trusting the generator, which is what a program checking its own arithmetic
    would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
    batten's invocation cost and landed under that name first. Two benchmarks with
    two subjects need two names, so the later and more specific one moves, and
    `bench/tokens/` leaves room for the sibling's own inputs beside it.
    
    Refs: CLOUD-119

Can't fast forward main (18bb416) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (5d60ebf). main (18bb416) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (5d60ebf). Branches appear to have diverged at a2ed37d:

* 5d60ebfcd13018ef85eef1202ce87ad910980062 feat(bench): measure the token economics instead of asserting them
| * 18bb41610d628560c9babe07224ae6c30478c81c ci(hooks): remind once when the instruction surface moves under a running session
|/  
* a2ed37d212a129bfc423cf34a378d890ce19a0d2 fix(lint): add the lint rows to the two committed surface lists, and fix the census's clippy denial

commit a2ed37d212a129bfc423cf34a378d890ce19a0d2
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:23:26 2026 +0000

    fix(lint): add the lint rows to the two committed surface lists, and fix the census's clippy denial
    
    `verify` found three things local iteration had not:
    
    - `spec.rs` pins the emitted row set and the derived read-only allowlist as two
      committed lists. Both are hand-maintained on purpose — a verb added, renamed
      or re-parented has to be stated — so `lint`/`lint brief` join them, the noun
      with its kind because the whole subtree is `read`.
    - The census's positional lookup used `map(..).unwrap_or_else(panic!)`, which is
      `clippy::map_unwrap_or`; a `let ... else` says the same thing.
    - rustfmt over the new module, the surface row and the tests.
    
    Refs: CLOUD-84

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

Heads-up on a name collision, and it is already resolved on my side.

CLOUD-207 landed mise-tasks/bench + bench-assert (latency measurement) in #324 an hour ago, which would have conflicted with this PR's bench + bench-check. CLOUD-172 renames the latency tasks into the perf namespace — perf, perf-assert, perf-pair, perf-compare, perf-gate — which is the vocabulary that ticket already specified.

So bench and bench-check are yours; nothing here needs to change. The rename is on wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency and will be on main shortly. If you rebase before it lands you will see mise-tasks/bench from #324 — that is the file being renamed away, not a competing implementation.

One thing worth knowing when you do rebase: mise.toml now pins aqua:sharkdp/hyperfine and ci.yml has a new perf job in final's needs and in CI_REQUIRED_CHECKS.

---

Generated by Claude Code

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (6947232) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (20d012e).

Target branch (main):

commit 6947232b13bc17174986a0e3bfe528d848617519 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:57:59 2026 +0000

    perf(hk): glob batten-check and macos-link-check, and gate the glob's honesty
    
    `hk check` ran three steps unconditionally, because a step with no `glob` is
    never skipped. `batten-check` is `cargo run -p batten -- check`, so every
    commit paid a debug build whatever it touched: 9.47s cold from a fresh clone
    (CLOUD-212 run 1, D8) against 0.16s for the cheapest gate in the same run, and
    133-141ms warm, forever.
    
    The glob is the union of what `batten check` ACTUALLY reads, which is not the
    two things the obvious reading gives. Read at run_check_with in
    crates/batten/src/lib.rs, it evaluates every `[[rule]]` over that rule's own
    glob — and those reach outside crates/, to mise.toml, .github/workflows/*.yml
    and tests/*.bats — plus `budget::measure_all`, because a declared budget is a
    gate under `check` and not only under `policy budget` (CLOUD-50). That last one
    makes AGENTS.md and .serena/project.yml genuine inputs, so "a Markdown-only
    change cannot change this verdict" is false for the most-edited Markdown file
    in the repo, and a glob written from the obvious reading would have switched
    the gate off for it. Cargo.toml and Cargo.lock are in because the task uses
    `cargo run` deliberately, to judge the engine and the config as the pair that
    ships.
    
    `macos-link-check` globs the manifests alone: its predicate is `cargo metadata
    --filter-platform` over the resolved graph, and nothing else can move that
    graph. `no-docs-tree` deliberately keeps no glob — its input is the whole
    index, including a docs/ path an earlier commit left behind, and at ~95ms that
    is affordable.
    
    Deriving batten-check's list from batten.toml by hand makes hk.pkl a second
    authority over a set the config already defines, and a second authority narrows
    silently: add a rule whose glob names a new path and the step stops running for
    commits that touch only it, with nothing going red. So the glob ships with its
    mechanism (non-negotiable 2). `mise run batten-glob-check` is a pure function
    of the two committed files — every rule glob and every declared budget path
    must be covered verbatim or subsumed by a `P/**` entry, and a config it can
    parse nothing out of is exit 2 rather than a vacuous pass.
    
    Selection is asserted as data, never as a timing: `hk check --plan --json`
    prints each step's status without running it, and tests/hk-selection.bats pins
    both directions over the repository's own hk.pkl — README.md skips
    batten-check, AGENTS.md does not.
    
    Refs: CLOUD-224

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit 20d012ec9d860710f6af6bd76504c5985e8a4e1b (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
    prices it with what `bench/tokens/method.toml` declares — no price, divisor,
    re-run coefficient or workload is baked into the program, because a constant
    written into a benchmark's code is a number nobody re-derives. Every arm runs
    three times and is compared byte-for-byte; an arm that is not byte-stable is
    reported as such rather than averaged, since byte-stability is the property the
    cache claim rests on.
    
    `mise run token-bench-check` is the mechanism that makes "reproducible"
    checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
    it re-reads the published bytes for the honesty predicate — a figure without
    workload, baseline, run count and method is refused, and so is a capability
    reporting "not measured" with no reason. The gate re-reads the artifact rather
    than trusting the generator, which is what a program checking its own arithmetic
    would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
    batten's invocation cost and landed under that name first. Two benchmarks with
    two subjects need two names, so the later and more specific one moves, and
    `bench/tokens/` leaves room for the sibling's own inputs beside it.
    
    Refs: CLOUD-119

Can't fast forward main (6947232) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (20d012e). main (6947232) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (20d012e). Branches appear to have diverged at 6569aeb:

* 20d012ec9d860710f6af6bd76504c5985e8a4e1b feat(bench): measure the token economics instead of asserting them
| * 6947232b13bc17174986a0e3bfe528d848617519 perf(hk): glob batten-check and macos-link-check, and gate the glob's honesty
|/  
* 6569aeb303e942642dd22feab96f221c67f7027e feat(rules)!: rename the command kind's `run` column to `check`, reserve `fix`

commit 6569aeb303e942642dd22feab96f221c67f7027e
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:02:59 2026 +0000

    feat(rules)!: rename the command kind's `run` column to `check`, reserve `fix`
    
    House style §9 specifies the `command` rule kind as a check/fix duality: an
    inspection-only `check` is the gate — the only side enforcement ever runs — and
    an optional mutating `fix` repairs what it condemned. The shipped `Rule` carried
    one `run` column, which does not express the split.
    
    §2 declares no back-compatibility surface and no deprecation aliases, so every
    config written against `run` becomes a breaking migration later. Today the key
    appears in no shipped config at all — only fixtures, tests and the README — so
    the rename costs nothing. It will not stay that way.
    
    `fix` parses and is executed by nothing: serialised fix execution is a
    capability this engine does not have. `enforce` refuses a row declaring one
    rather than running the check side and silently ignoring the repair, which would
    report green over work nobody did. Reserving the key now means an author who
    writes it is writing the final spelling.
    
    `run` survives as a refusal-only field, the shape `OverrideConfig::
    min_batten_version` already uses: "unknown field `run`" reads as a typo, where
    this is a rename with one specific fix, and the refusal has to carry it
    (CLOUD-122). It is deliberately outside `Rule::columns` — that census classifies
    the columns a kind may carry, and no kind carries this one.
    
    Both refusals are exit 1, not the exit 2 the issue's §5/§7 asked for. §7 makes 2
    the policy verdict — a violation found in the repository, or a mediated call
    denied — while a config naming a key or a capability this build does not have is
    the config-or-usage class. §5 spells the structurally identical sibling refusal
    (a rule the read-only half cannot honestly run) as exit 1, and `run_static` has
    implemented it that way since CLOUD-170; the issue's §3 describes that same
    landed refusal as exit 2, which is falsified by the code it describes. Emitting
    2 here would also make an engine gap read to a harness as a policy deny, which
    is what the structural fail-open exists to prevent.
    
    Refs: CLOUD-215

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[wenzowski]

/fast-forward

[wenzowski]

Triggered from #311 (comment) by @​wenzowski.

Trying to fast forward main (b642e2d) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (465534b).

Target branch (main):

commit b642e2dd35304790923f2a63bbbf0f9f68c9b4c2 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 20:47:33 2026 +0000

    perf(ci): run the shell suite in parallel, since it is the pole of the gate
    
    `hk` runs two chains concurrently, and the shell one was the longer by 133s:
    on run 31531728503 the cargo chain's last step finished 166s into `mise run
    ci`, while `test:bats` ran to 247.01s of the step's 298.87s. 800 tests, one at
    a time. `verify` runs the same gate, so a `land` lap paid it twice.
    
    That put it in the exponent of the lap economics. A lap dies whenever `main`
    moves under it, so with mean inter-land gap 487s the expected CI runs per
    landed PR is e^(L/G) — which is why shortening the lap is the one lever that
    cuts latency and the bill together rather than trading one for the other.
    
    Measured here: 248s serial -> 72.73s, 808 tests, same assertions, nothing
    skipped or sampled. The remaining pole is the cargo chain at 166s.
    
    `rush` rather than GNU parallel, which bats would otherwise look for: GNU
    parallel ships as a source tarball with no aqua/ubi/release coordinate, so it
    cannot be a [tools] key at all, and an `apt-get install` in ci.yml is the
    second-authority drift `ci-tools-check` exists to refuse. rush is prebuilt per
    platform, so `mise.lock` holds a checksum for every platform this repo
    installs on.
    
    Across files, never within one. The suites that model lock and race behaviour
    are precisely the ones a within-file schedule would interleave with their own
    setup, and across-files is already enough — the largest file is 49 of 808
    tests, so the schedule is not tail-bound.
    
    Two things ship with it:
    
    - `tests/attribution-check.bats` built its end-to-end fixture in the real
      worktree and the real index. Serially that was invisible; in parallel it is
      a race against the sibling case that reads the same tracked set, plus
      .git/index.lock contention. It now builds a throwaway repo, which is the
      convention issue-guard.bats already set.
    - A speed-up is the kind of fix that rots silently — drop the flag and
      everything still passes, just slower, and `timeout-check`'s 3.4x headroom
      absorbs a return to serial without a red build. So the invocation is now a
      gate over mise.toml: serial is an exit code, not a slower Tuesday.
    
    Refs: CLOUD-386

Pull request (wenzowski/cloud-119-build-the-token-economics-benchmark-harness):

commit 465534bc3a182573385a16f84cff8d8f52d1f2fc (pull_request/wenzowski/cloud-119-build-the-token-economics-benchmark-harness)
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 19:14:30 2026 +0000

    feat(bench): measure the token economics instead of asserting them
    
    The adoption claim is token economics, and the differentiator is that ours is
    reproducible: anyone can say they are cheaper, we ship the harness that produces
    the number from committed inputs with no credential and no network.
    
    `mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
    prices it with what `bench/tokens/method.toml` declares — no price, divisor,
    re-run coefficient or workload is baked into the program, because a constant
    written into a benchmark's code is a number nobody re-derives. Every arm runs
    three times and is compared byte-for-byte; an arm that is not byte-stable is
    reported as such rather than averaged, since byte-stability is the property the
    cache claim rests on.
    
    `mise run token-bench-check` is the mechanism that makes "reproducible"
    checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
    it re-reads the published bytes for the honesty predicate — a figure without
    workload, baseline, run count and method is refused, and so is a capability
    reporting "not measured" with no reason. The gate re-reads the artifact rather
    than trusting the generator, which is what a program checking its own arithmetic
    would not do.
    
    Two capabilities carry a measured figure; `log-handle` reports "not measured"
    because the capability is not shipped, and the cross-session cache hit rate
    reports "not measured" because it needs a live provider credential and is a
    property of the world rather than of this commit. Neither borrows a magnitude
    from its neighbour. No aggregate is published: averaging across capabilities
    needs a workload mix nobody here has measured.
    
    `build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
    exactly the discarded `tail` window rather than the log, which `exec` tees in
    full by design.
    
    Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
    batten's invocation cost and landed under that name first. Two benchmarks with
    two subjects need two names, so the later and more specific one moves, and
    `bench/tokens/` leaves room for the sibling's own inputs beside it.
    
    Refs: CLOUD-119

Can't fast forward main (b642e2d) to wenzowski/cloud-119-build-the-token-economics-benchmark-harness (465534b). main (b642e2d) is not a direct ancestor of wenzowski/cloud-119-build-the-token-economics-benchmark-harness (465534b). Branches appear to have diverged at 5532160:

* b642e2dd35304790923f2a63bbbf0f9f68c9b4c2 perf(ci): run the shell suite in parallel, since it is the pole of the gate
| * 465534bc3a182573385a16f84cff8d8f52d1f2fc feat(bench): measure the token economics instead of asserting them
|/  
* 55321601f0fb1cf8e5d9c5a97129e36b11df62fb chore(deps): drop dev-dependencies that duplicate the normal ones

commit 55321601f0fb1cf8e5d9c5a97129e36b11df62fb
Author: Claude <noreply@anthropic.com>
Date:   Tue Aug 11 20:04:52 2026 +0000

    chore(deps): drop dev-dependencies that duplicate the normal ones
    
    `serde_json` was listed in both `[dependencies]` and `[dev-dependencies]`, and
    so were `flate2`, `tar` and `toml` — the identical redundancy, in the same
    table, filed as one bullet because one of the four was noticed. A normal
    dependency is already in scope for every test target, so all four entries
    declared nothing; the comment above them explained a provision-suite need that
    `[dependencies]` had long since covered.
    
    `jsonschema` stays: it is the one crate the tests link and the shipped binary
    does not.
    
    Predicate is the Ready block's — the crate and its test targets still build.
    
    Refs: CLOUD-217

Rebase locally, and then force push to wenzowski/cloud-119-build-the-token-economics-benchmark-harness.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[wenzowski]

/fast-forward