Repository navigation
feat(perf): record the invocation cost as a series, and gate the regression (CLOUD-172) - #333
Conversation
CLOUD-172 Record batten's invocation cost as a series, so hook latency can't degrade invisibly
Batten runs as a hook, so its startup cost is paid on every mediated call. A policy engine slow enough to be annoying gets routed around, and the rule it enforces stops binding. Cost measured once degrades invisibly; a per-commit series makes a regression attributable to a SHA. Mechanism
Blocked on
Ready
Done
Decisions taken during implementationMetric: wall clock, paired — instruction counts are not available. What makes wall clock usable is pairing rather than a better clock: Threshold: 1.30x, derived from a measured null. Identical binary as both arms, 100 runs after 10 warmups, ten repeats across three paths — 30 ratios of a comparison that by construction measures nothing. They spread 0.966 to 1.102. 1.30 clears that measured maximum with room for a busier runner. Recording trigger: the daily schedule, not a push-to- Where the decision is recorded. The Ready block asks for it "in AGENTS.md beside it". AGENTS.md is at exactly 199 lines, the Namespace: CLOUD-207 Publish and defend the latency number: bench task + CI assertion on the no-op and hook paths
A Scope: a Refine against the DoR before pulling; the flakiness posture applies to the CI assertion (a shared-runner p95 needs a tolerance band, not a point equality). CLOUD-119 Build the token-economics benchmark harness
Backs the Batten adoption proof design doc (top priority). No existing ticket. Why. Adoption turns on a token-economics proof we can defend, not borrow. Batten's deterministic output contract (pointer-not-payload, byte-stable) is a measurable token-reduction mechanism; this harness measures it honestly. The market is full of unmethodical "cheaper" claims — the differentiator is that ours is reproducible. Scope. A reproducible in-repo benchmark. Per capability, measure cost per completed task (the headline unit) and tokens-per-answer for the raw tool + typical agent handling ( Acceptance.
Sources (public). Cost-of-Pass: An Economic Framework for Evaluating Language Models (arXiv:2504.13359); provider prompt-caching documentation. Positioning input (CLOUD-211 note): the benchmark proves the token-economics story that supports the completion-gate headline (CLOUD-120 owns the headline); it is not a category proof. The note's evidence-base corrections — borrowed magnitudes that drifted from their sources — are the concrete precedent this issue's honesty gate exists for: every published figure stays quote-exact against its primary. CLOUD-418 A new gate is never shown to fail, so a test that cannot discriminate ships as coverage
Why This repository's most-repeated failure is a claim nothing exercises. It happened again, live, while building the landing lease (CLOUD-393). A concurrency test was written for a real race — That was found only because someone chose to mutate and re-run — a discipline nothing asks for and nothing checks. The green suite before that check and the green suite after it were indistinguishable. Root cause. The obligation is stated as "a rule ships with a runnable gate" — a gate that exists. Nothing requires evidence the gate discriminates. A test that passes on both the fixed and the broken code satisfies every rule this repo currently has. Scope, deliberately narrow. Not mutation testing over the workspace, which is a research project and a large CI bill. The claim here is about Refinement — Ready
Test obligation The mechanism must catch the case that motivated it: the Commit / bump (§6): Blockers (§8): none. Acceptance
|
5f5c3b4 to
7877127
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit 55321601f0fb1cf8e5d9c5a97129e36b11df62fb (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:04:52 2026 +0000
chore(deps): drop dev-dependencies that duplicate the normal ones
`serde_json` was listed in both `[dependencies]` and `[dev-dependencies]`, and
so were `flate2`, `tar` and `toml` — the identical redundancy, in the same
table, filed as one bullet because one of the four was noticed. A normal
dependency is already in scope for every test target, so all four entries
declared nothing; the comment above them explained a provision-suite need that
`[dependencies]` had long since covered.
`jsonschema` stays: it is the one crate the tests link and the shipped binary
does not.
Predicate is the Ready block's — the crate and its test targets still build.
Refs: CLOUD-217Pull request ( commit 78771272504611cf0b1ae9174dd748df98582cd5 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * 78771272504611cf0b1ae9174dd748df98582cd5 feat(perf): record the invocation cost as a series, and gate the regression
| * 55321601f0fb1cf8e5d9c5a97129e36b11df62fb chore(deps): drop dev-dependencies that duplicate the normal ones
|/
* 6947232b13bc17174986a0e3bfe528d848617519 perf(hk): glob batten-check and macos-link-check, and gate the glob's honesty
commit 6947232b13bc17174986a0e3bfe528d848617519
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:57:59 2026 +0000
perf(hk): glob batten-check and macos-link-check, and gate the glob's honesty
`hk check` ran three steps unconditionally, because a step with no `glob` is
never skipped. `batten-check` is `cargo run -p batten -- check`, so every
commit paid a debug build whatever it touched: 9.47s cold from a fresh clone
(CLOUD-212 run 1, D8) against 0.16s for the cheapest gate in the same run, and
133-141ms warm, forever.
The glob is the union of what `batten check` ACTUALLY reads, which is not the
two things the obvious reading gives. Read at run_check_with in
crates/batten/src/lib.rs, it evaluates every `[[rule]]` over that rule's own
glob — and those reach outside crates/, to mise.toml, .github/workflows/*.yml
and tests/*.bats — plus `budget::measure_all`, because a declared budget is a
gate under `check` and not only under `policy budget` (CLOUD-50). That last one
makes AGENTS.md and .serena/project.yml genuine inputs, so "a Markdown-only
change cannot change this verdict" is false for the most-edited Markdown file
in the repo, and a glob written from the obvious reading would have switched
the gate off for it. Cargo.toml and Cargo.lock are in because the task uses
`cargo run` deliberately, to judge the engine and the config as the pair that
ships.
`macos-link-check` globs the manifests alone: its predicate is `cargo metadata
--filter-platform` over the resolved graph, and nothing else can move that
graph. `no-docs-tree` deliberately keeps no glob — its input is the whole
index, including a docs/ path an earlier commit left behind, and at ~95ms that
is affordable.
Deriving batten-check's list from batten.toml by hand makes hk.pkl a second
authority over a set the config already defines, and a second authority narrows
silently: add a rule whose glob names a new path and the step stops running for
commits that touch only it, with nothing going red. So the glob ships with its
mechanism (non-negotiable 2). `mise run batten-glob-check` is a pure function
of the two committed files — every rule glob and every declared budget path
must be covered verbatim or subsumed by a `P/**` entry, and a config it can
parse nothing out of is exit 2 rather than a vacuous pass.
Selection is asserted as data, never as a timing: `hk check --plan --json`
prints each step's status without running it, and tests/hk-selection.bats pins
both directions over the repository's own hk.pkl — README.md skips
batten-check, AGENTS.md does not.
Refs: CLOUD-224Rebase locally, and then force push to |
7877127 to
5766e2d
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit b642e2dd35304790923f2a63bbbf0f9f68c9b4c2 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:47:33 2026 +0000
perf(ci): run the shell suite in parallel, since it is the pole of the gate
`hk` runs two chains concurrently, and the shell one was the longer by 133s:
on run 31531728503 the cargo chain's last step finished 166s into `mise run
ci`, while `test:bats` ran to 247.01s of the step's 298.87s. 800 tests, one at
a time. `verify` runs the same gate, so a `land` lap paid it twice.
That put it in the exponent of the lap economics. A lap dies whenever `main`
moves under it, so with mean inter-land gap 487s the expected CI runs per
landed PR is e^(L/G) — which is why shortening the lap is the one lever that
cuts latency and the bill together rather than trading one for the other.
Measured here: 248s serial -> 72.73s, 808 tests, same assertions, nothing
skipped or sampled. The remaining pole is the cargo chain at 166s.
`rush` rather than GNU parallel, which bats would otherwise look for: GNU
parallel ships as a source tarball with no aqua/ubi/release coordinate, so it
cannot be a [tools] key at all, and an `apt-get install` in ci.yml is the
second-authority drift `ci-tools-check` exists to refuse. rush is prebuilt per
platform, so `mise.lock` holds a checksum for every platform this repo
installs on.
Across files, never within one. The suites that model lock and race behaviour
are precisely the ones a within-file schedule would interleave with their own
setup, and across-files is already enough — the largest file is 49 of 808
tests, so the schedule is not tail-bound.
Two things ship with it:
- `tests/attribution-check.bats` built its end-to-end fixture in the real
worktree and the real index. Serially that was invisible; in parallel it is
a race against the sibling case that reads the same tracked set, plus
.git/index.lock contention. It now builds a throwaway repo, which is the
convention issue-guard.bats already set.
- A speed-up is the kind of fix that rots silently — drop the flag and
everything still passes, just slower, and `timeout-check`'s 3.4x headroom
absorbs a return to serial without a red build. So the invocation is now a
gate over mise.toml: serial is an exit code, not a slower Tuesday.
Refs: CLOUD-386Pull request ( commit 5766e2df1aca1812c5e19a1a9257a58bbd3d81cd (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * 5766e2df1aca1812c5e19a1a9257a58bbd3d81cd feat(perf): record the invocation cost as a series, and gate the regression
| * b642e2dd35304790923f2a63bbbf0f9f68c9b4c2 perf(ci): run the shell suite in parallel, since it is the pole of the gate
|/
* 55321601f0fb1cf8e5d9c5a97129e36b11df62fb chore(deps): drop dev-dependencies that duplicate the normal ones
commit 55321601f0fb1cf8e5d9c5a97129e36b11df62fb
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:04:52 2026 +0000
chore(deps): drop dev-dependencies that duplicate the normal ones
`serde_json` was listed in both `[dependencies]` and `[dev-dependencies]`, and
so were `flate2`, `tar` and `toml` — the identical redundancy, in the same
table, filed as one bullet because one of the four was noticed. A normal
dependency is already in scope for every test target, so all four entries
declared nothing; the comment above them explained a provision-suite need that
`[dependencies]` had long since covered.
`jsonschema` stays: it is the one crate the tests link and the shipped binary
does not.
Predicate is the Ready block's — the crate and its test targets still build.
Refs: CLOUD-217Rebase locally, and then force push to |
5766e2d to
e10072e
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit 98aceb05393d0fdfc91d977603ae486cf34e1b7f (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:14:30 2026 +0000
feat(bench): measure the token economics instead of asserting them
The adoption claim is token economics, and the differentiator is that ours is
reproducible: anyone can say they are cheaper, we ship the harness that produces
the number from committed inputs with no credential and no network.
`mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
prices it with what `bench/tokens/method.toml` declares — no price, divisor,
re-run coefficient or workload is baked into the program, because a constant
written into a benchmark's code is a number nobody re-derives. Every arm runs
three times and is compared byte-for-byte; an arm that is not byte-stable is
reported as such rather than averaged, since byte-stability is the property the
cache claim rests on.
`mise run token-bench-check` is the mechanism that makes "reproducible"
checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
it re-reads the published bytes for the honesty predicate — a figure without
workload, baseline, run count and method is refused, and so is a capability
reporting "not measured" with no reason. The gate re-reads the artifact rather
than trusting the generator, which is what a program checking its own arithmetic
would not do.
Two capabilities carry a measured figure; `log-handle` reports "not measured"
because the capability is not shipped, and the cross-session cache hit rate
reports "not measured" because it needs a live provider credential and is a
property of the world rather than of this commit. Neither borrows a magnitude
from its neighbour. No aggregate is published: averaging across capabilities
needs a workload mix nobody here has measured.
`build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
exactly the discarded `tail` window rather than the log, which `exec` tees in
full by design.
Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
batten's invocation cost and landed under that name first. Two benchmarks with
two subjects need two names, so the later and more specific one moves, and
`bench/tokens/` leaves room for the sibling's own inputs beside it.
Refs: CLOUD-119Pull request ( commit e10072e2331d69f65d22b066ad270178e56e7e48 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * e10072e2331d69f65d22b066ad270178e56e7e48 feat(perf): record the invocation cost as a series, and gate the regression
| * 98aceb05393d0fdfc91d977603ae486cf34e1b7f feat(bench): measure the token economics instead of asserting them
|/
* b642e2dd35304790923f2a63bbbf0f9f68c9b4c2 perf(ci): run the shell suite in parallel, since it is the pole of the gate
commit b642e2dd35304790923f2a63bbbf0f9f68c9b4c2
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:47:33 2026 +0000
perf(ci): run the shell suite in parallel, since it is the pole of the gate
`hk` runs two chains concurrently, and the shell one was the longer by 133s:
on run 31531728503 the cargo chain's last step finished 166s into `mise run
ci`, while `test:bats` ran to 247.01s of the step's 298.87s. 800 tests, one at
a time. `verify` runs the same gate, so a `land` lap paid it twice.
That put it in the exponent of the lap economics. A lap dies whenever `main`
moves under it, so with mean inter-land gap 487s the expected CI runs per
landed PR is e^(L/G) — which is why shortening the lap is the one lever that
cuts latency and the bill together rather than trading one for the other.
Measured here: 248s serial -> 72.73s, 808 tests, same assertions, nothing
skipped or sampled. The remaining pole is the cargo chain at 166s.
`rush` rather than GNU parallel, which bats would otherwise look for: GNU
parallel ships as a source tarball with no aqua/ubi/release coordinate, so it
cannot be a [tools] key at all, and an `apt-get install` in ci.yml is the
second-authority drift `ci-tools-check` exists to refuse. rush is prebuilt per
platform, so `mise.lock` holds a checksum for every platform this repo
installs on.
Across files, never within one. The suites that model lock and race behaviour
are precisely the ones a within-file schedule would interleave with their own
setup, and across-files is already enough — the largest file is 49 of 808
tests, so the schedule is not tail-bound.
Two things ship with it:
- `tests/attribution-check.bats` built its end-to-end fixture in the real
worktree and the real index. Serially that was invisible; in parallel it is
a race against the sibling case that reads the same tracked set, plus
.git/index.lock contention. It now builds a throwaway repo, which is the
convention issue-guard.bats already set.
- A speed-up is the kind of fix that rots silently — drop the flag and
everything still passes, just slower, and `timeout-check`'s 3.4x headroom
absorbs a return to serial without a red build. So the invocation is now a
gate over mise.toml: serial is an exit code, not a slower Tuesday.
Refs: CLOUD-386Rebase locally, and then force push to |
e10072e to
22913a4
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit 87ed39733221612abd4f99ce64c5f93e773439d4 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 18:56:29 2026 +0000
fix(verify): refuse to receipt a HEAD whose bytes were not the ones validated
`verify` runs every gate over the WORKING TREE and then writes a receipt keyed
to HEAD. Nothing asserted the two agree, so a dirty tree could attest a commit
nobody verified: the receipt is written, `verified` matches it, `ready-guard`
accepts it, and CI runs a commit that was never the thing that passed.
Measured 2026-08-09 landing CLOUD-269 — a backgrounded `land` compiled a
mid-edit snapshot and reported `verify failed on c5bfe05`, a commit containing
none of that code. That direction is loud and self-correcting; the mirror is
silent, and it is the one this closes.
`mise-tasks/tree-clean` is the predicate: `git --no-optional-locks status
--porcelain --untracked-files=normal`, exit 1 with a count and the porcelain
lines as pointers, exit 2 when it cannot look. Untracked-but-not-ignored counts
as dirty — decided, not omitted — because `cargo test` auto-discovers
`crates/batten/tests/*.rs` and the bats suite globs `tests/*.bats`, so a new
untracked file is run by `verify` with zero tracked-file change. Ignored paths
are excluded structurally.
Wired into `[tasks.verify]` at both ends of the window: in `depends` so a dirty
tree fails in seconds rather than after ~170s, and in the body — the
load-bearing one, since `depends` completes before a tree can be dirtied
mid-run. Not an hk step: `pre-commit` runs over a tree that is dirty by
definition. `land` inherits it through `verify` and re-implements nothing.
Refs: CLOUD-277Pull request ( commit 22913a4fc97ed5d2cc9b83e22fd7fe24b4f86ed8 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * 22913a4fc97ed5d2cc9b83e22fd7fe24b4f86ed8 feat(perf): record the invocation cost as a series, and gate the regression
| * 87ed39733221612abd4f99ce64c5f93e773439d4 fix(verify): refuse to receipt a HEAD whose bytes were not the ones validated
|/
* 98aceb05393d0fdfc91d977603ae486cf34e1b7f feat(bench): measure the token economics instead of asserting them
commit 98aceb05393d0fdfc91d977603ae486cf34e1b7f
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:14:30 2026 +0000
feat(bench): measure the token economics instead of asserting them
The adoption claim is token economics, and the differentiator is that ours is
reproducible: anyone can say they are cheaper, we ship the harness that produces
the number from committed inputs with no credential and no network.
`mise run token-bench` executes what `bench/tokens/workloads.toml` declares and
prices it with what `bench/tokens/method.toml` declares — no price, divisor,
re-run coefficient or workload is baked into the program, because a constant
written into a benchmark's code is a number nobody re-derives. Every arm runs
three times and is compared byte-for-byte; an arm that is not byte-stable is
reported as such rather than averaged, since byte-stability is the property the
cache claim rests on.
`mise run token-bench-check` is the mechanism that makes "reproducible"
checkable: it regenerates the table into scratch and diffs it byte-for-byte, and
it re-reads the published bytes for the honesty predicate — a figure without
workload, baseline, run count and method is refused, and so is a capability
reporting "not measured" with no reason. The gate re-reads the artifact rather
than trusting the generator, which is what a program checking its own arithmetic
would not do.
Two capabilities carry a measured figure; `log-handle` reports "not measured"
because the capability is not shipped, and the cross-session cache hit rate
reports "not measured" because it needs a live provider credential and is a
property of the world rather than of this commit. Neither borrows a magnitude
from its neighbour. No aggregate is published: averaging across capabilities
needs a workload mix nobody here has measured.
`build-warning` is the honest one — Batten wins it by 1.47x, and the saving is
exactly the discarded `tail` window rather than the log, which `exec` tees in
full by design.
Named `token-bench` rather than `bench`: CLOUD-207's `mise run bench` measures
batten's invocation cost and landed under that name first. Two benchmarks with
two subjects need two names, so the later and more specific one moves, and
`bench/tokens/` leaves room for the sibling's own inputs beside it.
Refs: CLOUD-119Rebase locally, and then force push to |
22913a4 to
5df9003
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit 42319f43bfdff5cdb654394d2504d1ac173cb024 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:44:36 2026 +0000
ci(stop-posture): one flagging-verb set across the worth/bears openers
`HEDGES` gave its two openers different verb sets — `worth
(noting|flagging)` beside `bears (noting|mentioning)` — so `mentioning`
was a flagging verb to this file under one opener and unknown under the
other. Nothing distinguished them; the split was an accident of how the
alternation was written.
Measured, one sentence per row: `worth naming` and `worth mentioning`
were silent while `bears mentioning` fired. The first of those is a
witnessed miss. The CLOUD-347..356 audit closed its report with "One
open thread worth naming: the census … never interrogated host
settings" — a real finding that reached chat and nothing else, and
became CLOUD-380 only because a human asked. Nothing else covered it
either: `stop-guard` consults `finding-sink-check` precisely when this
rule is silent, and that gate needs a `path:line` citation the sentence
does not carry, so the turn's one advisory slot was never claimed.
One shared `FLAG_VERB` set now, every entry either witnessed (`naming`)
or already in the file (`noting`, `flagging`, `mentioning`). `calling
out` stays out and is asserted to stay out: unwitnessed and not already
present, which is the line between completing an inflection and
inventing a phrase list. The `I would flag` / `I should note` family
keeps `(flag|note)` — "I would name that" is not natural flagging.
Not the unmeasured-literal mistake CLOUD-323 and CLOUD-326 forbid:
those govern a new SHAPE, while this is inflection completion inside a
construction already measured at 3/3, which this gate's own header
sanctions and its own history already did once.
Refs: CLOUD-387Pull request ( commit 5df900392c9d9336ed15bf7de869f71bb3566db3 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * 5df900392c9d9336ed15bf7de869f71bb3566db3 feat(perf): record the invocation cost as a series, and gate the regression
| * 42319f43bfdff5cdb654394d2504d1ac173cb024 ci(stop-posture): one flagging-verb set across the worth/bears openers
|/
* 87ed39733221612abd4f99ce64c5f93e773439d4 fix(verify): refuse to receipt a HEAD whose bytes were not the ones validated
commit 87ed39733221612abd4f99ce64c5f93e773439d4
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 18:56:29 2026 +0000
fix(verify): refuse to receipt a HEAD whose bytes were not the ones validated
`verify` runs every gate over the WORKING TREE and then writes a receipt keyed
to HEAD. Nothing asserted the two agree, so a dirty tree could attest a commit
nobody verified: the receipt is written, `verified` matches it, `ready-guard`
accepts it, and CI runs a commit that was never the thing that passed.
Measured 2026-08-09 landing CLOUD-269 — a backgrounded `land` compiled a
mid-edit snapshot and reported `verify failed on c5bfe05`, a commit containing
none of that code. That direction is loud and self-correcting; the mirror is
silent, and it is the one this closes.
`mise-tasks/tree-clean` is the predicate: `git --no-optional-locks status
--porcelain --untracked-files=normal`, exit 1 with a count and the porcelain
lines as pointers, exit 2 when it cannot look. Untracked-but-not-ignored counts
as dirty — decided, not omitted — because `cargo test` auto-discovers
`crates/batten/tests/*.rs` and the bats suite globs `tests/*.bats`, so a new
untracked file is run by `verify` with zero tracked-file change. Ignored paths
are excluded structurally.
Wired into `[tasks.verify]` at both ends of the window: in `depends` so a dirty
tree fails in seconds rather than after ~170s, and in the body — the
load-bearing one, since `depends` completes before a tree can be dirtied
mid-run. Not an hk step: `pre-commit` runs over a tree that is dirty by
definition. `land` inherits it through `verify` and re-implements nothing.
Refs: CLOUD-277Rebase locally, and then force push to |
42953b6 to
cd775a3
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit ac7da09517068209d6d27197e908d2ef617762ef (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:02:37 2026 +0000
docs(agents): price the metered model call, and index every memory the repo ships
The workflow contract opened with a two-row price list — "Every CI run costs
real minutes; your own execution costs nothing" — and a token-consuming model
call is in neither row. It is not local and it is not free, so an agent reading
only the always-loaded surface correctly concluded its own fan-out was free.
Three categories now: local execution, free; a CI run, metered; a token-consuming
model call, metered in the same category, a subagent spawn above all. The
derivation stays out of the always-loaded surface and the clause points at
`mem:workflow/agent-fanout` for it — which is the memory that had no trigger
anywhere, along with `core` and `memory_maintenance`. All three get an index row,
so the routing table now covers every memory the repo ships.
Both were paid for out of reclaimed lines rather than appended: the surface sits
at its 199-line target with zero headroom, so `land`'s step, the governing
paragraph and two background paragraphs were repacked and their *why* clauses
dropped where `.claude/rules/toolchain.md` already carries them. 199 lines before
and after, 28 insertions against 28 deletions.
Refs: CLOUD-288Pull request ( commit cd775a319b56dc8d152ab6760d323abf2d0e6fab (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 20:36:37 2026 +0000
feat(perf): record the invocation cost as a series, and gate the regression
CLOUD-207 published one number. A number measured once decays in
silence: nobody notices 3ms becoming 40ms, they notice months later
that the hook got uninstalled.
`perf-record` appends each trunk measurement to git notes on
`refs/notes/perf`, keyed to the commit, and refuses to run off main —
a branch's numbers are not the trunk's, and a series mixing them
cannot be read. `perf-pair` builds the merge-base and HEAD binaries
and measures them back to back on one machine; `perf-compare` decides
that ratio; `perf-gate` composes the two and is the one name verify
and CI call.
WALL CLOCK, PAIRED. The issue specified instruction counts with wall
clock as a fallback "where valgrind is unavailable" — `mise registry
valgrind` reports "tool not found in registry", and no-source-built-tool
forbids compiling one, so the fallback is the only branch. What makes
it usable is not a better clock but a better experiment: both arms
measured on the same runner within seconds, so machine noise is
common-mode and divides out. The verdict is a ratio, never an absolute,
and every series entry carries `metric=wall-clock` so a later
instruction-count series cannot be diffed against it as a regression.
THE THRESHOLD IS MEASURED. A null comparison — identical binary as both
arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum;
`perf-pair --null` re-measures it and a bats case fails if the constant
is ever tightened below the recorded floor.
Recording trigger: the daily schedule, not a push-to-main workflow,
which AGENTS.md forbids and whose reason holds here. The cost is
resolution, affordable because the series is not the attribution
mechanism — perf-gate catches a regression on the PR that caused it.
Verify pays almost nothing: perf-gate exits clean without building when
the diff touches no crate source, manifest or lockfile, because a
binary that cannot have changed cannot have got slower.
Renames bench/bench-assert into the perf namespace the issue specifies,
freeing `bench` for #311's token-economics harness.
Refs: CLOUD-172Can't fast forward * cd775a319b56dc8d152ab6760d323abf2d0e6fab feat(perf): record the invocation cost as a series, and gate the regression
| * ac7da09517068209d6d27197e908d2ef617762ef docs(agents): price the metered model call, and index every memory the repo ships
|/
* b1669d6f503804d7102754c992931a9c31359d6d docs(readme): lead with the completion gate, not the policy-engine category
commit b1669d6f503804d7102754c992931a9c31359d6d
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:17:41 2026 +0000
docs(readme): lead with the completion gate, not the policy-engine category
"Policy engine" as the lead noun claims membership in a category whose
repo-level population measures near-zero, and it invites an expressiveness
benchmark against authorization engines Batten does not implement and must not
grow. The category is conformance checking: the subject is always the
repository's own state, and the question is always whether observed state
conforms to the committed model of how work lands here. A principal/action/
resource request shape has no field for "is this ref on `main`" or "did the
required checks conclude green for this exact SHA".
So the lead becomes the gate itself, hook-first, with CI and pre-commit demoted
to what they actually are — bypass defense over the same predicate, not two
more adoption surfaces. "Permission hooks" becomes "conformance gates" for the
same reason: the hook's allow/deny channel is the only authorization-shaped
part, and the verdict inside it is a conformance verdict.
Every "done" claim now carries its bound. Landed-plus-green is the minimum
falsifiable completion predicate: it kills false success, it does not certify
correctness, and review still gates release. Stated because the evidence
behind it demonstrates that merge alone is not a verification event — it does
not establish that the compound predicate certifies anything more.
The token-economics story moves in under the headline as the supporting
narrative it is, naming the three pains it addresses (tail-calling,
lost-in-the-middle, context rot) against the two contract properties that
answer them — pointer-not-payload and byte-stable output — plus the refusal
that points at the fix. Magnitude is deliberately absent: the benchmark
carries it, per workload with a stated baseline and run count, and a
capability with no defensible number reports "not measured" rather than
borrowing one.
The crate keywords go with the description: "policy" as a discovery term
recruits exactly the category readers this change is spending effort to stop
attracting.
Untouched on purpose: the exit-code table and the whole extension-surfaces
section, both asserted against the compiled binary by tests that would fail if
a rewrite clipped them.
Refs: CLOUD-120Rebase locally, and then force push to |
cd775a3 to
2f16b27
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit 2d3acc3416b5d9ed3ca9ec5c3280194b11ef757f (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 18:52:11 2026 +0000
ci(graph-check): make a frontier omission attributable, and say "could not look"
`graph-check` is the scheduler as well as the gate, and its header claims every
agent computing the frontier independently gets the same answer. It did not:
`:92` ran ready-lint as `| "$lint" >/dev/null 2>&1 || continue`, so exit 1 ("this
Ready block is wrong") and exit 2 ("stdin is not a payload with a .description
field") arrived as the same silent absence — one missing `frontier` line and
nothing else, exit 0 both times over a 25-payload closure.
The same shape reached the verdict: `:40` required only id and status, so a set
whose relations the caller projected away was reported an acyclic, non-dangling
board with no edges in it at all.
Three channels now, not one. `report()` keeps exit 1 for a board signalling
falsely; `unjudged()` takes exit 2 for anything the gate could not read; `note()`
attributes an honest exclusion without moving the exit code, because a board full
of unrefined issues is not an incoherent board. ready-lint's rule ids are
forwarded rather than re-derived, and its ::error:: summary dropped — annotating
a failure at an exit code that denies one is the false signal this repo catches.
The missing-blockedBy report is keyed to the `graph` pseudo-id, not to each
issue: `released`'s `refusal_for` greps this stderr for `^<id> <rule>`, and a
per-id line would turn every In Review issue in a relations-free sweep into a
REFUSED. An explicit `blockedBy: []` is data and is judged; only an absent key
is unjudgeable.
Exit 2 outranks exit 1 when both hold, with both report sets printed: a verdict
over a set the gate could only partly read is not a verdict, and the caller's
next action is a re-fetch.
Refs: CLOUD-251Pull request ( commit 2f16b275d2100232e515d0fdfbb81f15f1f6c010 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 21:35:17 2026 +0000
docs(fanout): record that a bundle serializes on land, inside one session
The fan-out memory reasoned about land contention BETWEEN sessions and
said nothing about within one. A session handed a bundle meets the same
constraint as strict serialization for a mechanical reason: `land`
drives the working tree, so ticket N+1 cannot start while N's land is in
flight.
Measured on this bundle — each land ran 5+ laps at roughly 5-7 minutes,
so wall clock was dominated by laps the session could only wait out. The
unblock is a worktree per ticket, and `batten worktree` already models
the question.
Refs: CLOUD-172Can't fast forward * 2f16b275d2100232e515d0fdfbb81f15f1f6c010 docs(fanout): record that a bundle serializes on land, inside one session
* eeb04963d014cf21d05b250879aa7bccc8fe7280 feat(perf): record the invocation cost as a series, and gate the regression
| * 2d3acc3416b5d9ed3ca9ec5c3280194b11ef757f ci(graph-check): make a frontier omission attributable, and say "could not look"
|/
* ac7da09517068209d6d27197e908d2ef617762ef docs(agents): price the metered model call, and index every memory the repo ships
commit ac7da09517068209d6d27197e908d2ef617762ef
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 19:02:37 2026 +0000
docs(agents): price the metered model call, and index every memory the repo ships
The workflow contract opened with a two-row price list — "Every CI run costs
real minutes; your own execution costs nothing" — and a token-consuming model
call is in neither row. It is not local and it is not free, so an agent reading
only the always-loaded surface correctly concluded its own fan-out was free.
Three categories now: local execution, free; a CI run, metered; a token-consuming
model call, metered in the same category, a subagent spawn above all. The
derivation stays out of the always-loaded surface and the clause points at
`mem:workflow/agent-fanout` for it — which is the memory that had no trigger
anywhere, along with `core` and `memory_maintenance`. All three get an index row,
so the routing table now covers every memory the repo ships.
Both were paid for out of reclaimed lines rather than appended: the surface sits
at its 199-line target with zero headroom, so `land`'s step, the governing
paragraph and two background paragraphs were repacked and their *why* clauses
dropped where `.claude/rules/toolchain.md` already carries them. 199 lines before
and after, 28 insertions against 28 deletions.
Refs: CLOUD-288Rebase locally, and then force push to |
2f16b27 to
4ec18ac
Compare
|
/fast-forward |
|
Triggered from #333 (comment) by @wenzowski. Trying to fast forward Target branch ( commit eee9bb4bade0799923aff7d983911a169743d839 (HEAD -> main, origin/main)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 21:10:41 2026 +0000
feat(config): gate the predecessor consumer's repo name
CLOUD-73 asks what stays in the consumer's repo and what moves into Batten. Its
structural half is settled by history — this repository was scaffolded fresh, so
none of the domain modules it names (entity grep, pdfium, CA privacy detectors,
evidence and provenance semantics, the ICM design protocol) has ever been here,
and there is no `tools/batten/` to keep them out of. Its other half is
non-negotiable rule 1, and that one had a hole.
CLOUD-26's acceptance named four literals. Two are rows; the comment beside them
dropped the other two on the ground that `forbid` was a case-sensitive literal
substring check, so a literal with an English reading was wrong in both
directions at once. CLOUD-283 gave `forbid` a `regex` alternative, which retires
that premise — so the two exclusions are re-decided on the merits rather than
inherited from a limit that no longer exists.
The predecessor consumer's repo name goes in, as `(?i)complian`: it is squarely
what rule 1 forbids, `(?i)` catches the capitalised form the old literal would
have missed, and it costs nothing — zero lines under crates/ carry it, and this
engine's vocabulary is policy / gate / predicate / verdict.
The vendor name stays out, and the comment now says that is a decision rather
than a gap: it names the vendor, not a consumer, `github.com/button-inc/batten`
is Batten's own home, and the consumer-facing half is already gated by
`no-origin-literal-in-fixtures`.
The fixture test spells the new shape with a capital, which is precisely the
half a literal could not have caught.
Refs: CLOUD-73Pull request ( commit 4ec18ac76112329a4b9408ef858c01f6ab5ac073 (pull_request/wenzowski/cloud-172-record-battens-invocation-cost-as-a-series-so-hook-latency)
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 21:35:17 2026 +0000
docs(fanout): record that a bundle serializes on land, inside one session
The fan-out memory reasoned about land contention BETWEEN sessions and
said nothing about within one. A session handed a bundle meets the same
constraint as strict serialization for a mechanical reason: `land`
drives the working tree, so ticket N+1 cannot start while N's land is in
flight.
Measured on this bundle — each land ran 5+ laps at roughly 5-7 minutes,
so wall clock was dominated by laps the session could only wait out. The
unblock is a worktree per ticket, and `batten worktree` already models
the question.
Refs: CLOUD-172Can't fast forward * 4ec18ac76112329a4b9408ef858c01f6ab5ac073 docs(fanout): record that a bundle serializes on land, inside one session
* d1b933c3c59db10d7c278e4e72cf0cb443b80bcf feat(perf): record the invocation cost as a series, and gate the regression
| * eee9bb4bade0799923aff7d983911a169743d839 feat(config): gate the predecessor consumer's repo name
|/
* 2d3acc3416b5d9ed3ca9ec5c3280194b11ef757f ci(graph-check): make a frontier omission attributable, and say "could not look"
commit 2d3acc3416b5d9ed3ca9ec5c3280194b11ef757f
Author: Claude <noreply@anthropic.com>
Date: Tue Aug 11 18:52:11 2026 +0000
ci(graph-check): make a frontier omission attributable, and say "could not look"
`graph-check` is the scheduler as well as the gate, and its header claims every
agent computing the frontier independently gets the same answer. It did not:
`:92` ran ready-lint as `| "$lint" >/dev/null 2>&1 || continue`, so exit 1 ("this
Ready block is wrong") and exit 2 ("stdin is not a payload with a .description
field") arrived as the same silent absence — one missing `frontier` line and
nothing else, exit 0 both times over a 25-payload closure.
The same shape reached the verdict: `:40` required only id and status, so a set
whose relations the caller projected away was reported an acyclic, non-dangling
board with no edges in it at all.
Three channels now, not one. `report()` keeps exit 1 for a board signalling
falsely; `unjudged()` takes exit 2 for anything the gate could not read; `note()`
attributes an honest exclusion without moving the exit code, because a board full
of unrefined issues is not an incoherent board. ready-lint's rule ids are
forwarded rather than re-derived, and its ::error:: summary dropped — annotating
a failure at an exit code that denies one is the false signal this repo catches.
The missing-blockedBy report is keyed to the `graph` pseudo-id, not to each
issue: `released`'s `refusal_for` greps this stderr for `^<id> <rule>`, and a
per-id line would turn every In Review issue in a relations-free sweep into a
REFUSED. An explicit `blockedBy: []` is data and is judged; only an absent key
is unjudgeable.
Exit 2 outranks exit 1 when both hold, with both report sets printed: a verdict
over a set the gate could only partly read is not a verdict, and the caller's
next action is a re-fetch.
Refs: CLOUD-251Rebase locally, and then force push to |
4ec18ac to
553c89c
Compare
|
/fast-forward |
dd2dd08 to
7b75370
Compare
7b75370 to
edfa3bd
Compare
|
❌ The last analysis has failed. |
edfa3bd to
351ec6f
Compare
83a8115 to
e8fad29
Compare
The `forbid` kind bans literals named in advance, and a credential is the banned
shape nobody can enumerate -- so a committed secret passed every gate this engine
had. Detection is adopted prior art (a pinned scanner, run as a child process);
what this adds is CONTAINMENT, because the scanner prints the byte it matched and
Batten copies what it sees into channels that retain it.
Not a `command` rule pointed at the same binary, and the difference is the whole
kind: a command rule's contract is the exit code alone with both child streams
nulled, so it yields one batch verdict per glob -- no per-secret path:line, no
per-secret identity, nothing to key. Reading the scanner's output is exactly what
makes this kind expressible and exactly what makes it dangerous.
Two controls, neither of which anyone has to remember. Every match is wrapped
into `identity::SecretSpan` at the parse boundary, and that type has no route
back to `&str`, so the unkeyed `code_fingerprint` cannot be handed one -- a
compile error rather than a recall obligation. And `Finding` has no field a span
could occupy, so pointer-only output is structural in text and `-J` alike rather
than a property of the renderer. `settling_check` returns `Reevaluate` and
deliberately not an `Argv`: an argv here would put the scanner's invocation on
the record, and re-running the scan is the honest answer anyway.
THE PARSER ANCHORS ON THE PATHS IT PASSED IN, never on separator position.
`path:line:span` reads like a `split(':')` problem and is not, twice over: a path
may contain a colon and so may a secret (`user:password@host` is a credential
shape the scanner matches). Splitting there does not fail -- it silently keys a
TRUNCATED span, a wrong answer wearing a right answer's clothes that no
colon-free fixture would catch. Longest-match, because `a.rs` is a prefix of
`a.rs.bak` and a first-match parser would credit the wrong file. A line naming a
path the batch never passed is a parse failure, not a guess.
`--only-matching` is pinned beside the parser rather than offered as an option:
it is what makes the third field the matched literal alone. The span has to be
the literal because `secret_code_fingerprint` forces `Verbatim` normalization,
and a whole-line span would fold every unrelated edit on that line into the
secret's identity.
Fail-closed, all exit 3, all pointer-only -- an unparseable line names its
ordinal and never its bytes, which matters most here because the line that failed
to parse is a line the scanner emitted BECAUSE it found a secret in it. The
exit-status/parse-count cross-check refuses both directions: a clean exit with
matches parsed would report secrets the tool says it did not find, and a found
exit with nothing parsed would report a clean tree the tool says is not clean --
the second being the silent false green the clause exists to prevent. Any other
code, and any signal, is neither verdict.
A missing scanner is exit 1 naming `batten provision apply`, never a silent pass:
a scanner that did not run is not evidence of a clean tree. The provision entry
name is a constant rather than a rule column -- the kind permits no new column,
and a `tool = "..."` key would let a config point the secret kind at an arbitrary
binary and read its output as secret spans, which is a wider capability than
"scan for credentials" and a worse one to hand a config file.
The provision manifest is threaded through the runner because the adapter has to
resolve a pinned binary and `run_rule` could not see one. `run_static` takes the
slice and ignores it: it refuses every spawning kind before any work, so the one
kind that would read it never reaches the dispatch.
THE CACHE ROOT IS CANONICALIZED, NOT RESOLVED THROUGH GIT, and the difference is
a hermeticity bug caught by the corpus fixture. The engine's anchor is a RELATIVE
path (`.`) whenever the config sits in the cwd, and `state::derive_repo_name`
cannot name `.` -- so the first version failed with "cannot derive a repository
name" instead of the missing-scanner refusal. Reaching for `git::repo_root`
instead would have been worse than wrong: fixtures materialize under `target/`,
inside THIS repository, so a fixture asserting "no scanner installed" would have
resolved to the real cache and started passing or failing on whatever the
developer happened to have warmed.
The corpus fixture is the uninstalled case on purpose. The happy path needs a
stub binary in a cache the corpus harness cannot populate -- it materializes a
tree and runs argv, nothing more -- and the cold-cache posture is the one that
matters most on a fresh machine anyway: a scanner that did not run must never
read as a clean tree. The happy path lands beside it in a suite that stands up
its own XDG_DATA_HOME.
BREAKING CHANGE: `rules::run_static` and `rules::run_all` take the provision
manifest as a second parameter, surfaced by `mise run semver` as
`function_parameter_count_changed`. The adapter has to resolve a PINNED binary and
`run_rule` could not see one, so the manifest had to reach it; the alternative was
a `tool = "..."` rule column, which would let any config point the secrets kind at
an arbitrary binary and have its stdout parsed as secret spans -- a materially
wider capability than the kind is for. `run_static` takes the slice and ignores
it, since it refuses every spawning kind before any work and the one kind that
reads it therefore never reaches the dispatch.
Refs: CLOUD-59
…ed binary The claim this kind rests on is a NEGATIVE -- that no surface Batten writes can carry a matched byte -- and a negative needs a witness that would be found if it were false. Every case plants a synthetic token and then searches for it in each place the run could have put it: stdout, stderr, the `-J` document, and every file under the state root, which is where the key and the provision cache live. THE SUITE'S FIRST RUN FAILED ON ITS OWN FIXTURE, and the fix is the interesting part. Five cases reported "the token reached a file under the state root" -- correctly: `provision` caches the exact artifact bytes by contract, and the stub scanner carried the token as a shell literal, so installing it wrote the token under the state root. The tempting repair was to exclude the provision cache from the search. That would have disabled the assertion in the one place it matters most: the state root is where Batten writes files nobody reads, so a leak there is invisible indefinitely, and narrowing the sweep to make a test pass removes exactly the coverage the test exists for. The stub now reconstructs the token from shell fragments instead, so no artifact holds the contiguous string and the sweep stays whole. The planted tokens are assembled from fragments at runtime for the same family of reason: consumer #1's `no-secrets` rule globs the tree, so a committed literal credential in this file would be a standing violation of the rule this suite exists to prove -- the repository failing its own gate, forever, on its own test fixture. The scanner is a stub provisioned through `file://`, because what is under test is the ADAPTER rather than ripsecrets' detection. It emits the output shape measured against 0.1.11, which is what makes the disagreement branches drivable at all: a real scanner cannot be asked to contradict its own exit status. Detection is adopted prior art and consumer #1's own gate exercises it once the pin lands. Proved rather than asserted: exit 2 with a pointer-only stdout and the token absent from all four surfaces; the emitted identity secret-class (the version tag is what shows the KEYED path was taken) and unequal to `code_fingerprint` of the same span; two repositories minting two keys and two identities for one secret, which is the machine-scoped trade observed rather than described; byte-identical output across two runs, ordered by path rather than by emission order; a cold cache as exit 1 naming `batten provision apply`; and all four fail-closed branches as exit 3. Two cases exist to keep the rest honest. A clean scan still exits 0 with no output -- without it every assertion above would pass on an engine that always errored. And a glob matching nothing is driven with a stub that would FAIL the run if executed, so "no match means no spawn" is proved by the run succeeding rather than by trusting the code path. Refs: CLOUD-59
`clippy::format_collect` over `shell_token`. A plain loop pushing the quotes and the fragment says the same thing without allocating per part, and the helper's whole job is to keep the contiguous token out of the artifact -- which a loop does as well as a chain. Also records the semver lesson this bundle measured twice, in `mem:toolchain-and-hooks`: `semver` is verify-time, so a break declared late has to be walked back onto the commit that MAKES it, and declaring it on whichever commit happens to be HEAD satisfies the gate while leaving a `test(...)!` in history claiming an API break it did not make. Both instances were authored by someone who knew the change was breaking -- once with the reason written in the commit body -- so knowing and marking are independent acts, and the second is the one that gets skipped. Recorded rather than mechanised, with the reason: a commit-time semver check needs a baseline rustdoc build per commit, which is the cost `verify` exists to amortise. Refs: CLOUD-59
e8fad29 to
146ddc1
Compare
…ression CLOUD-207 published one number and holds it to an absolute ceiling. A number measured once still decays invisibly, and a ceiling cannot see a branch that got slower while staying under it. `bench-record` appends each trunk measurement to git notes on `refs/notes/bench`, keyed to the commit, and refuses to run off main — a branch's numbers are not the trunk's, and a series mixing them cannot be read. `bench-pair` builds this branch's binary AND its merge base's and measures them back to back on one machine; `bench-compare` decides that ratio; `bench-gate` composes the two. WALL CLOCK, PAIRED. The issue specified instruction counts with wall clock as a fallback "where valgrind is unavailable" — `mise registry valgrind` reports "tool not found in registry", and no-source-built-tool forbids compiling one, so the fallback is the only branch. What makes it usable is not a better clock but a better experiment: both arms on the same runner seconds apart, so machine noise is common-mode and divides out. The verdict is a ratio, never an absolute, and every series entry carries `metric=wall-clock` so a later instruction-count series cannot be diffed against it as a regression. THE THRESHOLD IS MEASURED. A null comparison — identical binary as both arms, n=30 — spread 0.966 to 1.102. 1.30 clears that measured maximum; `bench-pair --null` re-measures it and a bats case fails if the constant is ever tightened below the recorded floor. IN VERIFY, NOT AS A CI JOB. `ready-guard` refuses `gh pr ready` without a verify receipt for this exact HEAD, so a regression stops the PR before it can spend a matrix — earlier and cheaper than a check that reds after the runner is paid for. It costs the common change nothing: `bench-gate` exits clean without building when the diff against the merge base touches no crate source, manifest or lockfile. Recording runs on the existing daily schedule rather than a push-to-main trigger, which AGENTS.md forbids and whose reason holds here. Refs: CLOUD-172
…heckout The base arm is a binary built from the merge base, and both arms ran in the repo root — so the older binary was handed HEAD's committed batten.toml. A head that adds a config key the base binary does not know makes that binary exit 1 at load, hyperfine abort on its first warmup, and the whole gate answer 2. Measured: a head carrying a `[worktree]` table produced "unknown field `worktree`" from the base arm and took bench-gate down — a could-not-look manufactured by the gate's own setup, on exactly the class of change the gate exists to judge. Every arm now runs in the materialised fixture, whose config is pinned and loadable by both binaries. That also makes the number reproducible: measuring against this repository's live config moved the baseline whenever the config changed. tests/bench-pair.bats pins the fix as a static property, because reproducing it needs two real binaries built an hour apart. Refs: CLOUD-172
…gate it in CI Two corrections to what this branch shipped, both against CLOUD-172's own Mechanism clause rather than against taste. THE NAMESPACE. The issue specifies `perf`, `perf:record` and `perf:compare`, and this branch's own PR body claimed the rename had happened. It had not: the tasks landed as `bench-compare`, `bench-gate`, `bench-pair` and `bench-record` beside CLOUD-207's `bench` and `bench-assert`, and the workflow as `bench.yml`. The whole latency family moves here — six tasks, four suites, the workflow, the job, the concurrency group and `refs/notes/bench` -> `refs/notes/perf`. No notes ref exists on the remote yet, so the series orphans nothing. Hyphens, not the colons the issue writes: `mise-tasks/` holds file tasks, where a colon is a path separator. `perf-record` is the same name in the convention this repository actually uses. `token-bench`, `token-bench-check` and `bench/tokens/**` are deliberately NOT renamed. They are CLOUD-119's token-economics benchmark, a different subject with a name of its own, and freeing the bare `bench` for it is the reason the issue gives for this move. THE GATE RAN NOWHERE THAT DECIDES A PULL REQUEST. `perf-gate` was a step in `[tasks.verify]` only. `[tasks.ci]` is `depends = ["hooks", "deny"]`, so no runner ever invoked it: the Done clause "a PR that regresses past the threshold fails a check that names the regressed measurement" was unmet, because `verify` is what an author runs locally and a regression would have reached `main` past a green required check. A draft-gated `perf` job now runs it, seated in `final`'s `needs:` and named in `CI_REQUIRED_CHECKS`, with `fetch-depth: 0` because `perf-pair` builds both arms and a shallow clone has no merge base. It stays affordable for the reason the gate's own header argues: no crate source, manifest or lockfile in the diff means no build, and most of this repository's traffic pays a `git diff --name-only`. The recording half stays on the schedule — `perf-record` refuses to run off `main`, and a push-to-`main` trigger is forbidden. AND THE ROSTER HAS A SECOND READER THAT HAND-COPIED IT. `tests/ci-wait.bats` reads the real $CI_REQUIRED_CHECKS deliberately, but built its green fixture from a hand-written list of the names -- so adding `perf` left the fixture answering green for every name it knew while `ci-wait` waited for the one it did not. The suite HUNG rather than failed, twice, for the better part of an hour each time: a poll with no answer is precisely what that task is built to do, and it cannot tell "the roster grew" from "the run is still going". `green_body` now derives its rows from the roster minus the absent-ok set, with an anti-vacuity guard, so the copy that drifted no longer exists. BREAKING CHANGE: `mise run bench`, `bench-assert`, `bench-compare`, `bench-gate`, `bench-pair` and `bench-record` no longer exist; every one is `perf`-prefixed. The series ref moves from `refs/notes/bench` to `refs/notes/perf`. Task names are not a Rust API, so release-plz reads this as a patch either way, but a caller with the old name in a script gets "task not found" and the honest type is what puts that in the changelog. Refs: CLOUD-172
`timeout-check` takes two forms and prose is neither: `grandfathered` for a value that predates measurement, or `p95=<n>s x3` where the declared minutes equal ceil(p95 x 3 / 60). A new job cannot honestly claim the first — nothing here predates anything — so it gets the second, from a measurement rather than a guess. `mise run perf-gate` was forced down its full path, in a worktree carrying a crate-source change so the skip could not fire: both release binaries built, both arms sampled, 109s wall on a warm cache. The declared 200s covers that plus the job's own preamble — checkout, mise install, cache restore — and headroom for a colder runner, which lands the budget at 10 minutes. Stated at the line rather than implied, because one sample is not a p95 and the next reader deserves to know which it is. Re-measure once the job has a run history. Refs: CLOUD-172
…econd copy `mandatory_green` wrote out the roster by name, under a header already calling it "$CI_REQUIRED_CHECKS minus $CI_ABSENT_OK_CHECKS" — a second copy of a list the suite deliberately reads from mise.toml so it could never disagree with the one landing uses. Adding `perf` made it disagree: six cases failed at once, each on a branch they were not written to exercise, because a reading missing a required name answers "no run at all" and exit 3 before it reaches the bucket under test. The same hand-listing in tests/ci-wait.bats did worse and is fixed in the commit that introduced `perf`: there the missing name HUNG the suite rather than failing it, twice, because polling until every required check is green cannot tell a roster that grew from a run still going. Derived, both fixtures are correct for whatever the roster says today, with an anti-vacuity guard so an empty derivation cannot make every composing case pass while asserting nothing. One expectation stays literal and gains `perf` by hand: the partial-set case asserts the whole "with no run at all: …" list, whose subject is the ORDER and the elision of tolerated names. Deriving that one would compare the roster with itself and assert nothing about the message. It fails with a readable diff when the roster moves, which is the sensor working — the failure mode a silently short fixture does not have. Refs: CLOUD-172
… did not The rule claimed "in `verify`, not as a CI job" and named the reason — a regression caught before the PR spends a matrix. That half is still true and still the point. What was wrong is the exclusion: `perf-gate` is now also the `perf` job in ci.yml, because `verify` is what an author runs and CLOUD-172's Done clause asks for a CHECK. `[tasks.ci]` is `depends = ["hooks", "deny"]`, so the gate ran on no runner at all for its whole life. `rules-drift` exists for exactly this shape: a claim that is present and WRONG, carrying the rule's authority to a reader who then acts without re-deriving it. Landing the job without this edit would have left one. Refs: CLOUD-172
146ddc1 to
ad982c3
Compare
|
|
/fast-forward |



CLOUD-207 published one number. A number measured once decays in silence: nobody notices 3 ms becoming 40 ms, they notice months later that the hook got uninstalled. This turns the snapshot into a series attributable to a SHA, and puts a comparison gate on the pull request that caused a regression — the half that reaches someone who can still cheaply act.
The tasks
perf-record— appends each trunk measurement to git notes onrefs/notes/perf, keyed to the commit. No external service, and a regression stays attributable without re-running history. Refuses to run offmain: a branch's numbers are not the trunk's, and a series mixing them cannot be read.perf-pair— builds the merge-base and HEAD binaries and measures them back to back on one machine.perf-compare— decides that ratio. A pure function of stdin, sotests/perf-compare.batsexercises every verdict on every commit with no build.perf-gate— composes the two, and is the one nameverifyand CI call, so a caller cannot wire in half of it.Three decisions, each measured rather than chosen
Wall clock, paired — instruction counts are not available. The issue specified callgrind with wall clock as a fallback "where valgrind is unavailable".
mise registry valgrindreports tool not found in registry, so it cannot be pinned, andbatten.toml'sno-source-built-toolrule forbids compiling one. The fallback is the only branch.What makes wall clock usable is not a better clock but a better experiment. A contended runner can double a 3 ms measurement for reasons no commit caused. What it cannot do is double one arm of a pair measured back to back on that same runner in that same second. Machine variance is common-mode across the pair and divides out. So the verdict is a ratio, never an absolute, and
perf-recordstampsmetric=wall-clockinto every entry so a later instruction-count series can never be diffed against this one and read as a step change.Threshold 1.30×, derived from a measured null. Identical binary as both arms, 100 runs after 10 warmups, ten repeats across three paths — 30 ratios of a comparison that by construction measures nothing. They spread 0.966 to 1.102. A 10% swing is what this experiment produces when nothing changed, so a threshold near 1.10 would be a coin flip. 1.30 clears that measured maximum with room for a busier runner.
mise run perf-pair --nullre-measures the floor, and a bats case fails if the constant is ever tightened below the recorded one.Recording trigger: the daily schedule, not a push-to-
mainworkflow. AGENTS.md forbids the latter and the reason holds rather than merely applying —mainadvances only by fast-forward to a SHA CI already judged, so a push trigger buys a runner per merge for a measurement a clock can take. The cost is resolution, and it is affordable because the series is not the attribution mechanism:perf-gatecatches a regression on the PR that caused it, at the commit that caused it.Two corrections to what this branch first shipped
Both are against CLOUD-172's own Mechanism clause rather than against taste, and both were found by re-reading the issue beside the diff instead of beside the PR body.
The namespace was never moved. The issue specifies
perf,perf:record,perf:compare, and an earlier version of this body claimed the rename had happened. It had not: the tasks landed asbench-*beside CLOUD-207'sbenchandbench-assert, and the workflow asbench.yml. The whole latency family moves now — six tasks, four suites, the workflow, the job, the concurrency group, andrefs/notes/bench→refs/notes/perf(no notes ref exists on the remote yet, so the series orphans nothing). Hyphens rather than the issue's colons, becausemise-tasks/holds file tasks where a colon is a path separator.token-bench,token-bench-checkandbench/tokens/**are deliberately untouched — they are CLOUD-119's token-economics benchmark, and freeing the barebenchfor it is the reason the issue gives for this move.The gate ran nowhere that decides a pull request. An earlier version of this body claimed a draft-gated
perfjob inci.yml, infinal'sneeds, and inCI_REQUIRED_CHECKS. None of the three existed.perf-gatewas a step in[tasks.verify]only, and[tasks.ci]isdepends = ["hooks", "deny"]— so no runner ever invoked it, and the Done clause "a PR that regresses past the threshold fails a check that names the regressed measurement" was unmet, becauseverifyis what an author runs locally. The job exists now, wired into all three, withfetch-depth: 0becauseperf-pairbuilds both arms and a shallow clone has no merge base.It stays affordable for the reason the gate's own header argues: no crate source, manifest or lockfile in the diff means no build, and most of this repository's traffic pays a
git diff --name-only.What the roster change then broke, which is the interesting part
Adding
perftoCI_REQUIRED_CHECKSfailed six cases intests/checks-green.batsand hungtests/ci-wait.bats— twice, for the better part of an hour each time.Both suites read the real roster from
mise.toml [env]on purpose, so they can never disagree with the one landing uses — and then both built their all-green fixture from a hand-written list of the names. A roster that grows leaves the fixture short, and short means a required name with no run: exit 3 inchecks-green, and inci-waita poll that cannot tell "the roster grew" from "the run is still going". The suite that tests the poller inherited the poller's blind spot.Both fixtures derive from the roster now, minus the absent-ok set, with an anti-vacuity guard so an empty derivation cannot make every composing case pass while asserting nothing. One expectation stays literal and gained
perfby hand — the partial-set case asserts the wholewith no run at all: …list, whose subject is the order and the elision, and deriving that one would compare the roster against itself.Two more things this turned up
fuzz/Cargo.lockwas four releases stale, so the first cargo invocation underfuzz/rewrote it andverify'stree-cleanstep failed on a file the run had just dirtied. (Resolved against trunk during the final rebase, which had refreshed it too.).claude/rules/toolchain.mdsaid the gate runs "inverify, not as a CI job". True when written, false the moment the job landed — andrules-driftexists for exactly that shape: a claim that is present and wrong, carrying the rule's authority to a reader who then acts without re-deriving it. Corrected in the same branch rather than left for the gate to catch.Timeout budget
timeout-checktakesgrandfatheredorp95=<n>s x3, and prose is neither. A new job cannot honestly claim the first, so the budget is measured:perf-gateforced down its full path in a worktree carrying a crate-source change — both release binaries built, both arms sampled — is 109s wall on a warm cache. The declared 200s covers that plus checkout, mise install and cache restore, with headroom for a colder runner; ×3 lands it at 10 minutes. One sample is not a p95 and the line says so; re-measure once the job has a history.Closes CLOUD-172
Refs: CLOUD-207, CLOUD-119, CLOUD-418