Skip to content

feat(gate): discover a fixture-repo corpus, and collapse nine copies of the materializer - #191

Merged
wenzowski merged 1 commit into
mainfrom
wenzowski/cloud-63-build-a-fixture-repo-test-harness
Aug 9, 2026
Merged

wenzowski merged 1 commit into
mainfrom
wenzowski/cloud-63-build-a-fixture-repo-test-harness

Conversation

@wenzowski

Copy link
Copy Markdown
Contributor

Lands CLOUD-63.

  • The corpus (crates/batten/tests/fixtures/repos/<name>/) is committed data: config, payload, and a hand-authored expected pinning argv, exit code and byte-exact stdout. No accept/bless mode — an expectation rewritten from observed output would agree with any behaviour at all.
  • The driver (tests/fixture_repos.rs) carries zero fixture facts. Because a discovery harness passes vacuously when it discovers nothing, the obligations are asserted: discovery is exact (a missing expected is an error, never a skip), the comparator is negatively self-tested in memory, every RuleKind::ALL variant is covered with the spawning kind driven by enforce, every fixture runs twice, and every committed corpus file is inert (.in).
  • The collapse. The materializer was re-typed once per integration target, and the copies had diverged on the two behaviours that decide hermeticity: cli.rs's builder did not clear its scratch directory, and the copied fn batten() scrubbed no environment at all. Both are unconditional in tests/common/mod.rs, and the scrub list is derived by walking SURFACE rather than copied — a flag that mints a new BATTEN_ variable is scrubbed the day it lands.
  • Policy-engine-first for the repo-agnosticism half: a forbid rule over crates/batten/tests/** in the committed batten.toml. "At least one fixture per rule" cannot be a rule — its domain is the engine's own RuleKind table, not the repository's files — so it lands as a #[test] over that table instead.

mise run verify green. 395 cargo tests, no warnings.

Refs: CLOUD-63


Generated by Claude Code

…of the materializer

The corpus is committed data under crates/batten/tests/fixtures/repos/, each
fixture carrying its config, its payload, and a hand-authored `expected` naming
the argv, the exit code and the byte-exact stdout. The driver
(tests/fixture_repos.rs) carries zero fixture facts and has no accept/bless
mode: an expectation rewritten from observed output would agree with any
behaviour at all.

Because a discovery harness passes vacuously when it discovers nothing, the
obligations are asserted rather than argued — discovery is exact rather than
merely non-empty (a missing `expected` is an error, never a skip), the
comparator is negatively self-tested in memory, every RuleKind::ALL variant is
covered with the spawning kind driven by `enforce`, and every fixture runs
twice for byte-stability. Corpus files are committed with a trailing `.in` so a
fixture may carry a shape this repo's own gates refuse.

The materializer was re-typed once per integration target and the copies had
already diverged on the two behaviours that decide hermeticity: cli.rs's builder
did not clear its scratch directory (312b320 was that drift being repaired one
file at a time), and the copied `fn batten()` scrubbed no environment at all.
Both are now unconditional in tests/common/mod.rs, and the scrub list is derived
by walking SURFACE rather than copied, so a flag that mints a new BATTEN_
variable is scrubbed the day it lands.

The repo-agnosticism half is policy-engine-first: a `forbid` rule over
crates/batten/tests/** in the committed batten.toml. "At least one fixture per
rule" cannot be a rule — its domain is the engine's own RuleKind table, not the
repository's files — so it lands as a #[test] over that table instead.

Refs: CLOUD-63
@linear-code

linear-code Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
CLOUD-63 Build a fixture-repo test harness

Why
Batten needs repo-agnostic fixture coverage instead of tests that implicitly depend on this repository.

Definition of done

  • Discover fixture repositories under the crate's tests/fixtures/ corpus root
  • Drive them through table tests with expected outputs
  • Avoid hardcoding any repo name

Acceptance

  • At least one fixture exists per rule
  • Tests discover fixtures rather than enumerate repo-specific ones
  • Test sources contain no repo-specific identifiers

Refinement — Ready (a discovered on-disk fixture corpus; the driver carries no fixture facts)

Refinement gate: Definition of Ready & Done. This body carries only specializations.

  • Source of truth (§1). The one authoritative artifact is the discovered corpus, rooted at crates/batten/tests/fixtures/repos/<name>/: each fixture directory carries its own config, its payload files, and a committed expected file naming the argv, the exit code, and the byte-exact stdout. The discovered root is a named subdirectory rather than tests/fixtures/ itself, so that enumeration and a suite-driven fixture tree can share that parent without either asserting over the other's contents — CLOUD-64's ruleset fixture at crates/batten/tests/fixtures/acceptance-corpus/ has its own named suite and no expected file, and sitting outside the discovered root it neither trips §7(a)'s exactness nor needs its path changed. The driver — a new crates/batten/tests/fixture_repos.rs — enumerates the discovered root and carries zero fixture facts: no fixture list, no expected bytes. Today the corpus is code — every fixture is an inline &str plus an assert_eq! — and the materializer is re-typed once per integration target: git grep -n 'fn batten()' -- crates/batten/tests/ enumerates the command builders, and git grep -nE 'fn (repo_with_config|repo|pr_fixture)\(' -- crates/batten/tests/ the repo builders, whose signatures have themselves diverged. Every file either grep names is in scope, because one surviving copy is a second authority — the collapse is total or it has not happened. Those copies have already drifted on the one behaviour deciding flakiness, clearing the scratch directory before writing it: git grep -n remove_dir_all -- crates/batten/tests/ partitions the builders that do from the ones that do not, and 312b320 test(fail-on-warning): start each fixture from an empty directory is that drift being repaired one file at a time, its message recording a suite turning red from a stray source file an earlier run left behind. So the materializer lands once at crates/batten/tests/common/mod.rs and every copy those greps name collapses onto it. Cargo compiles a tests/ subdirectory only when it holds a main.rs, so neither common/ nor fixtures/ becomes a test target. The consumer-ci: check in the main-branch protection ruleset #1 dogfood tests that read this repository's own committed configs — grep -n '\.\./\.\./batten' crates/batten/tests/cli.rs names every one of them, and it is not a fixed number — stay in tests/cli.rs: they are about this repo by design, which is exactly what excludes them from the corpus.
  • Computable predicate (§2). Two gates, both existing commands. (a) The corpus: mise run test:cargo — a fixture whose observed exit code or stdout differs from its committed expected fails the target. It reaches the hk gate through the test step and CI through mise run ci → hk check --all; that step is glob-scoped to **/*.rs, **/Cargo.toml, Cargo.lock, so its glob list must gain crates/batten/tests/fixtures/** or a fixture-only commit skips the suite at pre-commit while CI still runs it. (b) Policy-engine-first, for the repo-agnosticism half: one new [[rule]] in the committed batten.toml — kind = "forbid", glob = "crates/batten/tests/**", pattern = "button-inc", severity = "deny", scope = "tree" — evaluated by batten check and already gated by mise run batten-check, an hk step with no glob. The org literal rather than batten, because min_batten_version and batten.toml legitimately carry the crate name while nothing a fixture needs names this repository's origin; it lands green, since button-inc occurs in the files git grep -l button-inc names — the workspace manifest, the toolchain and taplo config, a release workflow, the shell-gate suites, and the release-plz-generated crates/batten/CHANGELOG.md — but nowhere under crates/batten/tests/ (git grep -n button-inc -- crates/batten/tests/, no match), which is the half that decides greenness. The enumeration is the command's, not a list re-typed here: an earlier draft of this bullet named six files and was already stale by one (tests/claim-check.bats landed hours before it was written), and nowhere under crates/batten/tests/. What the engine cannot express, named rather than quietly routed to bash: "at least one fixture per rule" is a required-presence predicate whose domain is the engine's own RuleKind::ALL table in crates/batten/src/rules.rs, not the repository's files, and forbid — the only non-spawning kind — reports lines matching a literal and cannot assert that something exists. It therefore lands as a #[test] iterating batten::rules::RuleKind::ALL (pub, so the integration target reads it), the same discipline rules.rs's own all_covers_every_kind uses to keep that table total. No capability-gap issue is owed: a rule kind reading the crate's enum would not be a predicate over a repository at all.
  • Effect (§3). No new verb, so no new row in the effect table. The corpus must record the verb per fixture instead of assuming one, because the §5 split is load-bearing: a command-kind fixture runs under enforce (unclassified), and check (read) refuses it with exit 1 rather than skipping it. expected therefore pins the argv, not only the outcome.
  • Generated artifacts (§4). Nothing here is derived: the expected files are hand-authored specification and the driver has no accept/bless mode. A harness that rewrote expected from observed output would gate nothing. Re-deriving that comparison through insta is CLOUD-106's deliberate change, and it swaps the comparison mechanism without touching this layout.
  • Output & exit (§5). Each fixture pins one code from the 0/1/2/3 table plus byte-exact stdout, which is pointer-only (path:line rule-id). Findings print repo-relative, /-separated paths, so the temp directory a fixture is materialized into never appears on stdout and the expectations need no scrubbing to stay byte-stable across runs and platforms. The corpus covers the codes reachable today — 0 clean; 0 with a printed finding (a warn rule, fail_on_warning absent); 1 for a rule omitting severity; 2 for a deny finding — and 3 has no command that reaches it, so its numeric contract stays in the exit unit tests. Where the payload is a diagnostic rather than a finding, expected pins stderr too: exit 1 prints nothing on stdout, so a stdout-only expectation would pin empty bytes and could not regress the named-key refusal that a_rule_omitting_severity_is_refused_with_a_named_key in tests/cli.rs asserts on stderr. Substrings there, not byte-equality — the parse error interpolates its source, which for a file is the config's own path (crates/batten/src/config.rs), i.e. the per-run temp directory the fixture materialized into. Every run also scrubs BATTEN_* from the environment and fences discovery with GIT_CEILING_DIRECTORIES, so a developer's shell cannot move a verdict. That is a widening rather than an established precedent: git grep -n GIT_CEILING_DIRECTORIES -- crates/batten/tests/ names the per-suite command builders that fence it today, and the copied fn batten() of §1 scrubs nothing at all, so the collapsed materializer is where both become unconditional for every case.
  • Commit / bump (§6). feat → patch until 0.1.0. Below 0.1.0 release-plz bumps the patch whatever the type says; the regime is read out of [workspace.package] version in Cargo.toml by mise-tasks/ready-lint, so it is never restated here. feat(gate) matches the house precedent for a change that adds a live rule to the committed config.
  • Test obligation (§7). Self-referential, so what must be proved is that the harness cannot pass vacuously — a driver discovering zero fixtures, or silently skipping a malformed one, would be green. E2E over the compiled binary (env!("CARGO_BIN_EXE_batten")), in crates/batten/tests/fixture_repos.rs: (a) discovery is non-empty and exact — the count equals the directories directly under the discovered root of §1, so a fixture missing its expected file is an error, never a skip; (b) a negative self-test — the comparison, handed a deliberately wrong expectation in memory, reports a mismatch, so no red fixture is ever committed to prove the point; (c) one fixture per RuleKind::ALL variant, the command-kind one under enforce; (d) every fixture run twice with identical stdout, generalizing cli.rs's single byte-stability case to the whole corpus; (e) inertness — each corpus file is committed with a trailing .in that materialization strips, so a fixture carrying a banned shape cannot trip this repo's own gates over the same tree: the committed no-conflict-markers rule globs crates/**/*.rs, taplo lints and formats the tracked TOML (no glob override in hk.pkl), and prettier covers **/*.md. tests/cli.rs already assembles its conflict marker at runtime for that reason; the .in suffix makes the dodge uniform, and a green mise run ci over a corpus holding both a marker fixture and an invalid-config fixture is the proof.
  • Blockers (§8). None. Every capability the corpus exercises is in the tree today: both rule kinds, the severity model, fail_on_warning, and the config loader. relatedTo CLOUD-106 (the snapshot mechanism that later replaces the byte comparison, corpus layout unchanged), CLOUD-217 (its item 43 records the stale-temp-dir drift the copies carry; the §1 collapse removes it at the source), CLOUD-169 (a later corpus reusing this layout), and CLOUD-40 (fixtures for hook are stdin payloads — a different matrix from a repo directory), and CLOUD-64, whose ruleset fixture shares the tests/fixtures/ parent — §1 fixes the boundary so that neither suite enumerates the other's tree, and neither issue waits on the other.

Review in Linear

@wenzowski
wenzowski marked this pull request as ready for review August 9, 2026 16:14
@wenzowski

Copy link
Copy Markdown
Contributor Author

/fast-forward

@wenzowski
wenzowski merged commit 4bdc023 into main Aug 9, 2026
10 checks passed
@wenzowski
wenzowski deleted the wenzowski/cloud-63-build-a-fixture-repo-test-harness branch August 9, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants