Skip to content

specs(tools/tri): real tests for 10 tool cards, misread to red (Closes #7951) - #7958

Open
gHashTag wants to merge 2 commits into
masterfrom
vac/tools-tri-2
Open

specs(tools/tri): real tests for 10 tool cards, misread to red (Closes #7951)#7958
gHashTag wants to merge 2 commits into
masterfrom
vac/tools-tri-2

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Closes #7951. Part of #6063 (t27b coverage), which is item C10 of #6488.

What changes

Ten more tool cards in specs/tools/tri had no test block (the first 11 got these tests in #7904). They passed vacuously (0 runtime asserts) under the reference and under t27b, and broke L4 TESTABILITY. Each now carries the four tests of #7904, which check the rules the card is written under. The rules are specs/tools/catalog.t27 (ID_RULE, SCHEMA_2_RULE) and the card's own header (the tri <name> command, read off a clap #[derive(Subcommand)] enum).

test what it checks
qualified_id_is_repo_colon_id QUALIFIED_ID is REPO, a :, then ID, byte by byte; SCHEMA is 2
id_command_and_variant_name_one_subcommand ID is tri/<name>, COMMAND is tri <name>, and VARIANT is Commands::<Name>, where <name> is the kebab-case of <Name> (OneAway is one-away)
clap_names_are_kebab_case the command name and every action are lower-case words joined by -; one ACTIONS_ABOUT per action; every ARGS entry is a <positional> or a --flag
the_queen_holds_every_tool_beside_its_lead AGENTS has the Queen (T) first, as docs/agents/AGENTS_ALPHABET.md binds every tool to her, then the domain lead(s): one letter each, none twice

No test restates a constant: each one relates two or more fields, or a field to the grammar it must follow. Each test walks strings with a var index, so its asserts are runtime asserts.

Specs changed (all specs/tools/tri/<name>.t27), with the runtime asserts their 4 tests execute (reference / t27b):

spec reference t27b
misread 4/4 4/4, 70 asserts
mods 4/4 4/4, 63 asserts
mutate 4/4 4/4, 80 asserts
now 4/4 4/4, 62 asserts
one-away 4/4 4/4, 75 asserts
orphaned 4/4 4/4, 81 asserts
pr 4/4 4/4, 65 asserts
prose 4/4 4/4, 67 asserts
quantifiers 4/4 4/4, 101 asserts
red 4/4 4/4, 60 asserts
  • Reference: t27c test-report <spec> on the t27c lab (zig 0.16.0): 4 pass, 0 fail, 0 vacuous for each.
  • t27b: t27b test <spec> --check on the t27b lab (aarch64 under qemu).
  • Seals: t27c seal --save, then t27c seal --verify: all hashes MATCH. The 10 seal files are new; these cards had none.
  • No existing line of any spec changed; the tests are appended.

Mutants

13 hand mutants per card, each a plausible slip in the data. Each was applied to a copy of the card and run under t27c test-report:

  • QUALIFIED_ID names the other repository, or its name drifts from ID;
  • COMMAND joined by -, or its name drifts;
  • VARIANT with the enum renamed, the name drifted, or written in lower case;
  • ID loses its /;
  • AGENTS with the lead before the Queen, or the Queen twice;
  • the first action not kebab-case;
  • the first arg neither <...> nor --...;
  • SCHEMA 1.

Over the 31 cards of specs/tools/tri, 359 mutants applied and 359 were killed; the 10 cards here are among them. The other 31 did not apply: an action or arg mutant on a card with no actions or no args. None survived.

Skipped

None in this directory: all 31 cards of specs/tools/tri get these tests, in #7904 (gen to merging), this PR and one more (reseal to worktrees). The specs this lane leaves alone elsewhere (module-path fixtures such as specs/a/b_c.t27, and empty placeholder modules) are named in the PRs of their directories.

Gates

On the t27c lab, from master's gen/c/policy/own_language.c, with lefthook.yml's one-line C main:

  • check_all() over origin/master:tools/policy/foreign-exceptions.txt, -- and git diff --name-status origin/master...HEAD: exit 0. A planted A cli/foo/new.rs line exits 1.
  • check_budget() over git diff --numstat --no-renames origin/master...HEAD: exit 0. No foreign line is added.

Ledger

docs/reports/t27b_expectations.json: the 10 rows move from pass_vacuous to pass; nothing else changes. The rows come from bless in scripts/tri_loop/t27b.py (what tri t27b ratchet --bless runs) over lab run 967bb7b with this branch's t27b verdicts for the 10 specs, composed onto master's ledger with only these rows. Master's ledger has since moved and now stores no counts (they are derived); master was merged in from master's side (9f0202a) and the 10 rows recomposed onto it from this branch's own lab run 8d8842b: derived pass 962 -> 972, pass_vacuous 141 -> 131, max_not_pass unchanged.

tri t27b ratchet --run of this branch's lab run has no finding on any of the 10 specs against the branch ledger.

t27b lab: signed receipts (#7686)

Head 8d8842b (this branch before the merge of master), requested with a fresh 32-byte challenge, against base 6af01bd. The base is the newest master ancestor the lab has a receipt for; the merge-base 4daabae (#7904) and its parent have none. Both are signed runs of the deployed lab.

/work/t27c-master corpus-receipt compare BASE HEAD --challenge-head <mine>, from /work/t27 on the t27b lab (the input/output/verdict changed lines left out):

base "6af01bd4cf3dfcdecb1622512b7698b38b642a05" AUTH_MISSING_NONE AUTHOR leaves-bound true
head "8d8842b820e3670056143b53920e8f90c641c711" AUTH_MISSING_NONE FRESH leaves-bound true
  lane neutral specs/automation/bot-neuro-office.t27
  lane improved specs/tools/tri/gen.t27          (and the other 10 cards of #7904)
  lane improved specs/tools/tri/misread.t27
  lane improved specs/tools/tri/mods.t27
  lane improved specs/tools/tri/mutate.t27
  lane improved specs/tools/tri/now.t27
  lane improved specs/tools/tri/one-away.t27
  lane improved specs/tools/tri/orphaned.t27
  lane improved specs/tools/tri/pr.t27
  lane improved specs/tools/tri/prose.t27
  lane improved specs/tools/tri/quantifiers.t27
  lane improved specs/tools/tri/red.t27
totals false inputs false verdicts false outputs false: IMPROVED_ONLY

Exit 3, IMPROVED_ONLY. Between the base and the merge-base sit #7904 (its 11 cards are the other improved rows) and a bot spec (neutral). The PR head 9f0202a is 8d8842b with master merged in and the ledger rows recomposed; it changes no spec beyond master's.

base 6af01bd head 8d8842b
t27b pass / pass_vacuous (where the reference passes) 1004 / 152 1025 / 131 (11 of the +21 are #7904's)
reference pass 1189 1189
reference disagree (files / tests) 0 / 0 0 / 0
jit_interp_mismatch / crash 0 / 0 0 / 0

The 10 specs here, in the head run: pass, 4 tests each, 60 to 101 runtime asserts, reference pass, no disagreement. 10 specs move from pass_vacuous to pass; none regresses.

Generated with Claude Code

@claude
specs(tools/tri): real tests for 10 tool cards, misread to red (Closes …
8d8842b 
…#7951)

Each card held no test block and passed vacuously (0 runtime asserts) under
both the reference and t27b. The same four tests as #7904 check the rules
the card is written under, byte by byte:

- QUALIFIED_ID is REPO, ':' and ID (specs/tools/catalog.t27 ID_RULE);
- ID is tri/<name>, COMMAND is `tri <name>`, VARIANT is Commands::<Name>
  with <name> its kebab-case;
- the command name and every action are kebab-case, one ACTIONS_ABOUT per
  action, every ARGS entry a <positional> or a --flag;
- AGENTS holds the Queen (T) first, then the domain lead(s), no letter twice.

t27c test-report: 4/4 per spec, 0 vacuous. t27b test --check: 4/4, 60 to
101 runtime asserts. Sealed with seal --save and checked with --verify.

docs/reports/t27b_expectations.json: `bless` of scripts/tri_loop/t27b.py
over lab run 967bb7b with this branch's t27b verdicts for the 10 specs,
composed onto master's ledger with only these rows: pass_vacuous -> pass.
pass 942 -> 952, pass_vacuous 140 -> 130.

Closes #7951

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
Open
Open
Open
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-08 22:41:22 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 41
PRs with All Checks Green 9
READY 3
FAILING 41
PENDING 0
NO CHECKS YET 0

These columns do not partition: 3 + 41 + 0 + 0 = 44, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=3c0ade9e73e4 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 8, 2026
Merged
Merged
Merged
Merged
Merged
Open
Open
Open
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Open
Merged
Merged
Merged
Merged
Open
Merged
Open
Merged
Merged
This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Open
Open
Merged
Merged
@claude
Merge master into vac/tools-tri-2; recompose the 10 ledger rows (Closes
9f0202a 
#7951)

docs/reports/t27b_expectations.json conflicted with master's ledger
(7a07828, which now stores no counts: they are derived). Resolved from
master's side: master's ledger with only this branch's 10 rows recomposed by
`bless` over this branch's lab run 8d8842b (pass_vacuous -> pass).

Closes #7951

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag enabled auto-merge (squash) October 9, 2026 06:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

t27b coverage: real tests for 10 tool cards in specs/tools/tri (misread to red)

2 participants

@gHashTag @claude