Skip to content

t27c: test-report exits non-zero on FAIL or BLOCKED (Closes #7370) - #8152

Merged
gHashTag merged 6 commits into
masterfrom
claude/test-report-exit-7370
Oct 9, 2026
Merged

t27c: test-report exits non-zero on FAIL or BLOCKED (Closes #7370)#8152
gHashTag merged 6 commits into
masterfrom
claude/test-report-exit-7370

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Pull Request Checklist

  • PR body includes Closes #7370
  • Tests added/updated: cargo test --release -p t27c (3000 pass, 0 fail), cargo test --release -p t27b (134 pass)
  • Specs changed -> seal saved: t27c seal specs/compiler/test_report_exit.t27 --save (7/7 tests; --verify: all hashes MATCH; no twins)

Description

t27c test-report <spec> exited 0 even when a test FAILed or the report was BLOCKED (every path of run_test_report returned Ok). Now, once the report is printed:

report exit
BLOCKED (does not build, or no zig) 2
at least one test FAILs (a timed-out test is a FAIL) 1
otherwise, including a spec with no test block 0

stdout is byte-for-byte the same. --all (the census of the whole tree) keeps exit 0. An error that prints no report still exits 1 (not a file) or 2 (usage error); neither prints the --- test report: line.

Spec first: the rule is specs/compiler/test_report_exit.t27 (7 tests, 1 invariant, sealed). t27c runs it through t27c gen-rust (bootstrap/gen/rust/compiler/test_report_exit.rs, mounted with #[path] like the verified/* rules), and a test fails if the copy drifts from the spec. The wording follows the "the exit agrees" rule in #7400's specs/tri/test/report.t27; this PR does not depend on #7400, and #7400's verdict(text, exited_ok) already reads a red report with a non-zero exit as that red verdict.

Opt-out: T27C_TEST_REPORT_EXIT_ZERO=1 (exactly "1") gives the exit of before this change on every printed report, for callers that read the words and take any non-zero exit for a broken tool. It is an environment variable, not a flag, so t27b can ask once for every test-report it spawns, and a t27c from before this change ignores it.

Changes

  • specs/compiler/test_report_exit.t27 - the rule (new), .trinity/seals/compiler_CompilerTestReportExit.json - its seal
  • bootstrap/gen/rust/compiler/test_report_exit.rs - t27c gen-rust output (force-added under the gen/ ignore, like its siblings)
  • bootstrap/src/main.rs - run_test_report exits by the generated rule after printing; help text names the codes
  • bootstrap/tests/test_report_timeout.rs - new tests; the looping-test case now expects exit 1
  • callers, below

Every caller (grep for test-report over the whole repo)

caller reads before -> after this PR done
cli/t27b/src/blockers.rs reference runner (used by t27b corpus --reference, t27b-native.yml, scripts/tri_loop/t27b*.py) c.code == Some(0), else BLOCKED and cached would record a failing spec as blocked: t27c test-report exited Some(1) cli/t27b/src/main.rs sets the opt-out at start. blockers.rs is not on the exception list, so it is not edited; it could read the new codes directly once someone may edit it
contrib/railway/t27b-lab/lab.py reference_one returncode != 0 -> blocked same as above sets the opt-out in its env
tools/queen/criteria_backfill.py run_check (also tools/bees/reviewer.py measure_criteria) each t27c call in a criterion must also exit 0 on its own a t27c test-report X 2>&1 | grep -c BLOCKED prints 0 criterion would fail on a spec that builds but has a red test sets the opt-out: an error still fails, a red test does not
.github/workflows/spec-guards.yml affected-specs score() set -euo pipefail; h=$(score f) the step would stop at the first spec with a FAIL instead of comparing head with base T27C_TEST_REPORT_EXIT_ZERO=1 in score(); a t27c error still ends the step
.github/workflows/xilinx7-l3-bench.yml:59 t27c test-report ... | tee, then grep -q 'FAIL 0' default bash -e, no pipefail: the pipe's status is tee's unchanged
tools/queen/feed_untested.py, feed_empty_bodies.py, feed_roadmap.py, feed_defects.py bee_shell = bash -c, stdout only; criteria text | grep -c BLOCKED rc not read unchanged
tools/check_seal_coverage.py:332 advice t27c test-report <spec> && t27c seal <spec> --save the && now stops on a red spec, as the advice meant unchanged
bootstrap/src/main.rs seal test_report::run in-process not through the CLI exit unchanged; seal_refuses_failing_tests and seal_reseal_is_stable pass
bootstrap/tests/string_eq_zig.rs text only - unchanged, passes
bootstrap/tests/test_report_timeout.rs asserted success with a timed-out FAIL - now asserts exit 1
cli/tri on master does not call test-report (#7400 will, and handles both exits) - unchanged
scripts/tri_loop/toolbelt.py, t27b_reduce.py, docs/BEE_TOOLBELT.md, the providers-steward skill prose / comments - unchanged
docs/ROADMAP_BRIEF.md:208 told bees the exit is 0 either way stale prose updated

Outside this repo: 999-multibots-telegraf scripts/check-t27-specs.mjs uses execFileSync, which throws on a non-zero exit; its catch already reads e.stdout and judges the table, so it keeps working. Not changed here.

Files beyond the issue's Boundary: the spec, its generated copy and seal (spec first), bootstrap/tests/test_report_timeout.rs (tests), cli/t27b/src/main.rs and spec-guards.yml (callers the issue's list did not name), docs/ROADMAP_BRIEF.md (prose).

Testing

cargo build --release -p t27c
# before (master): red spec rc=0, BLOCKED spec rc=0; after:
t27c test-report red.t27      # rc=1   (tests 2, pass 1, FAIL 1)
t27c test-report blocked.t27  # rc=2   (BLOCKED does not compile)
t27c test-report specs/ternary/gft_relu.t27   # rc=0
T27C_TEST_REPORT_EXIT_ZERO=1 t27c test-report red.t27   # rc=0; =0 or unset: rc=1
t27c test-report /nonexistent.t27   # rc=1 with or without the opt-out
cargo test --release -p t27c --test test_report_timeout
  • New test the_exit_code_says_what_the_report_says fails on master's main.rs ([[0,0,0],[0,0,0],[0,0,0]] vs [[0,1,2],[0,1,2],[0,0,0]]) and passes here; with zig absent it expects 2 for every spec, and passes that way too. Negative controls: the opt-out row is 0 before and after, and a missing file is 1 under the opt-out. the_exit_rule_copy_is_what_gen_rust_writes guards the generated copy.
  • Mutation check of the spec's tests: 6 one-line mutants (failed > 1, BLOCKED judged after FAIL, opted_out without the length check, opt-out only when blocked, EXIT_BLOCKED = 1, the green return made EXIT_FAILED) - all 6 red.
  • cargo test --release -p t27c: 3000 pass, 0 fail (zig 0.16.0). cargo test --release -p t27b: 134 pass.
  • t27b end to end on {gft_relu, red, blocked} with this t27c: this branch's t27b gives reference pass / fail (1 of 2 tests fail) / blocked (does not compile ...); master's t27b gives pass / blocked (t27c test-report exited Some(1)) / blocked (exited Some(2)) - the defect the opt-out prevents.
  • run_check with this t27c: red spec grep -c BLOCKED -> passed (0), BLOCKED spec -> failed (1), missing file -> failed ("exits 1"); master's run_check with this t27c fails the red spec ("exits 1").
  • spec-guards score() under set -euo pipefail with this t27c: old form ends the step on the red (rc 1) and BLOCKED (rc 2) specs; new form scores 1 and 1000000 and still ends the step on a missing file.
  • scripts/ci/test_*t27b*.py: 5 of 6 pass; test_a_t27b_spec_cannot_move_silently.py fails one ledger check ("the cap equals reality (26, cap 27)") identically on master be97577, not touched here.
  • criteria_backfill.py --self-test, reviewer.py self-test: 0 failures. tools/ci/check_specs_still_parse.py: ok. tools/check_json_parses.py: OK. t27c seal ... --verify: all hashes MATCH.
  • own-language: path gate passes (every foreign file touched is on the exception list); budget: 61 added foreign lines (main.rs 19, test_report_timeout.rs 35, t27b main.rs 3, criteria_backfill.py 2, lab.py 1, spec-guards.yml 1), max 35 per file.

Review Notes

  • The opt-out exists because cli/t27b/src/blockers.rs cannot be edited without the owner's label. Reading the codes there (0/1/2 with the report line = a verdict) would let t27b drop the opt-out later.
  • Codes 1 and 2 are shared with a pre-report error and a clap usage error; the spec says so and names the --- test report: line as the way to tell them apart.

Closes #7370

🤖 Generated with Claude Code

https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k

Generated by Claude Code

@claude
t27c: test-report exits non-zero on FAIL or BLOCKED (Closes #7370)
7664944 
`t27c test-report <spec>` returned Ok on every path, so it exited 0 when
a test failed and when the spec was BLOCKED. The rule now lives in
specs/compiler/test_report_exit.t27 (7 tests, 1 invariant, sealed) and
reaches t27c through `t27c gen-rust` (bootstrap/gen/rust/compiler/
test_report_exit.rs): exit 2 when BLOCKED, 1 when a test FAILs, 0
otherwise. The report's text is unchanged. `--all` stays a census at 0.

T27C_TEST_REPORT_EXIT_ZERO=1 keeps the old exit for callers that read the
report's words and take any non-zero exit for a broken tool; an error
that prints no report still exits 1. Callers adapted in this commit:
- cli/t27b/src/main.rs sets it, so the reference runner in blockers.rs
  does not cache a failing spec as BLOCKED;
- contrib/railway/t27b-lab/lab.py sets it for its reference runs;
- tools/queen/criteria_backfill.py run_check sets it, so a
  `grep -c BLOCKED` criterion is not failed by a red test;
- spec-guards.yml score() sets it: under `set -euo pipefail` the
  assignment h=$(score ...) would end the step at the first red spec.

Tests: bootstrap/tests/test_report_timeout.rs checks 0/1/2, the opt-out,
an error under the opt-out, and that the generated copy matches its spec;
the looping-test case now expects exit 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Merged
@claude
ledger(t27b): test_report_exit.t27 passes natively -- one entry (Closes
598ed07 
#7370)

The native ratchet on this PR counted 1077 / 1236 over 1720 files,
one file and one pass above master in the same window, with
"reference disagree 0": the new spec runs and passes on t27b, and the
reference path (t27b's blockers.rs under T27C_TEST_REPORT_EXIT_ZERO=1)
still reads every report. Its row goes in as "pass" instead of
staying unlisted; counts are derived (#7862), no header moves.

scripts/ci/test_a_t27b_spec_cannot_move_silently.py fails the same
way with and without this row (on master too), so it is not this PR's.

phi^2 + 1/phi^2 = 3 | TRINITY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
@gHashTag Claude

gHashTag commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

t27b-native-ratchet is red here as on master (ratchet red), but the run also confirms what this PR needed to show:

  • 1077 / 1236 … 1720 files: one file more than master in the same window, and one pass more. The new spec runs and passes natively.
  • reference disagree 0: the reference path still reads every report under the new exit codes, with t27b's T27C_TEST_REPORT_EXIT_ZERO=1.

I've pushed the spec's ledger row as pass (598ed07), so it isn't counted as unlisted. The check isn't one of the required four.

Generated by Claude Code

This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
@claude
ledger(t27b): hold self_attention.t27 as blocked, ported from #8143 (…
4ebf901 
…Refs #8141, Closes #7370)

master stores max_not_pass 26 over 25 non-passing rows, so
scripts/ci/test_a_t27b_spec_cannot_move_silently.py ("the cap equals
reality") and with it loop-tools-tracked are red on master and on every
PR's merge with it; t27b-native-ratchet also reports the missing
self_attention.t27 row as UNLISTED. #8143 puts the row back (the same
row master had before #8059); this is that one line, byte-identical,
so it no-ops once #8143 lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
This was referenced Oct 9, 2026
Merged
Closed
Merged
Merged
Merged
Merged
Merged
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-09 12:05:40 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 46
PRs with All Checks Green 4
READY 0
FAILING 46
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 46 + 0 + 0 = 46, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=3c0ade9e73e4 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

1 similar comment
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-09 12:05:40 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 46
PRs with All Checks Green 4
READY 0
FAILING 46
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 46 + 0 + 0 = 46, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=3c0ade9e73e4 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Open
Closed
claude added 2 commits October 9, 2026 14:23
@claude
Merge master into test-report-exit-7370 (Refs #7370)
b2396f0 
bootstrap/src/main.rs: both sides added a generated module after
independence.rs -- test_report_exit (this PR) and seal_identity /
silicon_queue (#8095) -- so both are kept. docs/reports/
t27b_expectations.json is master's (#8173: cap 21, self_attention.t27
passes; #8143 closed) plus this PR's test_report_exit.t27 "pass" row;
the self_attention "blocked" row ported from #8143 is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
@claude
test(t27c): each test-report test gets its own scratch dir (Closes #7370
9678dc9 
)

`tri harness scratch --gate` failed on this PR: test_report_timeout.rs
now holds 3 tests, and the timeout test named its directory by the pid
alone (t27-test-timeout-{pid}), the exit test likewise. One scratch()
helper, pid AND counter, like the other bootstrap tests: the pid
separates runs, the counter the tests of one run.

cargo test -p t27c --test test_report_timeout: 3 passed;
tri harness scratch --gate and --self-check: rc 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
This was referenced Oct 9, 2026
Merged
Merged
Draft
Merged
@gHashTag Claude

gHashTag commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

Two checks are red on 9678dc9. Neither comes from this PR; both come from master.

Neither check is one of the required four. This head also fixes this PR's own red check, No test binary shares one scratch directory: each test now gets its own scratch directory, keyed by pid plus a counter.

Generated by Claude Code

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-09 15:00:39 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 45
PRs with All Checks Green 5
READY 0
FAILING 45
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 45 + 0 + 0 = 45, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=557cd271f4e3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions github-actions Bot mentioned this pull request Oct 9, 2026
Merged
@claude
proofs: drop the Lean models of the 22 deleted stubs, ported from #8223…
488ac9c 
… (Refs #8222, Closes #7370)

test-ratchet was red on this PR (and on master): #8054 deleted 22 empty
spec stubs, but proofs/lean4/Trinity/IcarusLowerable/Completeness.lean
still modelled them, so corpus_classifier_matches_lean_completeness
found 26 envs without specs. #8223 fixes it; this is its two files,
byte-identical to its head ad0b510, so it no-ops once #8223 lands.

cargo test -p t27c --test icarus_lowerable
corpus_classifier_matches_lean_completeness: ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScHVrGSr6zZUdwkdC8DR9k
This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-09 15:21:04 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 47
PRs with All Checks Green 3
READY 0
FAILING 47
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 47 + 0 + 0 = 47, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=557cd271f4e3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
@gHashTag
gHashTag merged commit ca9866b into master Oct 9, 2026
43 of 45 checks passed
@gHashTag gHashTag mentioned this pull request Oct 9, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

t27c test-report exits 0 when tests FAIL or the report is BLOCKED

2 participants

@gHashTag @claude