Repository navigation
fix(gates): a report-only gate must not print a FAIL verdict - #731
Merged
rubenvdlinde merged 1 commit intoSep 9, 2026
Merged
Conversation
check_newman_reach.py printed a literal FAIL line whenever it had findings, regardless of whether gate-112 blocks in that repo. The runner cats that log and then emits its own _warn, so one run carried two contradicting verdicts for one gate, 45 lines apart, with the FAIL arriving first and inside the RESULT block. check_exclusion_evidence.py had the same shape, so gate-113 read the same way. Two readers, on two separate executions, counted a failure that had not happened. The cause is structural: the verdict word is chosen in the helper and the blocking decision is made in the runner, and the helper cannot know the answer. Both helpers now state a count and let the exit code carry the verdict, which is the shape gate-114's helper already uses. A rule rather than a habit: test_gate_acceptance_matrix.sh now refuses any run in which one gate prints more than one verdict-shaped line, over every gate that emitted rather than only the rows a bundle declares. That distinction is load-bearing. It found a third instance the minute it was added: gate-104 prints a verdict-shaped NOT APPLICABLE note and then PASSes, in five bundles, and is a declared row in none of them. The note keeps its information and loses its verdict word. Whether _pass is the right verdict at all over a manifest that gate did not open is an accounting change across 21 repos and is left for its own review. Both expect.conf rows now say WARNING, which is what the runner actually reaches. They said FAIL and passed anyway, on the helper's own FAIL line: the fixture meant to prove gate-112 works was being satisfied by the defect. The subject assertion, which is the load-bearing half of the planted arm, now runs for WARNING as well as FAIL, so writing an advisory gate's row honestly no longer switches it off. Closes #729
rubenvdlinde
deleted the
fix/a-report-only-gate-must-not-print-a-fail-verdict
branch
September 9, 2026 16:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #729.
What was wrong
check_newman_reach.pyprinted a literalFAILverdict line whenever it had findings. The runnercats that log to stdout and then emits its own_warn, so one run carried two contradicting verdicts for one gate, 45 lines apart, with theFAILarriving first and inside theRESULT: N GATE(S) FAILEDblock.check_exclusion_evidence.pyhad the identical shape, so gate-113 read the same way.The cause is structural, as the issue says: the verdict word is chosen in the helper, the blocking decision is made in the runner, and the helper cannot know the answer.
What this changes
1. Both helpers print a count, never a verdict word. Gate-114's helper is the pattern: state what was measured, let the exit code carry the answer, let the runner choose the word.
The
PASSlines went the same way, for the same reason: on the clean path the runner greps that line out of the log and prints it just above its own_pass, so it was a second verdict word too.Exit codes, log contents, finding text and the runner's own counting greps are untouched.
2. A rule, so this cannot come back: one verdict line per gate.
test_gate_acceptance_matrix.shnow fails any run in which one gate prints more than one verdict-shaped line, in either arm of any bundle. The runner already stated this contract in_fail's own comment ("the contract of this runner's stdout is ONE[gate-N] name: VERDICTline per gate") and nothing enforced it.It runs over every gate that emitted, not only the rows a bundle declares, and that detail is load-bearing: the third instance below is a declared row in none of the five bundles it appears in, so a check scoped to
expect.confrows would have reported it zero times. It also refuses to grade a run whose verdict lines it could not read at all, because a check over zero lines passes for free.3. It found a third instance the same minute it was added. Gate-104 prints
in every repo with no
type:"reports"page: 12 occurrences across the fixture bundles, and every parser here takes the first. The note is worth keeping, so it is kept, with the verdict word removed. Whether_passis the right verdict at all over a manifest this gate did not open is a separate decision about coverage accounting across 21 repos, and I have deliberately not taken it here. It is written up in the comment and in the report.4. The two
expect.confrows now sayWARNING, which is what the runner actually reaches. They saidFAILand passed anyway, on the helper's ownFAILline. In other words, the fixture that is supposed to prove gate-112 works was being satisfied by the defect.5. The subject assertion now runs for
WARNINGtoo. This is the part worth reviewing closely. The planted arm's load-bearing assertion is not the verdict word, it is "the finding NAMES the planted subject". It was gated on[ "${_pv}" = "FAIL" ], so writing an advisory gate's row honestly would have silently switched that assertion off for gates 19, 112 and 113. A gate may be demoted. The requirement that it name what it found is not part of the demotion.What I watched fail
Every arm was mutation checked: break the behaviour, confirm the right assertion reddens, restore from a byte copy,
diffagainst the backup.FAIL —back incheck_newman_reach.pyFAILED (failures=1), exit 1, naming the lineFAIL —back incheck_exclusion_evidence.pyFAILED (failures=1), exit 1, naming the lineFAIL —back, then run the acceptance matrixFAILwanted 'FAIL', got: ... WARNINGNOT APPLICABLEon gate-104WARNING[ "$_pv" = "FAIL" ]Full local runs, read by exit code:
test_check_newman_reach.py23 tests,test_check_exclusion_evidence.py39 tests,test_gate_acceptance_matrix.shover all bundles.Not in here
Nothing merges from me. Turning either gate blocking is still per repo, via
HYDRA_GATE_NEWMAN_REACH_BLOCKING=1andHYDRA_GATE_EXCLUSION_EVIDENCE_BLOCKING=1, and this change does not move that switch.