Skip to content

fix(gates): a report-only gate must not print a FAIL verdict - #731

Merged
rubenvdlinde merged 1 commit into
mainfrom
fix/a-report-only-gate-must-not-print-a-fail-verdict
Sep 9, 2026
Merged

rubenvdlinde merged 1 commit into
mainfrom
fix/a-report-only-gate-must-not-print-a-fail-verdict

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Closes #729.

What was wrong

check_newman_reach.py printed a literal FAIL verdict line whenever it had findings. The runner cats that log to stdout and then emits its own _warn, so one run carried two contradicting verdicts for one gate, 45 lines apart, with the FAIL arriving first and inside the RESULT: N GATE(S) FAILED block. check_exclusion_evidence.py had the identical shape, so gate-113 read the same way.

The cause is structural, as the issue says: the verdict word is chosen in the helper, the blocking decision is made in the runner, and the helper cannot know the answer.

What this changes

1. Both helpers print a count, never a verdict word. Gate-114's helper is the pattern: state what was measured, let the exit code carry the answer, let the runner choose the word.

-[gate-112] newman-reach: FAIL — 17 finding(s). 305 of 620 committed request(s) run.
+[gate-112] newman-reach: 17 finding(s). 305 of 620 committed request(s) run.

The PASS lines went the same way, for the same reason: on the clean path the runner greps that line out of the log and prints it just above its own _pass, so it was a second verdict word too.

Exit codes, log contents, finding text and the runner's own counting greps are untouched.

2. A rule, so this cannot come back: one verdict line per gate. test_gate_acceptance_matrix.sh now fails any run in which one gate prints more than one verdict-shaped line, in either arm of any bundle. The runner already stated this contract in _fail's own comment ("the contract of this runner's stdout is ONE [gate-N] name: VERDICT line per gate") and nothing enforced it.

It runs over every gate that emitted, not only the rows a bundle declares, and that detail is load-bearing: the third instance below is a declared row in none of the five bundles it appears in, so a check scoped to expect.conf rows would have reported it zero times. It also refuses to grade a run whose verdict lines it could not read at all, because a check over zero lines passes for free.

3. It found a third instance the same minute it was added. Gate-104 prints

[gate-104] reports-one-page: NOT APPLICABLE — ... Nothing was inspected ...
[gate-104] reports-one-page: PASS

in every repo with no type:"reports" page: 12 occurrences across the fixture bundles, and every parser here takes the first. The note is worth keeping, so it is kept, with the verdict word removed. Whether _pass is the right verdict at all over a manifest this gate did not open is a separate decision about coverage accounting across 21 repos, and I have deliberately not taken it here. It is written up in the comment and in the report.

4. The two expect.conf rows now say WARNING, which is what the runner actually reaches. They said FAIL and passed anyway, on the helper's own FAIL line. In other words, the fixture that is supposed to prove gate-112 works was being satisfied by the defect.

5. The subject assertion now runs for WARNING too. This is the part worth reviewing closely. The planted arm's load-bearing assertion is not the verdict word, it is "the finding NAMES the planted subject". It was gated on [ "${_pv}" = "FAIL" ], so writing an advisory gate's row honestly would have silently switched that assertion off for gates 19, 112 and 113. A gate may be demoted. The requirement that it name what it found is not part of the demotion.

What I watched fail

Every arm was mutation checked: break the behaviour, confirm the right assertion reddens, restore from a byte copy, diff against the backup.

Mutation Expected to redden Result
Put FAIL — back in check_newman_reach.py the new unit arm FAILED (failures=1), exit 1, naming the line
Put FAIL — back in check_exclusion_evidence.py the new unit arm FAILED (failures=1), exit 1, naming the line
Put FAIL — back, then run the acceptance matrix one-verdict-line rule reddens, printing both lines
Set the newman row back to FAIL the verdict grade wanted 'FAIL', got: ... WARNING
Restore the verdict-shaped NOT APPLICABLE on gate-104 one-verdict-line rule reddens in both arms, printing both lines
Strip the collection path out of the finding text the subject assertion under WARNING reddens, which is what proves change 5 actually runs
The same, with change 5 reverted to [ "$_pv" = "FAIL" ] nothing passes clean, which is the control: without change 5 a gate-112 finding that names nothing is green

Full local runs, read by exit code: test_check_newman_reach.py 23 tests, test_check_exclusion_evidence.py 39 tests, test_gate_acceptance_matrix.sh over all bundles.

Not in here

Nothing merges from me. Turning either gate blocking is still per repo, via HYDRA_GATE_NEWMAN_REACH_BLOCKING=1 and HYDRA_GATE_EXCLUSION_EVIDENCE_BLOCKING=1, and this change does not move that switch.

check_newman_reach.py printed a literal FAIL line whenever it had findings,
regardless of whether gate-112 blocks in that repo. The runner cats that log
and then emits its own _warn, so one run carried two contradicting verdicts
for one gate, 45 lines apart, with the FAIL arriving first and inside the
RESULT block. check_exclusion_evidence.py had the same shape, so gate-113 read
the same way. Two readers, on two separate executions, counted a failure that
had not happened.

The cause is structural: the verdict word is chosen in the helper and the
blocking decision is made in the runner, and the helper cannot know the answer.
Both helpers now state a count and let the exit code carry the verdict, which
is the shape gate-114's helper already uses.

A rule rather than a habit: test_gate_acceptance_matrix.sh now refuses any run
in which one gate prints more than one verdict-shaped line, over every gate
that emitted rather than only the rows a bundle declares. That distinction is
load-bearing. It found a third instance the minute it was added: gate-104
prints a verdict-shaped NOT APPLICABLE note and then PASSes, in five bundles,
and is a declared row in none of them. The note keeps its information and
loses its verdict word. Whether _pass is the right verdict at all over a
manifest that gate did not open is an accounting change across 21 repos and is
left for its own review.

Both expect.conf rows now say WARNING, which is what the runner actually
reaches. They said FAIL and passed anyway, on the helper's own FAIL line: the
fixture meant to prove gate-112 works was being satisfied by the defect. The
subject assertion, which is the load-bearing half of the planted arm, now runs
for WARNING as well as FAIL, so writing an advisory gate's row honestly no
longer switches it off.

Closes #729
@rubenvdlinde
rubenvdlinde merged commit 930e33d into main Sep 9, 2026
40 checks passed
@rubenvdlinde
rubenvdlinde deleted the fix/a-report-only-gate-must-not-print-a-fail-verdict branch September 9, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gate-112 (and gate-113) print a literal FAIL line while report-only, so counting FAIL lines gives the wrong answer

1 participant