You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gate-16 spec-coverage prints PASS over 1466 findings on every --full run: a hardcoded 'origin/development' base manufactures an empty diff (sibling of #347, but falsely GREEN) #361
Sibling of #347 (gate-61 dead on --full), same root cause, worse symptom: gate-61 emits NOT APPLICABLE, which the runner excludes from the verdict. gate-16 emits PASS, which counts as an applicable gate that ran and passed — so it inflates the COVERAGE line and reads as a clean bill of health.
The default is hardcoded and non-empty. On a --full / workflow_dispatch run the runner sets no base ref, so instead of scanning the tree the gate silently diffs origin/development...HEAD. On the development branch itself that diff is empty — zero changed files, zero findings, PASS.
Positive control — one tree, one checker, only the scope input differs
Canonical package at 3c8da4c, run against a worktree pinned to pipelinq development @ 206db4be:
And CI agrees. pipelinq run 31505417691, whose own log says
[hydra-gates] Base ref: n/a — --full requested, scanning the entire tree.
prints
[gate-16] spec-coverage: PASS
over 1466 untagged methods, inside COVERAGE: 58 of 64 declared gates reported a result (6 not applicable; 58 of 58 applicable gates ran).
So the same repository has three gate-16 numbers, and the reassuring one is the fiction:
scope
result
--full (what a fleet audit runs)
PASS
diff vs origin/beta
185
--mode report (whole tree)
1466
It is exactly one checker
Grepping every HYDRA_GATE_BASE_REF consumer in scripts/lib/, check_spec_coverage.py is the only one with a non-empty hardcoded default. The rest default to "" and correctly report NOT APPLICABLE with an honest reason — in the very same run, gates 29/47/48 say:
this run is not diff-scoped (no --scope-to-diff / no base ref), so there is no change set to classify
Suggested fix — gate-19 already does it correctly
check_e2e_coverage.py is the model: no base ref means scan everything, and an empty diff when a base ref was given is reported as EMPTY SCOPE (exit 3), explicitly "not a pass".
base_ref=os.environ.get("HYDRA_GATE_BASE_REF")
ifbase_ref:
changed=changed_lines(base_ref, app_dir)
stripped=spec_tags_removed(base_ref, app_dir)
else:
# --full: no diff to scope to. Audit the whole tree.
...
Two properties worth preserving from gate-19's implementation:
No base ref ⇒ whole tree, never a manufactured diff.
Base ref given but diff empty ⇒ its own distinct exit, never PASS. A gate that had nothing to look at must not print the same word as a gate that looked and found nothing.
⚠️ Landing this turns a permanently-green gate red in every repo at once, exactly as #347 does — so it probably wants the same release handling.
Second-repo confirmation: procest. Hidden gate-16 debt = 445, and the 0 reproduces exactly as described.
Three-state positive control on ONE tree (procest@4061a600e, which isorigin/development), changing only the scope input. Gate package b8c7ead; origin/beta=3dfbb74f5, origin/main=38ae9724c.
base ref
result
origin/development — the hardcoded default, i.e. a --full CI run on development
count=0 → PASS
origin/beta
count=113
origin/main
count=445
--mode report (whole tree, scope-free)
uncovered_count=445
The --mode report figure and the origin/main figure agree at 445, which is a useful cross-check: the whole-tree answer is reachable two independent ways, and neither is the number CI prints.
Why the zero is structural, not incidental
On a full-scope run on development, HEAD == origin/development, so changed_lines(base_ref, app_dir) diffs a commit against itself. The diff is empty by construction — the gate opens no files. It is not that it looked and found nothing; it never looked.
And as filed, that PASS counts toward the "N of N applicable gates ran" coverage line, so the table reads as more thorough for having measured nothing. This is the inverse of #347/gate-61, which at least prints NOT APPLICABLE and is excluded from coverage. A false PASS that raises the coverage denominator is strictly worse than a false NOT APPLICABLE.
Effect on procest's published table
procest's measured full-scope table was 51 PASS / 7 FAIL, ~1386 findings, with no gate-16 line at all. Restated with the base named:
~1499 against origin/beta (the human-gated dev→beta scope)
~1831 whole-tree
One scoping caution for the fleet write-up
procest's 445 does not dwarf its table the way pipelinq's 1466 does. If the fleet summary generalises pipelinq's ratio ("hidden gate-16 exceeds the whole reported table"), it will overstate procest by roughly 4×. The hidden number needs measuring per repo, not scaling.
Note on my figure's provenance
The 445 was measured after procest#792 landed, which added four @spec docblocks and repointed one dangling anchor (@spec openspec/changes/dso-omgevingsloket/tasks.md#T07 — that change directory does not exist at all, and gate-46 reported nothing). So 445 is marginally better than the pre-session tree. It is a current-state number, not a before/after.
Confirmed in a second repo: openregister, gate-16 = 232, CI printing PASS
Canonical package ConductionNL/.github@main = 3c8da4c, run on a pristine worktree of openregister@origin/development = fdb0d6ca8. One tree, one checker, only the scope input changed:
(no HYDRA_GATE_BASE_REF — exactly what CI's --full run does) # count=0 EXIT=0 PASS
HYDRA_GATE_BASE_REF=origin/beta # count=232
HYDRA_GATE_BASE_REF=origin/main # count=232
--mode report uncovered_count=234
A sample finding from the base-ref run, so the 232 is not an abstraction:
openregister's honest full-scope run (31488412842) reported:
[hydra-gates] COVERAGE: 58 of 64 declared gates reported a result
(6 not applicable to this repo/diff; 58 of 58 applicable gates ran).
gate-16 is one of the 58. Unlike gate-61 — whose NOT APPLICABLE at least leaves a visible hole and does not count toward coverage — gate-16 contributes a PASS to the "58 of 58 applicable gates ran" line. So the very sentence used to certify that the run was not push-scope-masked is the sentence that launders this one. An agent auditing coverage sees gate-16 present and green and moves on.
Effect on this repo's published total
openregister was recorded fleet-wide as 935 findings (gate-19 848 + gate-25 66 + gate-26 21). With gate-16 measured against a real base ref:
gate
CI --full
honest
base
gate-19 e2e-coverage
848
848
full tree
gate-25 contract-coverage
66
66
full tree
gate-26 visual-coverage
21
16 (see #362)
full tree
gate-16 spec-coverage
0 — "PASS"
232
origin/beta and origin/main
total
935
1167
⚠️ Worth noting for anyone comparing numbers: origin/beta and origin/main return the same 232 here. That is not a given — openconnector's gate-19 moved 213 → 349 between those two bases. Quote the base ref regardless.
On the shared root cause
Agreed that #347 and this are one bug in two checkers. Reading it here, the distinguishing property is not "gate-61 vs gate-16" but what the checker does with an empty selection: gate-61 exits 3 (EMPTY_SCOPE) and the runner renders NOT APPLICABLE; gate-16 exits 0 and the runner renders PASS. The same defect is therefore visible in one gate and invisible in the other, and only the invisible one inflates the coverage count.
That suggests the fix wants a second half beyond or None: a checker that selected nothing must not be able to return EXIT_PASS.check_contract_coverage.py already carries the vocabulary for this (EXIT_EMPTY_SCOPE = 3, with .github#242 cited as the reason PASS and "I inspected nothing" were separated). Making gate-16 use it would mean that even if a base-ref default regressed again, the failure mode would be a visible NOT APPLICABLE rather than a silent green.
Not merging anything here — holding for #359's acceptance suite as you asked, so the fix lands with a fixture.
Independent confirmation from ConductionNL/openbuild, plus a committed positive control.
One tree (ff25c1be5 = origin/development), gate package @ 81c8c97, only the
base input changed:
base
gate-16
origin/development (ff25c1be5 — the unguarded --full default)
# count=0 → PASS
origin/beta (5ab286fba)
259
origin/main (e9c22311e)
445
--mode report, whole tree
439 (413 frontend / 26 backend)
main and report mode agree to within 6; the PASS is the outlier.
The positive control is the part worth copying, because the interesting row
is the second one, not the first. A public function plantedGateSixteenProbe(): JSONResponse carrying no @spec was added to lib/Controller/SettingsController.php and committed (uncommitted, it drops
out of the changed-method set entirely — check_spec_coverage.py:157 derives
that set from committed history while reading @spec from disk, so an
uncommitted plant manufactures a false "this gate is blind"):
The gate reports PASS over a finding it had named one command earlier. Plant reverted.
Confirming the mechanism against the canonical package, since the framing has
been corrected once already: the checker is not the culprit — it honours an
explicit base. bin/hydra-gates assembles --scope-to-diff --base "${BASE_REF}" insideif [ "${SCOPE_DIFF}" = "1" ], so on --full no base is forwarded and run-hydra-gates.sh:204's BASE_REF="origin/development" survives to HYDRA_GATE_BASE_REF="${BASE_REF}" at line 3034. Worth noting the base-IS-HEAD
rescue (resolve-push-base.sh) is also under SCOPE_DIFF=1, so it cannot
catch this either — the --full path has no base handling at all, not merely a
stale default.
⚠️Do not scale one app's ratio to another. pipelinq's hidden number was
1466 and procest's 445; openbuild's is 445 whole-tree. The ratio to each app's
reported table is not a constant.
Confirmed independently on hermiq — with a COMMITTED planted true positive showing the checker itself is not blind
One tree (ConductionNL/hermiqdevelopment @ 468f5f13), canonical package @main 81c8c97, only the base changed:
base
gate-16
origin/development — the checker's default, i.e. what the runner leaves it on for --full
# count=0 → PASS
origin/beta
467
origin/main
444
And on a --full run through bin/hydra-gates, the two lines land in the same log:
[hydra-gates] Base ref: n/a — --full requested, scanning the entire tree.
[gate-16] spec-coverage: PASS
[hydra-gates] COVERAGE: 58 of 64 declared gates reported a result (58 of 59 applicable gates ran).
The contrast in that same log is the sharpest statement of the bug I have seen. Gates 47
and 48 hit the identical missing-base condition and report it honestly:
[gate-47] NOT APPLICABLE — this run is not diff-scoped (no --scope-to-diff / no base ref), so there is no change set to classify.
…and a NOT APPLICABLE is excluded from the coverage denominator. gate-16 hits the same
condition, prints PASS, and is counted. Three gates, one root cause, two of them
truthful.
Positive control (committed, per the zaakafhandelapp lesson)
Because a 0 from a checker is indistinguishable from a dead checker, I planted a tracked, committedlib/Service/Gate16PlantService.php::plantedUnspeccedMethod() carrying no @spec. Expectation was written down before running: ≥1, naming the plant.
Plant reverted; count returned to 0. So check_spec_coverage.py is working correctly —
the 0 is purely the empty development…development self-diff, which matches the
correction already recorded in this issue that :508 honours an explicit base and the string
is only a default. The fix belongs in the runner.
One caution on extrapolation
hermiq's hidden number is 467 (dev→beta). Its previously-quoted push-scope figure was 473 — i.e. here the hidden number is roughly the same size as the push-scope one, not a
multiple of it. pipelinq's 1466 : reported-table ratio does not transfer; procest already
showed the same. Measure each app, and name the base beside every number.
Activity
rubenvdlinde commentedon Aug 11, 2026
Second-repo confirmation: procest. Hidden gate-16 debt = 445, and the
0reproduces exactly as described.Three-state positive control on ONE tree (
procest@4061a600e, which isorigin/development), changing only the scope input. Gate packageb8c7ead;origin/beta=3dfbb74f5,origin/main=38ae9724c.origin/development— the hardcoded default, i.e. a--fullCI run on developmentcount=0→ PASSorigin/betacount=113origin/maincount=445--mode report(whole tree, scope-free)uncovered_count=445The
--mode reportfigure and theorigin/mainfigure agree at 445, which is a useful cross-check: the whole-tree answer is reachable two independent ways, and neither is the number CI prints.Why the zero is structural, not incidental
On a full-scope run on development,
HEAD == origin/development, sochanged_lines(base_ref, app_dir)diffs a commit against itself. The diff is empty by construction — the gate opens no files. It is not that it looked and found nothing; it never looked.And as filed, that PASS counts toward the "N of N applicable gates ran" coverage line, so the table reads as more thorough for having measured nothing. This is the inverse of
#347/gate-61, which at least printsNOT APPLICABLEand is excluded from coverage. A falsePASSthat raises the coverage denominator is strictly worse than a falseNOT APPLICABLE.Effect on procest's published table
procest's measured full-scope table was 51 PASS / 7 FAIL, ~1386 findings, with no gate-16 line at all. Restated with the base named:
origin/beta(the human-gated dev→beta scope)One scoping caution for the fleet write-up
procest's 445 does not dwarf its table the way pipelinq's 1466 does. If the fleet summary generalises pipelinq's ratio ("hidden gate-16 exceeds the whole reported table"), it will overstate procest by roughly 4×. The hidden number needs measuring per repo, not scaling.
Note on my figure's provenance
The 445 was measured after procest#792 landed, which added four
@specdocblocks and repointed one dangling anchor (@spec openspec/changes/dso-omgevingsloket/tasks.md#T07— that change directory does not exist at all, and gate-46 reported nothing). So 445 is marginally better than the pre-session tree. It is a current-state number, not a before/after.rubenvdlinde commentedon Aug 11, 2026
Confirmed in a second repo: openregister, gate-16 = 232, CI printing
PASSCanonical package
ConductionNL/.github@main=3c8da4c, run on a pristine worktree ofopenregister@origin/development=fdb0d6ca8. One tree, one checker, only the scope input changed:A sample finding from the base-ref run, so the 232 is not an abstraction:
Why this went unseen here specifically
openregister's honest full-scope run (31488412842) reported:
gate-16 is one of the 58. Unlike gate-61 — whose
NOT APPLICABLEat least leaves a visible hole and does not count toward coverage — gate-16 contributes aPASSto the "58 of 58 applicable gates ran" line. So the very sentence used to certify that the run was not push-scope-masked is the sentence that launders this one. An agent auditing coverage sees gate-16 present and green and moves on.Effect on this repo's published total
openregister was recorded fleet-wide as 935 findings (gate-19 848 + gate-25 66 + gate-26 21). With gate-16 measured against a real base ref:
--full#362)origin/betaandorigin/mainorigin/betaandorigin/mainreturn the same 232 here. That is not a given — openconnector's gate-19 moved 213 → 349 between those two bases. Quote the base ref regardless.On the shared root cause
Agreed that
#347and this are one bug in two checkers. Reading it here, the distinguishing property is not "gate-61 vs gate-16" but what the checker does with an empty selection: gate-61 exits 3 (EMPTY_SCOPE) and the runner rendersNOT APPLICABLE; gate-16 exits 0 and the runner rendersPASS. The same defect is therefore visible in one gate and invisible in the other, and only the invisible one inflates the coverage count.That suggests the fix wants a second half beyond
or None: a checker that selected nothing must not be able to returnEXIT_PASS.check_contract_coverage.pyalready carries the vocabulary for this (EXIT_EMPTY_SCOPE = 3, with.github#242cited as the reason PASS and "I inspected nothing" were separated). Making gate-16 use it would mean that even if a base-ref default regressed again, the failure mode would be a visibleNOT APPLICABLErather than a silent green.Not merging anything here — holding for
#359's acceptance suite as you asked, so the fix lands with a fixture.rubenvdlinde commentedon Aug 11, 2026
Independent confirmation from
ConductionNL/openbuild, plus a committed positive control.One tree (
ff25c1be5=origin/development), gate package @81c8c97, only thebase input changed:
origin/development(ff25c1be5— the unguarded--fulldefault)# count=0→ PASSorigin/beta(5ab286fba)origin/main(e9c22311e)--mode report, whole treemainand report mode agree to within 6; the PASS is the outlier.The positive control is the part worth copying, because the interesting row
is the second one, not the first. A
public function plantedGateSixteenProbe(): JSONResponsecarrying no@specwas added tolib/Controller/SettingsController.phpand committed (uncommitted, it dropsout of the changed-method set entirely —
check_spec_coverage.py:157derivesthat set from committed history while reading
@specfrom disk, so anuncommitted plant manufactures a false "this gate is blind"):
origin/developmentcount=1—lib/Controller/SettingsController.php::plantedGateSixteenProbe — missing @specHEADcount=0→ PASSThe gate reports PASS over a finding it had named one command earlier. Plant reverted.
Confirming the mechanism against the canonical package, since the framing has
been corrected once already: the checker is not the culprit — it honours an
explicit base.
bin/hydra-gatesassembles--scope-to-diff --base "${BASE_REF}"inside
if [ "${SCOPE_DIFF}" = "1" ], so on--fullno base is forwarded andrun-hydra-gates.sh:204'sBASE_REF="origin/development"survives toHYDRA_GATE_BASE_REF="${BASE_REF}"at line 3034. Worth noting the base-IS-HEADrescue (
resolve-push-base.sh) is also underSCOPE_DIFF=1, so it cannotcatch this either — the
--fullpath has no base handling at all, not merely astale default.
1466 and procest's 445; openbuild's is 445 whole-tree. The ratio to each app's
reported table is not a constant.
rubenvdlinde commentedon Aug 11, 2026
Confirmed independently on hermiq — with a COMMITTED planted true positive showing the checker itself is not blind
One tree (
ConductionNL/hermiqdevelopment@468f5f13), canonical package@main81c8c97, only the base changed:origin/development— the checker's default, i.e. what the runner leaves it on for--full# count=0→ PASSorigin/betaorigin/mainAnd on a
--fullrun throughbin/hydra-gates, the two lines land in the same log:The contrast in that same log is the sharpest statement of the bug I have seen. Gates 47
and 48 hit the identical missing-base condition and report it honestly:
…and a
NOT APPLICABLEis excluded from the coverage denominator. gate-16 hits the samecondition, prints PASS, and is counted. Three gates, one root cause, two of them
truthful.
Positive control (committed, per the zaakafhandelapp lesson)
Because a
0from a checker is indistinguishable from a dead checker, I planted a tracked,committed
lib/Service/Gate16PlantService.php::plantedUnspeccedMethod()carrying no@spec. Expectation was written down before running: ≥1, naming the plant.Plant reverted; count returned to
0. Socheck_spec_coverage.pyis working correctly —the
0is purely the emptydevelopment…developmentself-diff, which matches thecorrection already recorded in this issue that
:508honours an explicit base and the stringis only a default. The fix belongs in the runner.
One caution on extrapolation
hermiq's hidden number is 467 (dev→beta). Its previously-quoted push-scope figure was
473 — i.e. here the hidden number is roughly the same size as the push-scope one, not a
multiple of it. pipelinq's 1466 : reported-table ratio does not transfer; procest already
showed the same. Measure each app, and name the base beside every number.
test(gate-16): the fixture #364 was missing
docs(gates): the runner header described the old default
feat(gates)!: scan the whole tree by default — ADR-020 superseded (#378)
fix(gate-16): a coding-standard reformat is not a set of changed meth…