Skip to content

gate-16 spec-coverage prints PASS over 1466 findings on every --full run: a hardcoded 'origin/development' base manufactures an empty diff (sibling of #347, but falsely GREEN) #361

Closed

Description

@rubenvdlinde

Sibling of #347 (gate-61 dead on --full), same root cause, worse symptom: gate-61 emits NOT APPLICABLE, which the runner excludes from the verdict. gate-16 emits PASS, which counts as an applicable gate that ran and passed — so it inflates the COVERAGE line and reads as a clean bill of health.

The line

hydra-gates/scripts/lib/check_spec_coverage.py:508

base_ref = os.environ.get("HYDRA_GATE_BASE_REF", "origin/development")

The default is hardcoded and non-empty. On a --full / workflow_dispatch run the runner sets no base ref, so instead of scanning the tree the gate silently diffs origin/development...HEAD. On the development branch itself that diff is empty — zero changed files, zero findings, PASS.

Positive control — one tree, one checker, only the scope input differs

Canonical package at 3c8da4c, run against a worktree pinned to pipelinq development @ 206db4be:

$ env -u HYDRA_GATE_BASE_REF  python3 .../check_spec_coverage.py <tree>
# count=0

$ HYDRA_GATE_BASE_REF=origin/beta python3 .../check_spec_coverage.py <tree>
...
src/views/widgets/MyLeadsWidget.vue::onViewAll — missing @spec
# count=185

$ python3 .../check_spec_coverage.py <tree> --mode report
uncovered_count = 1466

And CI agrees. pipelinq run 31505417691, whose own log says

[hydra-gates] Base ref: n/a — --full requested, scanning the entire tree.

prints

[gate-16] spec-coverage: PASS

over 1466 untagged methods, inside COVERAGE: 58 of 64 declared gates reported a result (6 not applicable; 58 of 58 applicable gates ran).

So the same repository has three gate-16 numbers, and the reassuring one is the fiction:

scope result
--full (what a fleet audit runs) PASS
diff vs origin/beta 185
--mode report (whole tree) 1466

It is exactly one checker

Grepping every HYDRA_GATE_BASE_REF consumer in scripts/lib/, check_spec_coverage.py is the only one with a non-empty hardcoded default. The rest default to "" and correctly report NOT APPLICABLE with an honest reason — in the very same run, gates 29/47/48 say:

this run is not diff-scoped (no --scope-to-diff / no base ref), so there is no change set to classify

Suggested fix — gate-19 already does it correctly

check_e2e_coverage.py is the model: no base ref means scan everything, and an empty diff when a base ref was given is reported as EMPTY SCOPE (exit 3), explicitly "not a pass".

base_ref = os.environ.get("HYDRA_GATE_BASE_REF")
if base_ref:
    changed = changed_lines(base_ref, app_dir)
    stripped = spec_tags_removed(base_ref, app_dir)
else:
    # --full: no diff to scope to. Audit the whole tree.
    ...

Two properties worth preserving from gate-19's implementation:

  1. No base ref ⇒ whole tree, never a manufactured diff.
  2. Base ref given but diff empty ⇒ its own distinct exit, never PASS. A gate that had nothing to look at must not print the same word as a gate that looked and found nothing.

⚠️ Landing this turns a permanently-green gate red in every repo at once, exactly as #347 does — so it probably wants the same release handling.

Found while re-measuring pipelinq's full-scope table after ConductionNL/pipelinq#769.

Activity

  1. rubenvdlinde commented on Aug 11, 2026

    @rubenvdlinde
    ContributorAuthor

    Second-repo confirmation: procest. Hidden gate-16 debt = 445, and the 0 reproduces exactly as described.

    Three-state positive control on ONE tree (procest@4061a600e, which is origin/development), changing only the scope input. Gate package b8c7ead; origin/beta=3dfbb74f5, origin/main=38ae9724c.

    base ref result
    origin/development — the hardcoded default, i.e. a --full CI run on development count=0 → PASS
    origin/beta count=113
    origin/main count=445
    --mode report (whole tree, scope-free) uncovered_count=445

    The --mode report figure and the origin/main figure agree at 445, which is a useful cross-check: the whole-tree answer is reachable two independent ways, and neither is the number CI prints.

    Why the zero is structural, not incidental

    On a full-scope run on development, HEAD == origin/development, so changed_lines(base_ref, app_dir) diffs a commit against itself. The diff is empty by construction — the gate opens no files. It is not that it looked and found nothing; it never looked.

    And as filed, that PASS counts toward the "N of N applicable gates ran" coverage line, so the table reads as more thorough for having measured nothing. This is the inverse of #347/gate-61, which at least prints NOT APPLICABLE and is excluded from coverage. A false PASS that raises the coverage denominator is strictly worse than a false NOT APPLICABLE.

    Effect on procest's published table

    procest's measured full-scope table was 51 PASS / 7 FAIL, ~1386 findings, with no gate-16 line at all. Restated with the base named:

    • ~1499 against origin/beta (the human-gated dev→beta scope)
    • ~1831 whole-tree

    One scoping caution for the fleet write-up

    procest's 445 does not dwarf its table the way pipelinq's 1466 does. If the fleet summary generalises pipelinq's ratio ("hidden gate-16 exceeds the whole reported table"), it will overstate procest by roughly 4×. The hidden number needs measuring per repo, not scaling.

    Note on my figure's provenance

    The 445 was measured after procest#792 landed, which added four @spec docblocks and repointed one dangling anchor (@spec openspec/changes/dso-omgevingsloket/tasks.md#T07 — that change directory does not exist at all, and gate-46 reported nothing). So 445 is marginally better than the pre-session tree. It is a current-state number, not a before/after.

  2. rubenvdlinde commented on Aug 11, 2026

    @rubenvdlinde
    ContributorAuthor

    Confirmed in a second repo: openregister, gate-16 = 232, CI printing PASS

    Canonical package ConductionNL/.github@main = 3c8da4c, run on a pristine worktree of openregister@origin/development = fdb0d6ca8. One tree, one checker, only the scope input changed:

    (no HYDRA_GATE_BASE_REF — exactly what CI's --full run does)   # count=0      EXIT=0   PASS
    HYDRA_GATE_BASE_REF=origin/beta                                # count=232
    HYDRA_GATE_BASE_REF=origin/main                                # count=232
    --mode report                                                  uncovered_count=234
    

    A sample finding from the base-ref run, so the 232 is not an abstraction:

    src/views/settings/sections/RetentionConfiguration.vue::set — missing @spec
    

    Why this went unseen here specifically

    openregister's honest full-scope run (31488412842) reported:

    [hydra-gates] COVERAGE: 58 of 64 declared gates reported a result
                  (6 not applicable to this repo/diff; 58 of 58 applicable gates ran).
    

    gate-16 is one of the 58. Unlike gate-61 — whose NOT APPLICABLE at least leaves a visible hole and does not count toward coverage — gate-16 contributes a PASS to the "58 of 58 applicable gates ran" line. So the very sentence used to certify that the run was not push-scope-masked is the sentence that launders this one. An agent auditing coverage sees gate-16 present and green and moves on.

    Effect on this repo's published total

    openregister was recorded fleet-wide as 935 findings (gate-19 848 + gate-25 66 + gate-26 21). With gate-16 measured against a real base ref:

    gate CI --full honest base
    gate-19 e2e-coverage 848 848 full tree
    gate-25 contract-coverage 66 66 full tree
    gate-26 visual-coverage 21 16 (see #362) full tree
    gate-16 spec-coverage 0 — "PASS" 232 origin/beta and origin/main
    total 935 1167

    ⚠️ Worth noting for anyone comparing numbers: origin/beta and origin/main return the same 232 here. That is not a given — openconnector's gate-19 moved 213 → 349 between those two bases. Quote the base ref regardless.

    On the shared root cause

    Agreed that #347 and this are one bug in two checkers. Reading it here, the distinguishing property is not "gate-61 vs gate-16" but what the checker does with an empty selection: gate-61 exits 3 (EMPTY_SCOPE) and the runner renders NOT APPLICABLE; gate-16 exits 0 and the runner renders PASS. The same defect is therefore visible in one gate and invisible in the other, and only the invisible one inflates the coverage count.

    That suggests the fix wants a second half beyond or None: a checker that selected nothing must not be able to return EXIT_PASS. check_contract_coverage.py already carries the vocabulary for this (EXIT_EMPTY_SCOPE = 3, with .github#242 cited as the reason PASS and "I inspected nothing" were separated). Making gate-16 use it would mean that even if a base-ref default regressed again, the failure mode would be a visible NOT APPLICABLE rather than a silent green.

    Not merging anything here — holding for #359's acceptance suite as you asked, so the fix lands with a fixture.

  3. rubenvdlinde commented on Aug 11, 2026

    @rubenvdlinde
    ContributorAuthor

    Independent confirmation from ConductionNL/openbuild, plus a committed positive control.

    One tree (ff25c1be5 = origin/development), gate package @ 81c8c97, only the
    base input changed:

    base gate-16
    origin/development (ff25c1be5 — the unguarded --full default) # count=0 → PASS
    origin/beta (5ab286fba) 259
    origin/main (e9c22311e) 445
    --mode report, whole tree 439 (413 frontend / 26 backend)

    main and report mode agree to within 6; the PASS is the outlier.

    The positive control is the part worth copying, because the interesting row
    is the second one, not the first. A public function plantedGateSixteenProbe(): JSONResponse carrying no @spec was added to
    lib/Controller/SettingsController.php and committed (uncommitted, it drops
    out of the changed-method set entirely — check_spec_coverage.py:157 derives
    that set from committed history while reading @spec from disk, so an
    uncommitted plant manufactures a false "this gate is blind"):

    tree base result
    plant committed origin/development count=1 — lib/Controller/SettingsController.php::plantedGateSixteenProbe — missing @spec
    identical tree, plant still on disk HEAD count=0 → PASS

    The gate reports PASS over a finding it had named one command earlier. Plant reverted.

    Confirming the mechanism against the canonical package, since the framing has
    been corrected once already:
    the checker is not the culprit — it honours an
    explicit base. bin/hydra-gates assembles --scope-to-diff --base "${BASE_REF}"
    inside if [ "${SCOPE_DIFF}" = "1" ], so on --full no base is forwarded and
    run-hydra-gates.sh:204's BASE_REF="origin/development" survives to
    HYDRA_GATE_BASE_REF="${BASE_REF}" at line 3034. Worth noting the base-IS-HEAD
    rescue (resolve-push-base.sh) is also under SCOPE_DIFF=1, so it cannot
    catch this either — the --full path has no base handling at all, not merely a
    stale default.

    ⚠️ Do not scale one app's ratio to another. pipelinq's hidden number was
    1466 and procest's 445; openbuild's is 445 whole-tree. The ratio to each app's
    reported table is not a constant.

  4. rubenvdlinde commented on Aug 11, 2026

    @rubenvdlinde
    ContributorAuthor

    Confirmed independently on hermiq — with a COMMITTED planted true positive showing the checker itself is not blind

    One tree (ConductionNL/hermiq development @ 468f5f13), canonical package @main
    81c8c97, only the base changed:

    base gate-16
    origin/development — the checker's default, i.e. what the runner leaves it on for --full # count=0 → PASS
    origin/beta 467
    origin/main 444

    And on a --full run through bin/hydra-gates, the two lines land in the same log:

    [hydra-gates] Base ref: n/a — --full requested, scanning the entire tree.
    [gate-16] spec-coverage: PASS
    [hydra-gates] COVERAGE: 58 of 64 declared gates reported a result (58 of 59 applicable gates ran).
    

    The contrast in that same log is the sharpest statement of the bug I have seen. Gates 47
    and 48 hit the identical missing-base condition and report it honestly:

    [gate-47] NOT APPLICABLE — this run is not diff-scoped (no --scope-to-diff / no base ref), so there is no change set to classify.

    …and a NOT APPLICABLE is excluded from the coverage denominator. gate-16 hits the same
    condition, prints PASS, and is counted. Three gates, one root cause, two of them
    truthful.

    Positive control (committed, per the zaakafhandelapp lesson)

    Because a 0 from a checker is indistinguishable from a dead checker, I planted a tracked,
    committed lib/Service/Gate16PlantService.php::plantedUnspeccedMethod() carrying no
    @spec. Expectation was written down before running: ≥1, naming the plant.

    $ HYDRA_GATE_BASE_REF=origin/development python3 .../check_spec_coverage.py .
    lib/Service/Gate16PlantService.php::plantedUnspeccedMethod — missing @spec
    # count=1
    

    Plant reverted; count returned to 0. So check_spec_coverage.py is working correctly —
    the 0 is purely the empty development…development self-diff
    , which matches the
    correction already recorded in this issue that :508 honours an explicit base and the string
    is only a default. The fix belongs in the runner.

    One caution on extrapolation

    hermiq's hidden number is 467 (dev→beta). Its previously-quoted push-scope figure was
    473 — i.e. here the hidden number is roughly the same size as the push-scope one, not a
    multiple of it. pipelinq's 1466 : reported-table ratio does not transfer; procest already
    showed the same. Measure each app, and name the base beside every number.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions