Skip to content

12 gates where a comment MANUFACTURES a finding (#415 class, false-positive half) #423

Open

Description

@rubenvdlinde

Found by the all-65 survey of the #415/#420 comment class. Full table with per-gate fixtures and verdicts: fleet-board/findings/gates-comment-class.md. Companion to #422 (the false-negative half).

This is the false-positive half: a comment crowds a bounded window, or a note about code that was REMOVED scores as that code.

🔑 This is not the milder half, it is the corrosive half. An author whose documented fix goes red and whose undocumented fix goes green learns to delete the explanation — and the gate then loses the only thing that would have let a human check it. 8 of gate-50's 16 opencatalogi findings were this shape.

gate site the comment that manufactures the finding
5 route-auth _head_block L1411 — WINDOW, not prose Two fixtures differing only in docblock length, both carrying a real multi-line #[AuthorizedAdminSetting(\n Application::APP_ID\n)]: 5 doc lines → PASS, 30 doc lines → FAIL. The contiguous run stops at the )] line, and past 20 doc lines the union no longer reaches the attribute. gate-5's prose axis is separately CLEAN — this is purely budget.
9 semantic-auth head/body sliced from raw src L632–633 A body whose only content is // We used to $this->requireAdmin(); here. Now anyone may read. → FAIL. _strip_comments exists in this file and is used only for the credential surface at L665.
15 dashboard-antipattern raw read_text() L196, DASHBOARD_TAG_RE.search(body) L210 <!-- removed 2026-08: <CnDashboardPage /> was nested here by mistake --> → FAIL. The removal note scores as the removal it documents.
21 conflict-markers runner inline bash, no mask at all A fenced code block in openspec/notes.md documenting what a conflict looks like, plus a marker inside a /* */ in PHP → FAIL — 2 file(s) with unresolved conflict markers. Neither file has a conflict. The existing .md setext carve-out does not help.
23 or-abstraction-anti-patterns lint-or-abstraction-anti-patterns.sh L418, grep -rln -e "search_path" A docblock reading "We deliberately do NOT set a Postgres search_path here — tenant isolation is OpenRegister's job (ADR-022)" → FAIL. The PDOK rule in the same file already routes through _code_lines(); the grep-kind rows never got it.
24 integration-parity runner inline ~L4527, raw grep -rqE 'new\s+LeafDescriptor' Two comments (// TODO: one day we could do \new LeafDescriptor(...)`, // We used to call registerIntegration({...}); removed) flip NOT APPLICABLE→SKIPPED (structural). Not a red gate — but structuralcounts against coverage, and--require-full-coverage` exit went 5 → 6 on the same tree.
27 no-phantom-cross-app-rpc check_phantom_cross_app_rpc.py L256–268, L384–389 Its comment test is stripped.startswith("//","*","/*","#") per LINE, so it only recognises comment-opening lines. One clean fixture → FAIL — 3: an interior /* */ line, a trailing // never do …getLeaf(…), and a string literal. No source_scope import in the file.
28 license-triangle check_license_triangle.py DECL A docblock line whose SENTENCE begins with the tag reads as a second declaration: a real * @license EUPL-1.2 plus * @license MIT was never used here — see the note above. → FAIL — 2. Mid-sentence and string forms correctly PASS, so the anchor is nearly right.
46 spec-anchor-existence check_spec_anchors.py TAG L65, unmasked and unanchored const spec = '@spec openspec/specs/imaginary/spec.md' → FAIL. Also a tag mid-sentence in prose (// See @spec openspec/specs/gone/spec.md#missing for why.) → FAIL. Gates 47 and 48 were given position-anchored regexes for exactly this; 46 never was. No false-negative direction exists here — the gate has no suppression path a comment can reach.
47 security-change-has-tests check_security_cochange.py _COMMENT_LINE_RE L145 Matches only lines starting with *, //, /*, #. A trailing // resolved via IUserSession elsewhere and an HTML comment (<!-- is not in its vocabulary at all, yet .vue is a candidate path) → FAIL — 2. This defeats the module's own stated promise: "Prose that merely mentions IUserSession is not — it is a sentence."
61 listener-work-placement check_listener_placement.py parse_registrations() L296 reads raw _strip_comments() L472 is applied at L482 and L520 but not here, so a listener existing only as two commented-out registration lines → FAIL, though it is not wired at all. The stripper is also not string-aware.
62 store-plane check_store_and_settings_surface.py L245 raw read_text() + plain substring test A docblock saying "We deliberately do NOT hit /apps/openregister/api/objects/ with an IClientService here — GenericStoreService owns that (ADR-080)" → FAIL — 1, identical to the real-violation fixture. This is exactly the doriath loadApp shape gate-64 was already fixed for.

Notes

  • The shared helper exists: lib/source_scope.py, offset-preserving so a line number computed on the mask still names the right line. Use it rather than a sixth hand-rolled stripper.
  • ⚠️ gate-5 is a WINDOW bug, not a prose bug, and it will not be fixed by masking anything. _head_block needs to find the attribute by structure (walk the contiguous annotation run to its start) rather than by distance — the same repair check_contract_coverage already made and documented at its L208, "WHY THIS IS NOT A LINE WINDOW".
  • ⚠️ Every one of these fixes REMOVES findings. That is the direction that needs the most care: pair each arm with the real defect it must still catch, or the repair is indistinguishable from switching the gate off. Run the positive control first — two of the survey's own fixtures initially printed the same verdict on both arms because the control had never failed.

Activity

  1. rubenvdlinde commented on Aug 13, 2026

    @rubenvdlinde
    ContributorAuthor

    Fixes for the four most serious instances of this class (gates 7, 25, 48, 49) are in #425, which also carries the full 65-gate survey table and the method notes.

  2. rubenvdlinde commented on Aug 13, 2026

    @rubenvdlinde
    ContributorAuthor

    10 of 12 fixed, in two PRs — #439 (checkers: 9, 27, 28, 46, 47, 61, 62) and #440 (runner: 5, 23, 24). Not merged. Full write-up with every number: fleet-board/findings/gates-423-fp.md.

    gate-15 is BLOCKED on #424. Reproduced — the comment-only fixture and the real one print byte-identical findings (Overview.vue:3 page=Overview component=Overview rule=dashboard-component-used-as-widget). The evidence is a .vue template and the only mask for it is markup_mask/vue_markup_mask, which #424 is mid-repair. Stopped at the boundary rather than adopting a mask whose delimiter handling is changing under me.

    gate-21 is DECLINED, with the trade measured rather than picked. The false positive is real (a fenced conflict example in openspec/notes.md plus a marker inside a PHP block comment → FAIL — 2 file(s), neither conflicted). But I let git write the counter-example rather than argue it: merging two branches that both edit the same docblock, and the same fenced block, produces markers inside the docblock and inside the fence, byte-for-byte the shape of the documentation. Masking PHP comments silences a genuinely conflicted docblock — which is the incident class this gate was built for — and exempting fences silences a genuinely conflicted fence. There is no textual discriminator because there is no textual difference. Closing this FP means accepting an FN of the same shape; that needs either an explicit decision here or a non-textual signal (merge state), and neither is this issue.

    Two false positives are kept ON PURPOSE and asserted in the suites so they cannot be closed silently: gate-27's string-literal finding (Rule B's evidence is ->call('decidesk', …)) and gate-47's string-literal line (requesttoken arrives as a header key). Blanking literals would trade a false positive for a false negative in gates whose findings are runtime fatals. Whoever closes #424 will see those arms go red and will have to decide deliberately.

    Fleet measurement (6 repos, SETS diffed not counted): every set identical, and that is the honest result — these shapes are latent in openregister/opencatalogi/procest/docudesk/larpingapp/softwarecatalog. Agreement across six repos is what a dead rig looks like, so each gate carries a liveness witness rather than a bare zero: gate-9's 12 surviving findings, gate-28's 19 (over 2,394 files), gate-46's 25 findings and 17,408 tags, gate-47's 460 classified files, gate-61's 69 parsed registrations, gate-5's 17,101 declarations inspected. The only fleet-visible movement is gate-46 seeing 2 fewer tags — // Per @spec …("the system MUST and NOTE: this used to read \@SPEC …#T07`.` — a sentence and a note about a tag that was removed, i.e. this class verbatim.

    One near-miss worth recording. gate-23's first two arms were exit-code assertions and both passed on the unfixed script: the capability rows have their own bake-in epoch, so the suite's forced BLOCK mode never reaches them and WARN mode exits 0 whatever it found. The arm was measuring the epoch, not the finding. They assert the file list now, with the capability epoch forced as a second witness. The tell was that the arm I knew should flip did not.

    And one where the anti-widening pair earned its place. gate-5's borrow fixture reports FAIL — 2 on the old runner — it was flagging the guarded method too — and FAIL — 1 on the new one, naming the unguarded one. The fix removes exactly the false finding and keeps the real one, in the same file.

    ⚠️ Overlap with #422: no checker file is touched by both. The contended file is run-hydra-gates.sh (#422 owns gates 1/3/14/30/38/45 there; this owns 5 and 24 — different functions). One genuine interaction: gate-46 and gate-16 read the same @spec tag from opposite directions and both want the same position anchor. If #422 anchors SPEC_RE, the two should converge on one shared rule rather than two spellings — worth doing at one site, not unilaterally by either of us.

  3. rubenvdlinde commented on Aug 13, 2026

    @rubenvdlinde
    ContributorAuthor

    ReDoS fixed — #439 is green, CodeQL included. All 16 non-skipped checks pass on c01dc48 (total_count=33, read with ?per_page=100), CodeQL pass, code-scanning alerts on refs/pull/439/head = 0, Package invariants (which runs the whole helper suite in CI) pass. Locally: 83 passed, 0 failed, 2 quarantined. Merges cleanly onto main at eb291f1 — re-checked after #434/#435/#436/#437/#441, not assumed.

    The alert was right and its example was wrong, and I nearly wrote its numbers down without running them. CodeQL names 'http://' + '!http://' * n; measured in CPython that is 0.000s at n=32,000, because \s*\Z can match empty and the engine's empty-repeat guard cuts the search short. The ambiguity is real — \S+ inside a starred alternation is ambiguous with itself — it just fires on a different subject. Sweeping shapes found the one that does, a repeated URL with one trailing byte that cannot match:

    'https://a' * n + ' \x01' OLD NEW
    n=18, len 164 0.0949s 0.0000s
    n=22, len 200 1.5284s 0.0000s
    n=24, len 218 6.6797s 0.0000s
    n=26, len 236 26.1822s 0.0000s
    n=28, len 254 >60s 0.0000s

    Every +2 multiplies by ~4. 254 characters — shorter than plenty of real headers — on a gate that reads every tracked lib/**/*.php. Rewritten as a linear token scan rather than tightened, so the ambiguity cannot exist rather than being harder to reach: flat on that input and still flat at 256 KB (0.0103s).

    The sweep found a second one the alert did not flag. gate-46's new anchor was [^\S\n]*(?:lead-in)?[^\S\n]* — two whitespace runs adjacent whenever the lead-in matches empty, i.e. O(N²). 64,000 spaces took >20s; each optional group now swallows its own trailing whitespace, and the same subject is 0.0072s (256,000 spaces: 0.0270s).

    A ReDoS fix that narrows a pattern is a gate weakening in a security fix's clothes, so both were re-measured rather than eyeballed. gate-28: fleet declaration counts identical per repo (1437/76/610/151/23/97 over 2,394 files), finding sets identical, larpingapp's 19 drift findings all still there. gate-46: finding sets identical (25 = 25), delta still exactly the same two prose tags. Same arms flip on revert against origin/main (T2/T3), same controls hold (T1/T4–T8). Both suites green, 37→38 and 57→58. The two new arms are shape assertions, not benchmarks — each bounds a true cost of ~0.00003s / ~0.007s at 5s.

    One thing worth having on the record for #440, which is already merged: locally test_gate_acceptance_matrix.sh failed once with 136 passed / 1 failed and no FAIL — line naming the arm, then passed alone on the same tree. Rather than calling it a flake, I checked a run with no contention: #440's own merge commit 8387110 ran hydra-gates package to success in 10m48s, whole suite included. It was the machine, not the change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions