Repository navigation
Description
Confirmed: gate-26 is satisfied by a bare component name in a COMMENT
Reproduced on doriath, three consecutive runs of
HYDRA_GATE_BASE_REF=4b825dc… python3 scripts/lib/check_visual_coverage.py .:
10 findings → 9 → 10.
The only thing that changed between runs was the existence of
tests/e2e/visual/_DEFECT_PROBE.ts, whose entire content was:
// PROBE — temporary. A bare mention, in a comment, of PersonalActivityView.
// Nothing here navigates anywhere, asserts anything, or shoots a screenshot.src/views/PersonalActivityView.vue dropped out of the finding list. No
screenshot, no navigation, no test( — not even valid test code. A .txt file
works identically.
Mechanism
is_covered() matches the needle against the raw text of the corpus:
for needle in needles: # {file stem, manifest page id}
if needle and needle in visual_corpus: # plain substring, whole file
return Trueand _e2e_corpus() concatenates every .ts .js .png .txt .json under
tests/e2e/visual/** (falling back to all of tests/e2e/**) without stripping
anything.
Why this is worse than a wrong number
Two second-order effects, both observed:
-
It fires on the documentation of itself. While writing up this defect in
doriath I put a warning paragraph intests/e2e/visual/_visual-helpers.ts
that named one view. That file is in the corpus, so the paragraph covered the
component. It surfaced only because the negative control's expected number
had been computed in advance: 7 findings came back where 8 were expected.
Anyone who fixes their app and only checks "did it go green" will not see it. -
A baseline of the wrong screen satisfies it just as well. The gate never
looks at what a screenshot contains._visual-helpers.ts::shootByNav— copied
verbatim into several apps — used to end with "falls back to a shot of
wherever we land if the link is absent, so a baseline is still produced".
That produces a stable, never-failing PNG named after a component it does not
contain. Combined with the above, an app can be fully green on gate-26 with
zero real visual proof.
Fleet measurement — how many components are covered ONLY by a comment
Script below imports the gate's own discover_new_pages / is_covered /
_visual_exclude_status, so its scope agrees with the gate by construction, and
compares is_covered against a comment-stripped corpus. Full-tree scope,
excludes-with-reason skipped. Run today:
| app | page components | covered ONLY by a comment |
|---|---|---|
| docudesk | 22 | 7 |
| opencatalogi | 18 | 3 |
| procest | 78 | 2 |
| softwarecatalog | 12 | 2 |
| pipelinq | 24 | 2 |
| openregister | — | 6 |
| openconnector | 10 | 0 |
| doriath | 14 | 0 (after this session's fix) |
| nldesign | 0 | 0 |
comparison is deliberately crude (/*…*/, <!--…-->, and // outside an
obvious ://), so a component named only inside a URL string on a line the
stripper mangled would be counted here and should not be. Each row needs eyes
before anyone acts on it. The doriath 10 → 9 → 10 result above is the part
that is not an estimate.
Proposed fix, and why it is not being merged unilaterally
Strip comments from the .ts/.js corpus before matching, string-aware so a
https:// inside a literal survives. check_e2e_coverage.py already carries a
tokeniser that does this properly (_code_mask / _strip_ts_comments); the
honest move is to lift it rather than re-hand-roll it here.
There is a false-RED risk that needs deciding first. A perfectly good spec
can look like this:
/** Baseline for PersonalActivityView. */
test('my activity', async ({ page }) => {
await shootSurface(page, `${APP}/#/my-activity`, 'my-activity.png')
})The route slug is my-activity and the PNG is my-activity.png; the only place
the component's NAME appears is the docstring. Stripping comments turns that
red even though a real, running baseline exists. The remedy for the app is
one word (put the component name in the screenshot filename or the test title),
and arguably that is better practice — but it is churn, and two agents are
working gate-26 in other repos right now, so flipping the semantics
mid-flight would invalidate measurements in progress.
Hence: filed, measured, not merged. Happy to raise the PR on a go-ahead.
Related
- A second, independent way to a meaningless green in the same gate: the
tests/e2e/visual/directory may not be executed by the app's CI at all. On
doriath the CI config lists 56 tests in 14 files and the root config 65 in 16;
the CI config's only project carriestestIgnore: ['**/visual/**']. Baselines
added there turn gate-26 green with nothing running. Worth a note in the
finding text so apps route their coverage to the suite CI actually runs. git addmatters when planting a true positive here.added_files()reads
git diff, so an untracked plant is never seen and would "prove" the gate
blind when it is not. Verified on doriath: a tracked plant took it
PASS → FAIL — 1 → PASS.
Activity
fix(gate-26): cover the reachable page components, waive the unreacha…
test(e2e): remove 7 latent false @e2e anchors — they would have manuf…
rubenvdlinde commentedon Aug 11, 2026
Sequencing note from the gate-fixes wave (2026-08-11). Landed this session, smallest blast radius first, each with a fixture observed red-before / green-after: #366 (the gate-16 fixture #364 shipped without), #368 (gate-7
hasPermission/canAccess, #360), #370 (gate-24 both registration APIs, #349), #371 (gate-61 honest empty-scope reason, #347).This one was deliberately NOT landed in that wave. The per-issue reason is in the next comment / in
fleet-board/findings/github-gate-fixes.md; the common one is that every remaining item either changes what a gate counts rather than what it says, or lives inquality.ymlwhererun-helper-suites.shcannot reach it — and four gate changes in one session is already the limit at which the next fleet measurement stays attributable.