Repository navigation
Description
The mechanism
scripts/lib/check_e2e_coverage.py, _ref_is_live():
def _ref_is_live(doc: _TestFile, pos: int) -> bool:
node = doc.owner(pos)
if node is None:
# No declaration owns this tag — a file-level annotation. Live: this
# function exists to catch tests that were switched OFF, not to invent
# a structural requirement the gate never had.
return TrueA @e2e tag that sits outside every test() / test.describe() declaration — i.e. in a file header comment, above the first declaration — has no owner, so the function returns True without inspecting a single test body. Every other path in that function checks something: switched_off on the node or an ancestor, body_is_empty(node), has_own_unconditional_skip. The no-owner path checks nothing.
So an arbitrary number of scenarios can be marked covered by putting their tags in a header block. The file only has to be one a Playwright project runs, and it does not matter what the tests in it assert — or whether any test in the file relates to those scenarios at all.
This is not hypothetical and it is not a corner case: a header block is a natural way to write a file's provenance, and it reads exactly like documentation.
Measured
launchpad, development, hydra-gates@b8c7ead. Two files each opened with a block of @e2e tags above the first test():
tests/e2e/image-widget.spec.ts— 8 tags over 3 teststests/e2e/label-widget.spec.ts— 4 tags over 3 tests
Of those twelve, six were proven by nothing. No test touched the fit select, forced an upload error, read the cell cursor, or used a <script> tag; the "form pre-fills all six fields" tag sat above a test asserting one field. Gate-19 counted all twelve as covered.
Those two files happened to be excluded from the running Playwright config, so the gate reported them under its other rule ("referenced only by a file no Playwright project runs") — which is what made them visible at all. Had they been in the running config, gate-19 would have reported PASS on six scenarios with no assertion behind them, and nothing would have surfaced it. That is the shape worth fixing: the finding that exposed this was incidental.
They are now fixed in launchpad (each tag moved onto the test that proves it, real tests written for the six unproven scenarios) — but the gate would accept the broken form again tomorrow, in any repo.
Why this matters more than a normal false negative
Gate-19 exists so that a scenario cannot be declared covered without a test. The no-owner path is a hole straight through that guarantee, and it is cheaper to write than the correct form — one block at the top of a file is less effort than N tags placed individually. A gate whose easiest usage is also its unsound usage will drift toward the unsound usage on its own.
It also interacts badly with the fleet's current push: several repos are clearing large gate-19 backlogs right now (pipelinq alone had 369 findings). A header block is exactly the shortcut a bulk-annotation pass would reach for.
Suggested fix
Require a file-level tag to bind to something. The narrowest change that closes it:
if node is None:
# A file-level tag is only live if this file contains at least one
# test that is itself live — otherwise it asserts nothing.
return doc.has_any_live_test()That keeps the documented intent (the function catches switched-off tests; it does not impose structure) while refusing the degenerate case of a tag with no test behind it at all.
Stronger options, if you want them:
- Reject header blocks outright — require every
@e2etag to be owned by a declaration, and report an unowned tag as a finding with its own reason ("a file-level@e2etag is validated by nothing; move it onto the test that proves it"). This is the honest reading of what the tag means. - Cap them: allow at most one unowned tag per file, so a block of N cannot ride on one file.
Option 1 is a behaviour change that will produce new findings in repos using header blocks today, so it wants a fleet sweep first — grep -rn for @e2e occurrences preceding the first test( in each tests/e2e/** file would size it.
Positive control, for whoever fixes this
Put a header block of 3 @e2e tags naming real scenarios into a file whose tests assert nothing relevant, in a repo where the file is run by the config. The gate should report 3 findings; today it reports 0. That is the test the fix needs, and it is the one this issue is really asking for — the current suite cannot distinguish the two cases.
Activity
test(e2e): remove 7 latent false @e2e anchors — they would have manuf…