Skip to content

gate-19: a file-level @e2e tag is counted as live coverage without checking any test — _ref_is_live returns True when no declaration owns it #343

Open

Description

@rubenvdlinde

The mechanism

scripts/lib/check_e2e_coverage.py, _ref_is_live():

def _ref_is_live(doc: _TestFile, pos: int) -> bool:
    node = doc.owner(pos)
    if node is None:
        # No declaration owns this tag — a file-level annotation. Live: this
        # function exists to catch tests that were switched OFF, not to invent
        # a structural requirement the gate never had.
        return True

A @e2e tag that sits outside every test() / test.describe() declaration — i.e. in a file header comment, above the first declaration — has no owner, so the function returns True without inspecting a single test body. Every other path in that function checks something: switched_off on the node or an ancestor, body_is_empty(node), has_own_unconditional_skip. The no-owner path checks nothing.

So an arbitrary number of scenarios can be marked covered by putting their tags in a header block. The file only has to be one a Playwright project runs, and it does not matter what the tests in it assert — or whether any test in the file relates to those scenarios at all.

This is not hypothetical and it is not a corner case: a header block is a natural way to write a file's provenance, and it reads exactly like documentation.

Measured

launchpad, development, hydra-gates@b8c7ead. Two files each opened with a block of @e2e tags above the first test():

  • tests/e2e/image-widget.spec.ts — 8 tags over 3 tests
  • tests/e2e/label-widget.spec.ts — 4 tags over 3 tests

Of those twelve, six were proven by nothing. No test touched the fit select, forced an upload error, read the cell cursor, or used a <script> tag; the "form pre-fills all six fields" tag sat above a test asserting one field. Gate-19 counted all twelve as covered.

Those two files happened to be excluded from the running Playwright config, so the gate reported them under its other rule ("referenced only by a file no Playwright project runs") — which is what made them visible at all. Had they been in the running config, gate-19 would have reported PASS on six scenarios with no assertion behind them, and nothing would have surfaced it. That is the shape worth fixing: the finding that exposed this was incidental.

They are now fixed in launchpad (each tag moved onto the test that proves it, real tests written for the six unproven scenarios) — but the gate would accept the broken form again tomorrow, in any repo.

Why this matters more than a normal false negative

Gate-19 exists so that a scenario cannot be declared covered without a test. The no-owner path is a hole straight through that guarantee, and it is cheaper to write than the correct form — one block at the top of a file is less effort than N tags placed individually. A gate whose easiest usage is also its unsound usage will drift toward the unsound usage on its own.

It also interacts badly with the fleet's current push: several repos are clearing large gate-19 backlogs right now (pipelinq alone had 369 findings). A header block is exactly the shortcut a bulk-annotation pass would reach for.

Suggested fix

Require a file-level tag to bind to something. The narrowest change that closes it:

if node is None:
    # A file-level tag is only live if this file contains at least one
    # test that is itself live — otherwise it asserts nothing.
    return doc.has_any_live_test()

That keeps the documented intent (the function catches switched-off tests; it does not impose structure) while refusing the degenerate case of a tag with no test behind it at all.

Stronger options, if you want them:

  1. Reject header blocks outright — require every @e2e tag to be owned by a declaration, and report an unowned tag as a finding with its own reason ("a file-level @e2e tag is validated by nothing; move it onto the test that proves it"). This is the honest reading of what the tag means.
  2. Cap them: allow at most one unowned tag per file, so a block of N cannot ride on one file.

Option 1 is a behaviour change that will produce new findings in repos using header blocks today, so it wants a fleet sweep first — grep -rn for @e2e occurrences preceding the first test( in each tests/e2e/** file would size it.

Positive control, for whoever fixes this

Put a header block of 3 @e2e tags naming real scenarios into a file whose tests assert nothing relevant, in a repo where the file is run by the config. The gate should report 3 findings; today it reports 0. That is the test the fix needs, and it is the one this issue is really asking for — the current suite cannot distinguish the two cases.

Activity

  1. added a commit that references this issue on Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions