Skip to content

AGENTS.md #394 says the Playwright specs are not in CI; they have been since 2026-08-08 #767

Closed

Description

@mforce

What

AGENTS.md's #394 rule says the Playwright specs are not covered by CI. They have been since
2026-08-08. The k6 half of the same sentence is still correct, so the sentence is half-stale rather
than wrong, which is the harder kind to notice.

The current text, under Operations and callers:

A write-contract change must update its non-CI callers (#394). Coverage is not uniform: the
seeders are covered only to the handler layer, so a validator-only tightening is invisible to
every seeder test, and tools/simulation/k6/ plus the Playwright specs are uncovered while a green
baseline hides it. Verify the request/status contract by reading the callers, not by running.

The evidence

.github/workflows/e2e-smoke.yml runs on: pull_request, path-filtered to src/**, web/**,
tools/simulation/**, deploy/**, the three root build inputs and the workflow itself. Its own
header comment dates the change:

The quick suite runs ON PULL REQUESTS (owner call, 2026-08-08, revising the …)

and describes a ~3-minute run of 28 specs, excluding only the real 15-minute token-expiry spec and
canary. So any PR touching the app runs 28 of the specs across the 14 files in
tools/simulation/ui/specs/.

.github/workflows/k6-baseline.yml, by contrast, is workflow_dispatch: only. tools/simulation/k6/
genuinely is uncovered
, exactly as the rule says.

Observed rather than inferred: on #766 the check "Playwright smoke over the simulation fixture"
ran and passed, and that run included worker-sale-allocation.spec.ts, which carries no slow or
canary tag.

Why it is worth fixing

The error points the safe way, so nobody has shipped a defect because of it. The cost is different.
On #766 the Playwright half of the #394 caller read was done by hand under the belief that a green
baseline would hide a break, when CI would in fact have caught it. A rule in the canonical rule set
that misstates what CI covers spends reviewer effort on the wrong things, and it also makes the
genuinely-uncovered k6 case look like one item in a list of two rather than the single thing that
still needs reading by hand.

It is also a live trap in the other direction: someone who checks the workflow, sees Playwright is
covered, and concludes the whole bullet is stale would stop reading the k6 callers too.

Suggested change

Split the claim so each half says what is true, and name the mechanism rather than the state, since
the state has now moved once:

  • tools/simulation/k6/ is dispatch-only (k6-baseline.yml), so it is never exercised by a PR
    and must be read by hand.
  • tools/simulation/ui/ is exercised on every PR touching src/**, web/**,
    tools/simulation/** or deploy/** (e2e-smoke.yml), but not on a docs-only PR, and not for
    the slow and canary specs, which stay dispatch-only.

The seeder half of the rule is unaffected and still correct: DemoDataSeeder and
SimulationDataSeeder call handlers directly, so a validator-only tightening is still invisible to
them.

Worth checking docs/decisions/394-write-contract-callers.md for the same sentence in the same PR.

Not in scope

Whether the path filter should also cover a docs-only PR that changes a spec's expectations. It
should not, but that is a separate argument from this correction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions