Skip to content

E2E is not a required check anywhere: 0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all #190

Closed

Description

@rubenvdlinde

Summary

The stated goal is "everything tested through Playwright E2E". The wiring is done — 20 fleet repos ship a playwright.config.ts and run E2E Tests (Playwright) in the shared Code Quality workflow. Not one of them requires that job to pass before a merge.

A pull request whose E2E job is red merges into development today in every repo in the fleet, and into main and beta in all but three.

This is deliberate-looking enough that it needs a decision rather than a silent fix, so I am filing it instead of changing protection.

Measured

GET /repos/ConductionNL/<repo>/branches/development/protection, all 20 repos that ship a Playwright config:

repo required checks on development
nldesign PHP Quality, CSS Quality, Branch Policy Check
pipelinq PHP Quality, Frontend Quality, Branch Policy Check
procest PHP Quality, Frontend Quality, Branch Policy Check
app-versions, decidesk, docudesk, doriath, hermiq, hrmq, larpingapp, nextcloud-app-template, openbuild, opencatalogi, openconnector, openregister, petstore, portaliq, scholiq, shillinq, softwarecatalog branch not protected — no required checks at all
  • 0 of 20 require E2E Tests (Playwright).
  • 17 of 20 have no branch protection on development whatsoever.

main and beta are the same picture. Spot-checked across the four repos currently being burned down:

repo main beta
pipelinq PHP Quality, Frontend Quality, Branch Policy Check same
nldesign PHP Quality, CSS Quality, Branch Policy Check same
decidesk not protected not protected
openbuild not protected not protected

Where protection does exist it requires the static-analysis jobs and the branch-policy check, and never E2E.

Why this matters right now

Four repos are actively red on E2E as of 2026-08-06 development:

repo passed failed skipped did not run
decidesk 56 53 62 0
pipelinq 140 31 40 1
openbuild 168 5 76 0
nldesign 97 2 1 10

Those failures reached development because nothing stopped them. The suites are doing their job — they are finding real defects — but the finding has no consequence, so the red state persists and accumulates. A test suite that cannot block a merge is a report, not a gate.

The decision, stated honestly

Turning E2E into a required check today would block every merge in four repos, because those suites are genuinely red. That is not a reason to skip the decision, but it does mean sequencing matters:

  1. Burn the suites to green (in progress).
  2. Then make E2E Tests (Playwright) a required status check on development, per repo, as each one reaches green — so no repo is blocked on another repo's debt.
  3. Promote to main/beta once development has held green for a while.

Open questions that need a human:

  • Is the absence of protection on 17 repos intentional? The fleet runs agent-driven merges, and --admin bypasses required checks anyway, so protection may have been dropped on purpose to keep automation unblocked. If so, the honest fix is different: keep protection off, but make a red E2E block the automation rather than the branch.
  • Should the skipped counts gate too? decidesk is skipping 62 of 171 and openbuild 76 of 249. A suite can go "green" by skipping everything, so a required check on failures alone does not deliver the stated goal. A skip ratio ceiling may be the more meaningful gate.
  • did not run is a third state. nldesign left 10 tests with no verdict. Whatever gate is chosen must treat "not measured" as not-passing.

Related

Together those three are the same shape: the E2E stack is fully built and produces no enforceable consequence at any layer — not the coverage threshold, not the trace artifacts, not the merge.

Activity

  1. rubenvdlinde commented on Aug 8, 2026

    @rubenvdlinde
    ContributorAuthor

    Re-measured before acting, and two of this issue's three claims are stale. Recording the measurement rather than the conclusion.

    Current state, all 25 fleet repos, 2026-08-08

    branch ruleset required contexts
    main Main Branch Protection, active branch-protection / check-branch, quality / Quality Report
    beta Beta Branch Protection, active branch-protection / check-branch, quality / Quality Report
    development Development Branch Protection, active none

    Uniform across all 25. "17 have no branch protection at all" is false — the legacy /branches/*/protection API reads mostly empty because the gating moved to rulesets, which is where the fleet's protection actually lives.

    development stays unprotected — by design

    Product decision: we only block bad merges on main and beta. Admin merges into development are this team's routine, authorised workflow; requiring checks there would block the thing the team does every day. The existing Development Branch Protection ruleset requires zero checks, which is already correct.

    Recording this so nobody "fixes" it later. A PR that adds required checks to development is a regression, not an improvement.

    E2E already gates main and beta

    quality / Quality Report is not a pass-through — it carries if: ${{ always() && contains(needs.*.result, 'failure') }} → exit 1. Verified on real runs rather than read off the YAML:

    repo run Playwright Quality Report
    pipelinq 31181419092 failure failure
    scholiq 31164073861 failure failure
    nldesign 31252937998 failure failure

    The one apparent counter-example — pipelinq 31181387613, Playwright cancelled, Quality Report success — is not a hole. That run's head SHA ce44766c was superseded by 47a1a590 twenty-nine seconds later, and the cancelled-job guard deliberately stays quiet for a superseded run because the newer run carries the verdict. The newer runs on 47a1a590 did report failure.

    So "0 of 20 repos gate on Playwright" is true only of the literal context list. Functionally, a red E2E has been blocking main and beta in every repo all along.

    No explicit E2E context was added, deliberately

    Requiring quality / E2E Tests (Playwright) would duplicate a gate measured to work, and would introduce the failure mode this fleet has been bitten by three times: openklant, opentalk, openzaak and shillinq leave enable-playwright UNSET, so the job is SKIPPED there. A required check that does not report leaves PRs permanently pending — worse than no protection, because it looks like caution.

    Branch protection is outward-facing. A duplicate context that can wedge four repos' merge queues is not an improvement over a gate that already works. No ruleset in any repo was modified.

    What this hunt did find — #229, merged

    Quality Report's needs: named fifteen jobs. The workflow has sixteen. frontend-tests (Frontend Tests (unit)) was missing.

    Both gating steps read needs.*.result, and a job absent from needs: is absent from needs.*. So a real unit suite with no continue-on-error could fail — or be cancelled mid-run — and the fleet's only meaningful required check stayed green. A failing frontend unit suite could not block a merge to main or beta anywhere.

    Latent rather than active, and stated as such: sampled across ten repos the job is green in every run, and a search for Frontend Tests (unit) = failure alongside Quality Report = success found none. Nothing was waved through. A gate that would not catch a failure is still a dead gate.

    Fixed, plus scripts/assert-quality-report-gates-every-leg.py asserting every top-level job is inside that list — proven both ways (it fails against origin/main, naming frontend-tests; it passes here; its positive control drops a job and must notice).

    Closing: the protection this issue asks for exists and is measured, development is deliberately excluded, and the one real hole found along the way is fixed in #229.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions