Repository navigation
Description
Summary
The stated goal is "everything tested through Playwright E2E". The wiring is done — 20 fleet repos ship a playwright.config.ts and run E2E Tests (Playwright) in the shared Code Quality workflow. Not one of them requires that job to pass before a merge.
A pull request whose E2E job is red merges into development today in every repo in the fleet, and into main and beta in all but three.
This is deliberate-looking enough that it needs a decision rather than a silent fix, so I am filing it instead of changing protection.
Measured
GET /repos/ConductionNL/<repo>/branches/development/protection, all 20 repos that ship a Playwright config:
| repo | required checks on development |
|---|---|
| nldesign | PHP Quality, CSS Quality, Branch Policy Check |
| pipelinq | PHP Quality, Frontend Quality, Branch Policy Check |
| procest | PHP Quality, Frontend Quality, Branch Policy Check |
| app-versions, decidesk, docudesk, doriath, hermiq, hrmq, larpingapp, nextcloud-app-template, openbuild, opencatalogi, openconnector, openregister, petstore, portaliq, scholiq, shillinq, softwarecatalog | branch not protected — no required checks at all |
- 0 of 20 require
E2E Tests (Playwright). - 17 of 20 have no branch protection on
developmentwhatsoever.
main and beta are the same picture. Spot-checked across the four repos currently being burned down:
| repo | main |
beta |
|---|---|---|
| pipelinq | PHP Quality, Frontend Quality, Branch Policy Check | same |
| nldesign | PHP Quality, CSS Quality, Branch Policy Check | same |
| decidesk | not protected | not protected |
| openbuild | not protected | not protected |
Where protection does exist it requires the static-analysis jobs and the branch-policy check, and never E2E.
Why this matters right now
Four repos are actively red on E2E as of 2026-08-06 development:
| repo | passed | failed | skipped | did not run |
|---|---|---|---|---|
| decidesk | 56 | 53 | 62 | 0 |
| pipelinq | 140 | 31 | 40 | 1 |
| openbuild | 168 | 5 | 76 | 0 |
| nldesign | 97 | 2 | 1 | 10 |
Those failures reached development because nothing stopped them. The suites are doing their job — they are finding real defects — but the finding has no consequence, so the red state persists and accumulates. A test suite that cannot block a merge is a report, not a gate.
The decision, stated honestly
Turning E2E into a required check today would block every merge in four repos, because those suites are genuinely red. That is not a reason to skip the decision, but it does mean sequencing matters:
- Burn the suites to green (in progress).
- Then make
E2E Tests (Playwright)a required status check ondevelopment, per repo, as each one reaches green — so no repo is blocked on another repo's debt. - Promote to
main/betaoncedevelopmenthas held green for a while.
Open questions that need a human:
- Is the absence of protection on 17 repos intentional? The fleet runs agent-driven merges, and
--adminbypasses required checks anyway, so protection may have been dropped on purpose to keep automation unblocked. If so, the honest fix is different: keep protection off, but make a red E2E block the automation rather than the branch. - Should the skipped counts gate too? decidesk is skipping 62 of 171 and openbuild 76 of 249. A suite can go "green" by skipping everything, so a required check on failures alone does not deliver the stated goal. A skip ratio ceiling may be the more meaningful gate.
did not runis a third state. nldesign left 10 tests with no verdict. Whatever gate is chosen must treat "not measured" as not-passing.
Related
- Playwright instrument: 2 repos write zero traces (retries:0 + on-first-retry), and no repo in the fleet sets globalTimeout #188 — Playwright trace and
globalTimeoutinstrument defects (2 repos write zero traces; no repo bounds the run under the 45-minute job timeout, so a cancelled run leaves no artifacts and no tally). - playwright-coverage-threshold has never gated: warning-only, not line coverage, and zero scenarios scores 100% #189 —
playwright-coverage-thresholdhas never gated: warning-only, not line coverage, and zero scenarios scores 100%.
Together those three are the same shape: the E2E stack is fully built and produces no enforceable consequence at any layer — not the coverage threshold, not the trace artifacts, not the merge.
Activity
fix(ci): Frontend Tests (unit) sat OUTSIDE the fleet's only required …
rubenvdlinde commentedon Aug 8, 2026
Re-measured before acting, and two of this issue's three claims are stale. Recording the measurement rather than the conclusion.
Current state, all 25 fleet repos, 2026-08-08
mainMain Branch Protection, activebranch-protection / check-branch,quality / Quality ReportbetaBeta Branch Protection, activebranch-protection / check-branch,quality / Quality ReportdevelopmentDevelopment Branch Protection, activeUniform across all 25. "17 have no branch protection at all" is false — the legacy
/branches/*/protectionAPI reads mostly empty because the gating moved to rulesets, which is where the fleet's protection actually lives.developmentstays unprotected — by designProduct decision: we only block bad merges on
mainandbeta. Admin merges intodevelopmentare this team's routine, authorised workflow; requiring checks there would block the thing the team does every day. The existingDevelopment Branch Protectionruleset requires zero checks, which is already correct.Recording this so nobody "fixes" it later. A PR that adds required checks to
developmentis a regression, not an improvement.E2E already gates
mainandbetaquality / Quality Reportis not a pass-through — it carriesif: ${{ always() && contains(needs.*.result, 'failure') }}→exit 1. Verified on real runs rather than read off the YAML:The one apparent counter-example — pipelinq
31181387613, Playwright cancelled, Quality Report success — is not a hole. That run's head SHAce44766cwas superseded by47a1a590twenty-nine seconds later, and the cancelled-job guard deliberately stays quiet for a superseded run because the newer run carries the verdict. The newer runs on47a1a590did report failure.So "0 of 20 repos gate on Playwright" is true only of the literal context list. Functionally, a red E2E has been blocking
mainandbetain every repo all along.No explicit E2E context was added, deliberately
Requiring
quality / E2E Tests (Playwright)would duplicate a gate measured to work, and would introduce the failure mode this fleet has been bitten by three times: openklant, opentalk, openzaak and shillinq leaveenable-playwrightUNSET, so the job is SKIPPED there. A required check that does not report leaves PRs permanently pending — worse than no protection, because it looks like caution.Branch protection is outward-facing. A duplicate context that can wedge four repos' merge queues is not an improvement over a gate that already works. No ruleset in any repo was modified.
What this hunt did find — #229, merged
Quality Report'sneeds:named fifteen jobs. The workflow has sixteen.frontend-tests(Frontend Tests (unit)) was missing.Both gating steps read
needs.*.result, and a job absent fromneeds:is absent fromneeds.*. So a real unit suite with nocontinue-on-errorcould fail — or be cancelled mid-run — and the fleet's only meaningful required check stayed green. A failing frontend unit suite could not block a merge tomainorbetaanywhere.Latent rather than active, and stated as such: sampled across ten repos the job is green in every run, and a search for
Frontend Tests (unit) = failurealongsideQuality Report = successfound none. Nothing was waved through. A gate that would not catch a failure is still a dead gate.Fixed, plus
scripts/assert-quality-report-gates-every-leg.pyasserting every top-level job is inside that list — proven both ways (it fails againstorigin/main, namingfrontend-tests; it passes here; its positive control drops a job and must notice).Closing: the protection this issue asks for exists and is measured,
developmentis deliberately excluded, and the one real hole found along the way is fixed in #229.