Repository navigation
Description
Fleet Playwright e2e coverage (Gate-19): status, way-of-working & how to continue
This issue captures a large fleet-wide effort (2026-05-26/27) to consolidate pending work and stand up Playwright UI-test coverage across all Conduction Nextcloud apps, plus exactly where we stopped and how to continue.
1. What was done
Consolidation (goals 1, 2, 4, 5):
- Committed + pushed all uncommitted WIP across ~18 app repos; opened and merged ~25 PRs to
development/beta(incl. all production apps). - Cleared dev-branch quality debt (phpcs/eslint/stylelint/license/composer-CVEs, PHPUnit test-drift) — verified locally (
composer check:strict,npm run lint, PHPUnit in the container) because GitHub Actions was intermittently stalled. - Smoke-tested the enabled apps in the browser.
- Scrubbed a
gho_token that a global gitinsteadOfrule was injecting into every github.com URL (now via theghcredential helper).
Playwright e2e coverage system (goal 3):
- Built Gate-19 (
hydra/scripts/lib/check_e2e_coverage.py, skillhydra-gate-e2e-coverage, PRs hydra#347 + fix(gate-24): the JS probe knew one of two registration APIs, so it cleared repos that have leaves #349) — a mechanical gate that measures spec→Playwright coverage.
2. The way of working (follow this going forward)
Gate-19 model — every #### Scenario: and alt-format **Scenarios:** numbered item in openspec/specs/*/spec.md is a coverage unit, and must be either:
- Covered: a
tests/e2e/**Playwright test carries@e2e openspec/specs/<spec>/spec.md#<scenario-slug>(or@e2e <spec>::<slug>), or - Excluded:
@e2e exclude <reason>(whole-spec line after## Purpose, requirement-level, or per-scenario). A bare@e2e exclude(no reason) fails the gate.
Run: python3 scripts/lib/check_e2e_coverage.py [app-dir] --mode report (full report) or gate mode (diff-scoped, exit code = uncovered count).
Hard rules (non-negotiable):
- Playwright tests the UI only — log in, click, fill, submit, assert rendered DOM. API contracts → Newman. Backend units → PHPUnit. Do not write API-assertion tests in Playwright.
- Never change production code to make a test pass. In particular, never add
@NoCSRFRequiredto controllers for test convenience (a prior attempt did this on docudesk and was reverted, fix(gate-23): PDOK rule counted comments as violations; ADR-022 epoch never matched its comment #279) — the browser sends the CSRF token automatically. - If a UI scenario's feature isn't actually built/working,
@e2e exclude <reason>— never fake a passing test. Real functionality only. - Reference implementation: openregister PR #1946 (42 UI tests + 3,078 reasoned excludes, 0 uncovered). Copy its structure.
Tooling/process learned:
- Run tests locally against the dev container at
http://localhost:8080(admin/admin) —npx playwright test(Playwright is installed; no MCP needed). Set base URL to :8080. - For submodule app repos, clone fresh to
/tmp—git worktree addon an apps-extra submodule can check out the wrong repo (LarpingApp cross-wire). Worktrees are fine for plain repos. - Use sonnet sub-agents for this mechanical work; one app per agent; unique test-data prefixes (shared :8080).
- Many apps' SPA routers use history-mode base
/apps/<app>— if/index.php/apps/<app>/...renders empty, use/apps/<app>/....
3. Current status (per app)
Done — Gate-19 0 uncovered, with real UI tests:
| App | UI tests | PR |
|---|---|---|
| openregister | 42 | #1946 |
| pipelinq | 78 | #588 |
| mydash | 64 | #317 |
| openbuilt | 59 | #154 |
| opencatalogi | 39 | #727 |
| docudesk | 31 | #281 |
| procest | 9 | #633 |
| scholiq | 5 | #173 |
Done — 0 uncovered but currently all-excluded (see caveats):
| App | Note |
|---|---|
| openconnector | #998/#999 — wrongly classified as headless; see §4 |
| shillinq | #355 — v0.1.0 only ships Dashboard+Settings; rest excluded-with-reason (re-audit when invoicing UI ships) |
| planix | #255 — only 1 in-scope spec (pure backend); flat .md specs + changes/ specs are outside the gate's scan path |
Done in the gate, but green is a BROKEN-SPA artifact (0 real coverage):
| App | Problem | Bug |
|---|---|---|
| larpingapp | Vue SPA silently fails to mount (blank #content) |
#202 |
| softwarecatalog | SPA white-screen — webpack runtime chunk not loaded in templates | #322 |
Not done:
- nldesign (admin theming UI at
/settings/admin/theming), decidesk, zaakafhandelapp (deprecated — confirm before investing), doriath. hrmq is not installed in the dev container.
Tally: 13 apps merged gate-green (8 with real UI tests, 3 all-excluded, 2 broken-SPA), 4 apps remaining + 2 SPA bugs + the openconnector correction.
4. ⚠️ Correction: OpenConnector is NOT headless
openconnector was whole-spec-excluded (#998/#999) as a "headless connector — covered by Newman." That is wrong. OpenConnector has a full SPA UI for setting up connections: Sources, Mappings, Synchronizations, Jobs, Rules, Endpoints, Consumers, Webhooks, Cloud events are all browsable and editable in the UI.
Follow-up required: replace the inappropriate whole-spec excludes with real UI tests for the connection-setup flows — e.g. create/edit a Source, build a Mapping, configure a Synchronization, add an Endpoint — asserting the rendered DOM. Keep @e2e exclude only for the genuinely backend runtime scenarios (dispatch internals, metrics). Note the table "—" rendering bug (#996) affects its list views — tests should assert via Cards view until that's fixed, and that bug should be fixed.
5. How to unblock the broken-SPA apps
larpingapp (#202) — JS bundle loads (200), #content exists, but new Vue().$mount('#content') renders nothing, no console error. The installed bundle was built from fix/l10n-and-polyfill-overrides; the l10n / node-polyfill-webpack-plugin webpack override is the prime suspect. Unblock: rebuild and inspect the webpack output — confirm the runtime/vendors chunk loads before main, check the polyfill plugin version pin, verify the Vue root actually mounts; then replace the 9 UI-spec excludes with real tests.
softwarecatalog (#322) — both softwarecatalog-settings.js and softwarecatalog-main.js start with (self.webpackChunksoftwarecatalog=...).push(...) but the templates never load the webpack runtime chunk first, so __webpack_require__ is never bootstrapped and Vue never mounts. Unblock: fix templates/*.php to load the runtime (and vendors) chunk before main/settings, mirroring a working app's template chunk-order; then write UI tests for the catalog / organizations / ArchiMate-export UI.
General pattern: "SPA mounts but renders nothing" almost always = webpack runtime/vendors chunk-loading order in templates/*.php. Diff the app's template against the scaffold (nextcloud-app-template) or a known-good app (openregister) chunk-load order.
6. How it should continue (next steps, in priority order)
- Fix the 2 broken SPAs (larpingapp fix(gate-40): an external <label for> pointing at input-id is a real association #202, softwarecatalog gate-46 reports PASS over 353 @spec tags whose target FILE does not exist — tasks.md targets are never existence-checked #322) → then replace their excludes with real UI tests.
- Correct openconnector (§4) with real connection-setup UI tests.
- Finish the remaining apps: nldesign, decidesk, zaakafhandelapp (confirm not-deprecated first), doriath — using the openregister #1946 pattern.
- Normalize spec locations so the gate sees everything: planix has flat
.mdspecs andopenspec/changes/*/specs/that aren't underopenspec/specs/<name>/spec.md; shillinq's specs were only inchanges/(promoted during fix(quality): a required additional app that will not enable must fail the job #355). Standardize onopenspec/specs/<name>/spec.md. - Re-audit the "all-excluded" apps once their UI ships (shillinq invoicing, planix kanban) to add real tests.
- Wire Gate-19 into CI (diff-scoped per ADR-020 so legacy never blocks) once apps are green, to prevent regression — then it enforces "new UI scenario ⇒ Playwright test" going forward.
7. Org-level flags (need a human / org admin)
PROJECT_TOKENrotation — the reusable "OpenSpec Sync" workflow fails org-wide withBad credentials; the org secret is expired.- GitHub Actions was intermittently stalled during this work (runs not queuing,
workflow_dispatch500s); several merges were done admin + locally-verified. Confirm CI has since caught up and re-run on the mergeddevelopmenttips. - Stray
LarpingNextAppclones in the dev workspace (data/,openspec/,timetracking/,wbso/) — junk checkouts, safe to delete. - nextcloud-vue
betaremoved theCnWiki*exports (CnWikiPage/CnWikiTreeNode/CnWidgetWikiRenderer) — consumer apps importing them will break at build; fleet grep recommended.
Bugs/issues filed during this effort
openregister #1947 (ViewService organisation), #1948 (EntitiesSidebar slot); openconnector #996 (table "—"); larpingapp #202 (SPA mount); softwarecatalog #322 (SPA chunk-load).
Activity
rubenvdlinde commentedon May 27, 2026
Update — Fleet rollout complete (2026-05-27)
Following the original plan: finished nldesign + openconnector correction + remaining apps (decidesk, doriath, zaakafhandelapp). All 17 covered apps are now at Gate-19
0 uncovered. Total ~436 real UI Playwright tests authored across the fleet.Final per-app status
Done — real UI tests (12 apps):
⚠ nldesign #112 also fixed 2 real production bugs while writing tests (wrong CSS paths in
TokenSetPreviewService, missing dialog-overlay CSS inadmin.css) — flagged for review since coverage PRs should normally be test-only.Done — truthfully UI-less (legitimate all-excluded) (2):
register-schemas); flat.mdspecs +changes/specs outside the gate's scan path (normalize spec locations as follow-up).Done in gate, but broken-SPA artifacts (0 real coverage) — 3 apps:
node-polyfill-webpack-pluginoverrideImportant correction (NOT actually deprecated): zaakafhandelapp was assumed deprecated in our process memory, but the agent confirmed active development continues (recent i18n, store migration, manifest-renderer, phpcs/security fixes). It needs the SPA fixed, not retirement.
What's left
openspec/specs/<name>/spec.md).Org-level flags from the original issue still stand (PROJECT_TOKEN rotation, stray LarpingNextApp clones, GitHub Actions intermittent stalls, nc-vue CnWiki* exports).
rubenvdlinde commentedon May 27, 2026
Follow-up update (2026-05-27, later)
The 6 follow-ups from the previous comment were tackled. 5 merged, 1 blocked by GitHub Actions:
splitChunksproduced orphan chunks templates never loaded, ANDregistry.jsdeclared invalidkinds that crashedCnAppRoot._validateRegistry. Plus 28 real UI tests added.templates/index.php+templates/settings/admin.phpwere missing thesoftwarecatalog-runtime.jsload. Fixed chunk-order. 5 real UI tests added.<div id="zaakafhandelapp">butmain.js$mount('#content')after the manifest-renderer migration. One-line template fix. 26 real UI tests added.ViewMapperwas missingOrganisationMapper+IAppConfigconstructor injection, soMultiTenancyTrait::setOrganisationOnCreate()couldn't resolve the active org. Fixed + unit test + e2e test.NcAppContentin@nextcloud/vue8.x has no#detailsslot (silently discarded). Migrated to theNcAppSidebarpattern (matches sibling OR sidebars). 2 e2e tests un-skipped.CnDataTable.effectiveColumns()mishandled string-arraycolumns(returned strings unchanged →col.keyundefined → every cell "—"). Fix is onConductionNL/nextcloud-vueorigin/betatip (commitsa396bc08+3ded3d37). Issue #996 closed. Consuming-app PR (openconnector #1019) bumping nc-vue to^1.0.0-beta.108is open, awaiting the npm publish.The blocker
nextcloud-vueRelease workflow runs are not firing on push (last run 2026-05-26 06:01Z, fix commits pushed 2026-05-27 10:01Z), andworkflow_dispatchreturnsHTTP 500from the dispatch endpoint. So1.0.0-beta.108cannot publish until either (a) Actions dispatch recovers for this repo, or (b) someone triggers it via the UI / pushes another commit and Actions resumes.Once
beta.108publishes — cascading bump (4 more apps were also affected)CnDataTablestring-arraycolumnsis used by manifest-driven index pages in:openconnector(PR #1019 ready)decidesk(currently on beta.101)procest(beta.40)pipelinq(beta.66)zaakafhandelapp(beta.40)Each needs a
@conduction/nextcloud-vuebump to^1.0.0-beta.108so the Table view stops showing "—" everywhere. Cards view works on the current versions — these are cosmetic table fixes, not regressions.Other items from the prior comment
CnWiki*removed exports — fleet grep shows zero consumer imports, safe (no follow-up needed).hydra/scripts/run-hydra-gates.shas gate-19. Apps' CI that callsrun-hydra-gates.shalready runs it. Making it a required check (blocking merges) is a per-repo branch-protection setting → org-level config.Net result of this whole effort
@e2e exclude <reason>" going forward.rubenvdlinde commentedon May 27, 2026
Final fleet sweep complete (2026-05-27, end of session)
After the local-CI sweep diagnosed each app, fix waves landed 19 cleanup PRs to get the fleet to actual CI-green (Actions-independent — verified locally each time):
Wave F1 (7 PRs) — CVEs, runtime bug, infra
Application.php:411OrganizationContactSyncJobconstructor (missing$appManager+$logger— would crash at runtime)preflightedCorsexemption + 28 unit tests (fleet false-positive eliminated)15548a2d, all 19 gates green, no CVEsWave F2 (5 PRs) — PHPUnit drift + heavy cleanup
importref removed, PHPStan 12→0, PHPCS 103→0ComplaintSlaService::isOverdue/isOpenStatus+QueueService::isAtCapacity— domain logic, not mock-hacks)SettingsControlleris now#[AuthorizedAdminSetting]-only — test updated to match), PHPCS 5→0@specdocblock consolidation), PHPUnit 7→0, Psalm 14→0Wave F3 (5 PRs) — smaller-app gates + cleanup
??removed), PHPMD 19→11Wave F4 (2 PRs) — finishers
isWidgetAllowedimplementation, REQ-TMPL-013 grep-guard cleanup), Psalm 3→0#[AuthorizedAdminSetting]on 15 endpoints, routes wired for OverridesController), PHPCS 25→0, PHPMD 7→0, Psalm 11→0, stylelint 1→0Fleet status at session end
Still pending tonight's infra move
@conduction/nextcloud-vueto^1.0.0-beta.108for the Table-"—" fix. Waiting on the nc-vue Release workflow to publish beta.108 (Actions dispatch was 500-ing for that repo all day).beta.108publishes: merge #1019, then cascade the same bump to decidesk (beta.101), procest (beta.40), pipelinq (beta.66), zaakafhandelapp (beta.40).Org-level items still for the user
PROJECT_TOKEN(OpenSpec SyncBad credentials)LarpingNextAppclones in the workspacerubenvdlinde commentedon Aug 5, 2026
Gate-19 is dead in 11 repos: 105 committed Playwright specs that CI never executes
Found while trying to unblock openbuild#127, which is stuck behind six dangling
@specanchors whose correct fix (promote a change delta toopenspec/specs/) pulls seven scenarios under gate-19. The obvious answer — "write the e2e coverage" — turns out to be unavailable in that repo, and the reason generalises.The mechanism
Gate-19 asks one question: does some file under
tests/e2e/**contain an@e2e <spec>::<scenario-slug>reference for this scenario? It has no way to ask the question that matters — is that file ever run?The shared workflow gates the suite behind an input that defaults off:
A repo that never sets it gets
quality / E2E Tests (Playwright) skipping 0— and a green gate-19 on the same run. This is the shape #158 item 6 and the "count the JOBS, not the conclusions" rule are both about: the check's absence is indistinguishable from its success, and here it is worse than indistinguishable, because gate-19 actively reports PASS on the strength of a pointer into the unrun suite.Measurement
enable-playwrightread from each repo's.github/workflows/code-quality.yml; spec count istests/e2e/*.spec.*at each repo's current checkout.enable-playwrighttruetruetruetruetruetruetruetrueTwo separate defects fall out of the table:
@e2eannotation in them is currently load-bearing for gate-19 and backed by nothing. Nobody knows whether they pass — shillinq's 39 and openbuild's 30 are each a substantial body of unexamined claim.enable-playwright: trueand zero spec files. The job runs, finds nothing, and reports green. Enabled-and-empty passes exactly like enabled-and-passing.Why it bites right now
In openbuild the honest fix for openbuild#128 is blocked in both directions. Writing Playwright specs for the promoted scenarios would satisfy gate-19 with tests that never run — strictly worse than the
@e2e excludetwo agents have already declined, because it converts "no coverage" into "coverage, apparently" and passes the gate. So the tractable order is bottom-up: turn the suite on, get a first real verdict, then let gate-19 mean something.Suggested fixes
@e2ereference in a repo whose Playwright job is disabled. The runner can read.github/workflows/code-quality.ymlforenable-playwrightexactly as this table did. Where it is unset, the honest verdict is not PASS — it is "gate-19 could not be evaluated: this repo does not run its e2e suite", i.e. the_skippath with a stated reason, which is already how the runner reports an unrunnable gate elsewhere.enable-playwright: truewith an emptytests/e2e/should not report success either — same class as A disabled analyser leg publishes a green check named after the analyser — the tool's absence reads as its success #146 (a disabled analyser publishing a green check named after the analyser).Nothing was suppressed, excluded or weakened anywhere while measuring this.