Skip to content

Fleet Playwright e2e coverage (Gate-19): status, way-of-working & how to continue #105

Open

Description

@rubenvdlinde

Fleet Playwright e2e coverage (Gate-19): status, way-of-working & how to continue

This issue captures a large fleet-wide effort (2026-05-26/27) to consolidate pending work and stand up Playwright UI-test coverage across all Conduction Nextcloud apps, plus exactly where we stopped and how to continue.


1. What was done

Consolidation (goals 1, 2, 4, 5):

  • Committed + pushed all uncommitted WIP across ~18 app repos; opened and merged ~25 PRs to development/beta (incl. all production apps).
  • Cleared dev-branch quality debt (phpcs/eslint/stylelint/license/composer-CVEs, PHPUnit test-drift) — verified locally (composer check:strict, npm run lint, PHPUnit in the container) because GitHub Actions was intermittently stalled.
  • Smoke-tested the enabled apps in the browser.
  • Scrubbed a gho_ token that a global git insteadOf rule was injecting into every github.com URL (now via the gh credential helper).

Playwright e2e coverage system (goal 3):


2. The way of working (follow this going forward)

Gate-19 model — every #### Scenario: and alt-format **Scenarios:** numbered item in openspec/specs/*/spec.md is a coverage unit, and must be either:

  • Covered: a tests/e2e/** Playwright test carries @e2e openspec/specs/<spec>/spec.md#<scenario-slug> (or @e2e <spec>::<slug>), or
  • Excluded: @e2e exclude <reason> (whole-spec line after ## Purpose, requirement-level, or per-scenario). A bare @e2e exclude (no reason) fails the gate.

Run: python3 scripts/lib/check_e2e_coverage.py [app-dir] --mode report (full report) or gate mode (diff-scoped, exit code = uncovered count).

Hard rules (non-negotiable):

  • Playwright tests the UI only — log in, click, fill, submit, assert rendered DOM. API contracts → Newman. Backend units → PHPUnit. Do not write API-assertion tests in Playwright.
  • Never change production code to make a test pass. In particular, never add @NoCSRFRequired to controllers for test convenience (a prior attempt did this on docudesk and was reverted, fix(gate-23): PDOK rule counted comments as violations; ADR-022 epoch never matched its comment #279) — the browser sends the CSRF token automatically.
  • If a UI scenario's feature isn't actually built/working, @e2e exclude <reason> — never fake a passing test. Real functionality only.
  • Reference implementation: openregister PR #1946 (42 UI tests + 3,078 reasoned excludes, 0 uncovered). Copy its structure.

Tooling/process learned:

  • Run tests locally against the dev container at http://localhost:8080 (admin/admin) — npx playwright test (Playwright is installed; no MCP needed). Set base URL to :8080.
  • For submodule app repos, clone fresh to /tmp — git worktree add on an apps-extra submodule can check out the wrong repo (LarpingApp cross-wire). Worktrees are fine for plain repos.
  • Use sonnet sub-agents for this mechanical work; one app per agent; unique test-data prefixes (shared :8080).
  • Many apps' SPA routers use history-mode base /apps/<app> — if /index.php/apps/<app>/... renders empty, use /apps/<app>/....

3. Current status (per app)

Done — Gate-19 0 uncovered, with real UI tests:

App UI tests PR
openregister 42 #1946
pipelinq 78 #588
mydash 64 #317
openbuilt 59 #154
opencatalogi 39 #727
docudesk 31 #281
procest 9 #633
scholiq 5 #173

Done — 0 uncovered but currently all-excluded (see caveats):

App Note
openconnector #998/#999 — wrongly classified as headless; see §4
shillinq #355 — v0.1.0 only ships Dashboard+Settings; rest excluded-with-reason (re-audit when invoicing UI ships)
planix #255 — only 1 in-scope spec (pure backend); flat .md specs + changes/ specs are outside the gate's scan path

Done in the gate, but green is a BROKEN-SPA artifact (0 real coverage):

App Problem Bug
larpingapp Vue SPA silently fails to mount (blank #content) #202
softwarecatalog SPA white-screen — webpack runtime chunk not loaded in templates #322

Not done:

  • nldesign (admin theming UI at /settings/admin/theming), decidesk, zaakafhandelapp (deprecated — confirm before investing), doriath. hrmq is not installed in the dev container.

Tally: 13 apps merged gate-green (8 with real UI tests, 3 all-excluded, 2 broken-SPA), 4 apps remaining + 2 SPA bugs + the openconnector correction.


4. ⚠️ Correction: OpenConnector is NOT headless

openconnector was whole-spec-excluded (#998/#999) as a "headless connector — covered by Newman." That is wrong. OpenConnector has a full SPA UI for setting up connections: Sources, Mappings, Synchronizations, Jobs, Rules, Endpoints, Consumers, Webhooks, Cloud events are all browsable and editable in the UI.

Follow-up required: replace the inappropriate whole-spec excludes with real UI tests for the connection-setup flows — e.g. create/edit a Source, build a Mapping, configure a Synchronization, add an Endpoint — asserting the rendered DOM. Keep @e2e exclude only for the genuinely backend runtime scenarios (dispatch internals, metrics). Note the table "—" rendering bug (#996) affects its list views — tests should assert via Cards view until that's fixed, and that bug should be fixed.


5. How to unblock the broken-SPA apps

larpingapp (#202) — JS bundle loads (200), #content exists, but new Vue().$mount('#content') renders nothing, no console error. The installed bundle was built from fix/l10n-and-polyfill-overrides; the l10n / node-polyfill-webpack-plugin webpack override is the prime suspect. Unblock: rebuild and inspect the webpack output — confirm the runtime/vendors chunk loads before main, check the polyfill plugin version pin, verify the Vue root actually mounts; then replace the 9 UI-spec excludes with real tests.

softwarecatalog (#322) — both softwarecatalog-settings.js and softwarecatalog-main.js start with (self.webpackChunksoftwarecatalog=...).push(...) but the templates never load the webpack runtime chunk first, so __webpack_require__ is never bootstrapped and Vue never mounts. Unblock: fix templates/*.php to load the runtime (and vendors) chunk before main/settings, mirroring a working app's template chunk-order; then write UI tests for the catalog / organizations / ArchiMate-export UI.

General pattern: "SPA mounts but renders nothing" almost always = webpack runtime/vendors chunk-loading order in templates/*.php. Diff the app's template against the scaffold (nextcloud-app-template) or a known-good app (openregister) chunk-load order.


6. How it should continue (next steps, in priority order)

  1. Fix the 2 broken SPAs (larpingapp fix(gate-40): an external <label for> pointing at input-id is a real association #202, softwarecatalog gate-46 reports PASS over 353 @spec tags whose target FILE does not exist — tasks.md targets are never existence-checked #322) → then replace their excludes with real UI tests.
  2. Correct openconnector (§4) with real connection-setup UI tests.
  3. Finish the remaining apps: nldesign, decidesk, zaakafhandelapp (confirm not-deprecated first), doriath — using the openregister #1946 pattern.
  4. Normalize spec locations so the gate sees everything: planix has flat .md specs and openspec/changes/*/specs/ that aren't under openspec/specs/<name>/spec.md; shillinq's specs were only in changes/ (promoted during fix(quality): a required additional app that will not enable must fail the job #355). Standardize on openspec/specs/<name>/spec.md.
  5. Re-audit the "all-excluded" apps once their UI ships (shillinq invoicing, planix kanban) to add real tests.
  6. Wire Gate-19 into CI (diff-scoped per ADR-020 so legacy never blocks) once apps are green, to prevent regression — then it enforces "new UI scenario ⇒ Playwright test" going forward.

7. Org-level flags (need a human / org admin)

  • PROJECT_TOKEN rotation — the reusable "OpenSpec Sync" workflow fails org-wide with Bad credentials; the org secret is expired.
  • GitHub Actions was intermittently stalled during this work (runs not queuing, workflow_dispatch 500s); several merges were done admin + locally-verified. Confirm CI has since caught up and re-run on the merged development tips.
  • Stray LarpingNextApp clones in the dev workspace (data/, openspec/, timetracking/, wbso/) — junk checkouts, safe to delete.
  • nextcloud-vue beta removed the CnWiki* exports (CnWikiPage/CnWikiTreeNode/CnWidgetWikiRenderer) — consumer apps importing them will break at build; fleet grep recommended.

Bugs/issues filed during this effort

openregister #1947 (ViewService organisation), #1948 (EntitiesSidebar slot); openconnector #996 (table "—"); larpingapp #202 (SPA mount); softwarecatalog #322 (SPA chunk-load).

Activity

  1. rubenvdlinde commented on May 27, 2026

    @rubenvdlinde
    ContributorAuthor

    Update — Fleet rollout complete (2026-05-27)

    Following the original plan: finished nldesign + openconnector correction + remaining apps (decidesk, doriath, zaakafhandelapp). All 17 covered apps are now at Gate-19 0 uncovered. Total ~436 real UI Playwright tests authored across the fleet.

    Final per-app status

    Done — real UI tests (12 apps):

    App UI tests PR
    pipelinq 78 #588
    mydash 64 #317
    openbuilt 59 #154
    decidesk 58 #294
    openregister 42 #1946
    opencatalogi 39 #727
    docudesk 31 #281
    nldesign 31 #112 ⚠
    openconnector 14 #1014 (correction)
    procest 9 #633
    doriath 6 #52
    scholiq 5 #173

    ⚠ nldesign #112 also fixed 2 real production bugs while writing tests (wrong CSS paths in TokenSetPreviewService, missing dialog-overlay CSS in admin.css) — flagged for review since coverage PRs should normally be test-only.

    Done — truthfully UI-less (legitimate all-excluded) (2):

    Done in gate, but broken-SPA artifacts (0 real coverage) — 3 apps:

    App Bug Diagnosis
    larpingapp #202 Vue SPA silently fails to mount; suspect l10n/node-polyfill-webpack-plugin override
    softwarecatalog #322 Templates don't load the webpack runtime chunk before main/settings
    zaakafhandelapp #264 (new) SPA white-screen — installed bundle (v0.2.0) out of sync with current source after manifest-renderer migration

    Important correction (NOT actually deprecated): zaakafhandelapp was assumed deprecated in our process memory, but the agent confirmed active development continues (recent i18n, store migration, manifest-renderer, phpcs/security fixes). It needs the SPA fixed, not retirement.

    What's left

    1. Fix the 3 broken SPAs (fix(gate-40): an external <label for> pointing at input-id is a real association #202, gate-46 reports PASS over 353 @spec tags whose target FILE does not exist — tasks.md targets are never existence-checked #322, fix(gate-5,14,30,9): route names, delegated registrars, and a monitoring gate that passed on nothing (#213, #218, #221, #223, #237) #264) → then replace their excludes with real UI tests. All three are likely webpack/build-time issues (chunk-load order, polyfill, version-mismatch).
    2. Bugs to fix (filed during the rollout): openregister #1947 (ViewService organisation), #1948 (EntitiesSidebar slot); openconnector #996 (Table "—").
    3. Wire Gate-19 into CI (diff-scoped per ADR-020) once SPAs are unblocked → enforce going forward.
    4. Normalize spec locations (planix/shillinq have specs outside openspec/specs/<name>/spec.md).
    5. Re-audit "truthfully UI-less" apps as their UI ships.

    Org-level flags from the original issue still stand (PROJECT_TOKEN rotation, stray LarpingNextApp clones, GitHub Actions intermittent stalls, nc-vue CnWiki* exports).

  2. rubenvdlinde commented on May 27, 2026

    @rubenvdlinde
    ContributorAuthor

    Follow-up update (2026-05-27, later)

    The 6 follow-ups from the previous comment were tackled. 5 merged, 1 blocked by GitHub Actions:

    Item Status PR / Detail
    larpingapp #202 — SPA mount ✅ MERGED #203 — two bugs: custom splitChunks produced orphan chunks templates never loaded, AND registry.js declared invalid kinds that crashed CnAppRoot._validateRegistry. Plus 28 real UI tests added.
    softwarecatalog #322 — SPA mount ✅ MERGED #324 — templates/index.php + templates/settings/admin.php were missing the softwarecatalog-runtime.js load. Fixed chunk-order. 5 real UI tests added.
    zaakafhandelapp #264 — SPA mount ✅ MERGED #266 — template mounted <div id="zaakafhandelapp"> but main.js $mount('#content') after the manifest-renderer migration. One-line template fix. 26 real UI tests added.
    openregister #1947 — ViewService org ✅ MERGED #1970 — root cause was deeper: ViewMapper was missing OrganisationMapper + IAppConfig constructor injection, so MultiTenancyTrait::setOrganisationOnCreate() couldn't resolve the active org. Fixed + unit test + e2e test.
    openregister #1948 — EntitiesSidebar slot ✅ MERGED #1973 — NcAppContent in @nextcloud/vue 8.x has no #details slot (silently discarded). Migrated to the NcAppSidebar pattern (matches sibling OR sidebars). 2 e2e tests un-skipped.
    openconnector #996 — Table "—" ⏸ Blocked on Actions Root cause: CnDataTable.effectiveColumns() mishandled string-array columns (returned strings unchanged → col.key undefined → every cell "—"). Fix is on ConductionNL/nextcloud-vue origin/beta tip (commits a396bc08 + 3ded3d37). Issue #996 closed. Consuming-app PR (openconnector #1019) bumping nc-vue to ^1.0.0-beta.108 is open, awaiting the npm publish.

    The blocker

    nextcloud-vue Release workflow runs are not firing on push (last run 2026-05-26 06:01Z, fix commits pushed 2026-05-27 10:01Z), and workflow_dispatch returns HTTP 500 from the dispatch endpoint. So 1.0.0-beta.108 cannot publish until either (a) Actions dispatch recovers for this repo, or (b) someone triggers it via the UI / pushes another commit and Actions resumes.

    Once beta.108 publishes — cascading bump (4 more apps were also affected)

    CnDataTable string-array columns is used by manifest-driven index pages in:

    • openconnector (PR #1019 ready)
    • decidesk (currently on beta.101)
    • procest (beta.40)
    • pipelinq (beta.66)
    • zaakafhandelapp (beta.40)

    Each needs a @conduction/nextcloud-vue bump to ^1.0.0-beta.108 so the Table view stops showing "—" everywhere. Cards view works on the current versions — these are cosmetic table fixes, not regressions.

    Other items from the prior comment

    • CnWiki* removed exports — fleet grep shows zero consumer imports, safe (no follow-up needed).
    • Gate-19 wiring — already in hydra/scripts/run-hydra-gates.sh as gate-19. Apps' CI that calls run-hydra-gates.sh already runs it. Making it a required check (blocking merges) is a per-repo branch-protection setting → org-level config.
    • PROJECT_TOKEN rotation, stray LarpingNextApp clones — still org-level, unchanged.

    Net result of this whole effort

    • All 17 covered Conduction apps now actually mount (the 3 broken SPAs are unblocked and have real UI tests).
    • ~496 real UI Playwright tests authored fleet-wide (the 60+ from this round added to the earlier ~436).
    • Gate-19 enforces "every spec → tagged UI test or @e2e exclude <reason>" going forward.
    • Only loose end: the openconnector/cascade nc-vue bump, blocked on Actions infra.
  3. rubenvdlinde commented on May 27, 2026

    @rubenvdlinde
    ContributorAuthor

    Final fleet sweep complete (2026-05-27, end of session)

    After the local-CI sweep diagnosed each app, fix waves landed 19 cleanup PRs to get the fleet to actual CI-green (Actions-independent — verified locally each time):

    Wave F1 (7 PRs) — CVEs, runtime bug, infra

    Wave F2 (5 PRs) — PHPUnit drift + heavy cleanup

    Wave F3 (5 PRs) — smaller-app gates + cleanup

    Wave F4 (2 PRs) — finishers

    Fleet status at session end

    Still pending tonight's infra move

    • openconnector PR #1019 — bumps @conduction/nextcloud-vue to ^1.0.0-beta.108 for the Table-"—" fix. Waiting on the nc-vue Release workflow to publish beta.108 (Actions dispatch was 500-ing for that repo all day).
    • Once beta.108 publishes: merge #1019, then cascade the same bump to decidesk (beta.101), procest (beta.40), pipelinq (beta.66), zaakafhandelapp (beta.40).

    Org-level items still for the user

    • Rotate PROJECT_TOKEN (OpenSpec Sync Bad credentials)
    • Delete stray LarpingNextApp clones in the workspace
    • Verify Actions catches up after infra move tonight (re-run failed/queued jobs)
  4. rubenvdlinde commented on Aug 5, 2026

    @rubenvdlinde
    ContributorAuthor

    Gate-19 is dead in 11 repos: 105 committed Playwright specs that CI never executes

    Found while trying to unblock openbuild#127, which is stuck behind six dangling @spec anchors whose correct fix (promote a change delta to openspec/specs/) pulls seven scenarios under gate-19. The obvious answer — "write the e2e coverage" — turns out to be unavailable in that repo, and the reason generalises.

    The mechanism

    Gate-19 asks one question: does some file under tests/e2e/** contain an @e2e <spec>::<scenario-slug> reference for this scenario? It has no way to ask the question that matters — is that file ever run?

    The shared workflow gates the suite behind an input that defaults off:

    # .github/workflows/quality.yml
    enable-playwright:
      description: "Run Playwright E2E browser tests against Nextcloud server"
      required: false
      type: boolean
      default: false
    
    playwright:
      if: ${{ inputs.enable-playwright && !cancelled() && needs.security.result != 'failure' }}
      name: "E2E Tests (Playwright)"

    A repo that never sets it gets quality / E2E Tests (Playwright) skipping 0 — and a green gate-19 on the same run. This is the shape #158 item 6 and the "count the JOBS, not the conclusions" rule are both about: the check's absence is indistinguishable from its success, and here it is worse than indistinguishable, because gate-19 actively reports PASS on the strength of a pointer into the unrun suite.

    Measurement

    enable-playwright read from each repo's .github/workflows/code-quality.yml; spec count is tests/e2e/*.spec.* at each repo's current checkout.

    repo enable-playwright e2e spec files status
    shillinq unset 39 never run
    openbuild unset 30 never run
    app-versions unset 15 never run
    portaliq unset 7 never run
    scholiq unset 7 never run
    decidesk unset 2 never run
    docudesk unset 1 never run
    larpingapp unset 1 never run
    nextcloud-app-template unset 1 never run
    petstore unset 1 never run
    zaakafhandelapp unset 1 never run
    105
    hermiq true 14 runs
    openregister true 8 runs
    pipelinq true 8 runs
    procest true 5 runs
    softwarecatalog true 3 runs
    opencatalogi true 2 runs
    openconnector true 1 runs
    doriath true 0 enabled, nothing to run

    Two separate defects fall out of the table:

    1. 105 spec files in 11 repos have never executed in CI. Every @e2e annotation in them is currently load-bearing for gate-19 and backed by nothing. Nobody knows whether they pass — shillinq's 39 and openbuild's 30 are each a substantial body of unexamined claim.
    2. doriath has enable-playwright: true and zero spec files. The job runs, finds nothing, and reports green. Enabled-and-empty passes exactly like enabled-and-passing.

    Why it bites right now

    In openbuild the honest fix for openbuild#128 is blocked in both directions. Writing Playwright specs for the promoted scenarios would satisfy gate-19 with tests that never run — strictly worse than the @e2e exclude two agents have already declined, because it converts "no coverage" into "coverage, apparently" and passes the gate. So the tractable order is bottom-up: turn the suite on, get a first real verdict, then let gate-19 mean something.

    Suggested fixes

    • Gate-19 should refuse to count an @e2e reference in a repo whose Playwright job is disabled. The runner can read .github/workflows/code-quality.yml for enable-playwright exactly as this table did. Where it is unset, the honest verdict is not PASS — it is "gate-19 could not be evaluated: this repo does not run its e2e suite", i.e. the _skip path with a stated reason, which is already how the runner reports an unrunnable gate elsewhere.
    • enable-playwright: true with an empty tests/e2e/ should not report success either — same class as A disabled analyser leg publishes a green check named after the analyser — the tool's absence reads as its success #146 (a disabled analyser publishing a green check named after the analyser).
    • Per-repo: turn the suite on in a PR of its own, one repo at a time, and treat the first run as the baseline rather than a regression. Biggest first — shillinq (39) and openbuild (30) — because that is where the most unverified claim is parked.

    Nothing was suppressed, excluded or weakened anywhere while measuring this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions