From d0315a3e456df1d70e285924dae61f18c3b85651 Mon Sep 17 00:00:00 2001 From: Conduction Release Bot Date: Sat, 8 Aug 2026 12:58:48 +0200 Subject: [PATCH] fix(ci): Frontend Tests (unit) sat OUTSIDE the fleet's only required check (#190) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MEASURED FIRST — the issue's premise no longer holds #190 says "0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all". Re-measured across 25 repos on 2026-08-08, the second half is simply false and the first half is misleading: * ALL 25 repos have ACTIVE rulesets on main, beta AND development. * `Main Branch Protection` and `Beta Branch Protection` each require exactly two contexts: `branch-protection / check-branch` and `quality / Quality Report`. * `Development Branch Protection` requires ZERO checks. That is correct and deliberate — development is unprotected BY DESIGN, because admin merges into it are this team's routine workflow. Nothing here changes it. And E2E already gates main and beta, transitively. `Quality Report` is not a pass-through: it exits 1 on `contains(needs.*.result, 'failure')`. Verified on real runs rather than read off the YAML — pipelinq 31181419092 Playwright failure -> Quality Report failure scholiq 31164073861 Playwright failure -> Quality Report failure nldesign 31252937998 Playwright failure -> Quality Report failure The one counter-example, pipelinq 31181387613 (Playwright CANCELLED, Quality Report SUCCESS), is not a hole: that run's head SHA ce44766c was superseded by 47a1a590 twenty-nine seconds later, and the cancelled-job guard deliberately stays quiet for a superseded run because the newer run carries the verdict. The newer runs on 47a1a590 did report failure. So no new required context was added. Requiring `quality / E2E Tests (Playwright)` explicitly would be redundant with a gate measured to work, and it carries the risk the fleet has already been bitten by three times: four repos (openklant, opentalk, openzaak, shillinq) leave `enable-playwright` UNSET, so the job is SKIPPED there, and a required check that does not report leaves PRs permanently PENDING — worse than no protection because it looks like caution. WHAT WAS ACTUALLY BROKEN `Quality Report`'s `needs:` list named fifteen jobs. The workflow has sixteen. `frontend-tests` ("Frontend Tests (unit)") was not in it. Both of that job's gating steps read `needs.*.result`. A job absent from `needs:` is absent from `needs.*`, so `Frontend Tests (unit)` — a real unit suite with no `continue-on-error` — could fail, or be cancelled mid-run, and the only meaningful required check on main and beta across all 25 repos would stay green. A failing frontend unit suite could not block a merge to main or beta anywhere in the fleet. LATENT, NOT ACTIVE, AND SAID SO. Sampled across ten repos the job is currently green in every run (with some supersession cancellations), and a search for a run with `Frontend Tests (unit) = failure` alongside `Quality Report = success` found none. Nothing has been waved through yet. A gate that would not catch a failure is still a dead gate, and "it has not bitten yet" is not a verdict. THE FIX AND THE GUARD `frontend-tests` joins the list, and `scripts/assert-quality-report-gates-every-leg.py` now asserts that EVERY top-level job is either the report job or inside its `needs:`. An exemption requires an ALLOWLIST entry with a stated reason, so the list can only shrink and it says why. Proven in both directions, against the real document: * run against `origin/main` -> FAILS, naming `frontend-tests` * run against this commit -> passes * `--positive-control` drops a job from the list in memory and requires the check to notice; it names the job it dropped It runs as its own job in quality-resolve-probe.yml, behind the same single `Shared-workflow guard` required check, with its positive control first — and it lives in .github's own CI, so it cannot itself go permanently pending. BLAST RADIUS FOR CONSUMERS The fleet tracks @main unpinned (#177), so this reaches all callers on their next run. Repos where `enable-frontend` is false see a SKIPPED job, which is not 'failure' and changes nothing. Repos where the suite passes see nothing. Only a genuine frontend unit failure — or a frontend-tests job cancelled while its run carries on — newly blocks a merge to main or beta. That is the point. Refs #190 --- .github/workflows/quality-resolve-probe.yml | 27 ++- .github/workflows/quality.yml | 16 +- .../assert-quality-report-gates-every-leg.py | 162 ++++++++++++++++++ 3 files changed, 201 insertions(+), 4 deletions(-) create mode 100644 scripts/assert-quality-report-gates-every-leg.py diff --git a/.github/workflows/quality-resolve-probe.yml b/.github/workflows/quality-resolve-probe.yml index 8754e2a10..35bdbe121 100644 --- a/.github/workflows/quality-resolve-probe.yml +++ b/.github/workflows/quality-resolve-probe.yml @@ -76,6 +76,26 @@ jobs: fi echo "OK — the lint fails when it should. Its clean pass above is a verdict." + gate-completeness: + name: "Every leg is inside the required check" + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + # POSITIVE CONTROL FIRST. Drop a job from the report's `needs:` in + # memory and require the lint to notice. A lint that had stopped reading + # the list would pass the real check forever. + - name: "Positive control — removing a leg must be detected" + run: python3 scripts/assert-quality-report-gates-every-leg.py --positive-control .github/workflows/quality.yml + + # THE MEASUREMENT. `quality / Quality Report` is the only meaningful + # required check on main and beta in all 25 fleet repos, and its gating + # steps read `needs.*.result`. A job outside that list cannot block a + # merge however red it goes — which is what `frontend-tests` was (#190). + - name: "Assert every verdict-producing job is in Quality Report's needs" + run: python3 scripts/assert-quality-report-gates-every-leg.py .github/workflows/quality.yml + seed-semantics: name: "A failing seed fails the job" runs-on: ubuntu-latest @@ -181,20 +201,21 @@ jobs: guard: name: "Shared-workflow guard" runs-on: ubuntu-latest - needs: [static-limits, seed-semantics, probe] + needs: [static-limits, seed-semantics, gate-completeness, probe] if: ${{ !cancelled() }} steps: - name: Assert both guards reached a verdict env: STATIC: ${{ needs.static-limits.result }} SEED: ${{ needs.seed-semantics.result }} + LEGS: ${{ needs.gate-completeness.result }} PROBE: ${{ needs.probe.result }} run: | set -eu - echo "static-limits=${STATIC} seed-semantics=${SEED} probe=${PROBE}" + echo "static-limits=${STATIC} seed-semantics=${SEED} gate-completeness=${LEGS} probe=${PROBE}" # `skipped` is failed here on purpose. A guard that did not run is # not a guard that passed. - for pair in "static-limits:${STATIC}" "seed-semantics:${SEED}" "probe:${PROBE}"; do + for pair in "static-limits:${STATIC}" "seed-semantics:${SEED}" "gate-completeness:${LEGS}" "probe:${PROBE}"; do name="${pair%%:*}"; result="${pair##*:}" [ "${result}" = "success" ] || { echo "::error::${name} did not succeed (result=${result}). A guard that \ diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index 8cbf6171c..985a2ed43 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -3737,7 +3737,21 @@ jobs: report: runs-on: ubuntu-latest name: "Quality Report" - needs: [php-quality, vue-quality, frontend-build, frontend-checks, security, license, phpunit, newman, playwright, journeydoc-capture, baseline-protection, sbom, features-check, features-extract, hydra-gates] + # EVERY verdict-producing job must be listed here, and `frontend-tests` + # was not (#190). This job is the ONLY meaningful required check on main + # and beta — measured across all 25 fleet repos, whose `Main`/`Beta Branch + # Protection` rulesets require exactly `branch-protection / check-branch` + # and `quality / Quality Report` — so a leg missing from this list cannot + # block a merge no matter how red it goes. `Frontend Tests (unit)` runs a + # real unit suite with no continue-on-error, and its failure reached + # nothing. + # + # Latent rather than observed: sampled across ten repos it is currently + # green everywhere (with some supersession cancellations), so nothing has + # been waved through yet. A gate that would not catch a failure is still a + # dead gate, and `scripts/assert-quality-report-gates-every-leg.py` now + # keeps the list complete. + needs: [php-quality, vue-quality, frontend-build, frontend-checks, frontend-tests, security, license, phpunit, newman, playwright, journeydoc-capture, baseline-protection, sbom, features-check, features-extract, hydra-gates] if: always() # Observed across 266 executions: median 0.6 min — but 2% of runs sit at # ~30 min because the `Install PDF tools` apt-get step stalls on the Ubuntu diff --git a/scripts/assert-quality-report-gates-every-leg.py b/scripts/assert-quality-report-gates-every-leg.py new file mode 100644 index 000000000..93e0df1a2 --- /dev/null +++ b/scripts/assert-quality-report-gates-every-leg.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: EUPL-1.2 +"""Fail when a verdict-producing job is missing from Quality Report's `needs:`. + +WHY +--- +`quality / Quality Report` is the ONLY meaningful required check on `main` and +`beta`. Measured 2026-08-08 across all 25 fleet repos: every one has active +`Main Branch Protection` and `Beta Branch Protection` rulesets, and both +require exactly two contexts — `branch-protection / check-branch` and +`quality / Quality Report`. (`Development Branch Protection` exists too and +requires ZERO checks, which is deliberate: development is unprotected by +design, because admin merges into it are the team's routine workflow.) + +That makes the report job's `needs:` list the fleet's whole gate. Its two +gating steps read `needs.*.result`: + + if: ${{ always() && contains(needs.*.result, 'failure') }} + if: ${{ !cancelled() && contains(needs.*.result, 'cancelled') }} + +A job absent from `needs:` is absent from `needs.*`, so it can fail as loudly +as it likes and the required check stays green. `frontend-tests` +("Frontend Tests (unit)") was missing exactly this way (#190) — a real unit +suite, no `continue-on-error`, whose failure could not block a merge to main +or beta in any repo in the fleet. + +Nothing had been waved through yet: sampled across ten repos the job is +currently green everywhere. A gate that would not catch a failure is still a +dead gate, and "it has not bitten yet" is not a verdict. + +WHAT THIS ASSERTS +----------------- +Every top-level job in the workflow is either the report job itself or listed +in the report job's `needs:`. An exemption requires an entry in ALLOWLIST with +a stated reason, so the list can only shrink and it says why. + +Usage: assert-quality-report-gates-every-leg.py + assert-quality-report-gates-every-leg.py --positive-control +Exit: 0 all legs gated, 1 at least one leg is ungated. +""" + +from __future__ import annotations + +import argparse +import sys + +import yaml + +REPORT_JOB_NAME = "Quality Report" + +# job-id -> reason. Empty on purpose: every job in this workflow renders a +# verdict, so every job belongs in the gate. An entry here is a claim that a +# job CANNOT fail meaningfully, and it has to be argued. +ALLOWLIST: dict[str, str] = {} + + +def find_report_job(jobs: dict) -> str | None: + for job_id, job in jobs.items(): + if isinstance(job, dict) and job.get("name") == REPORT_JOB_NAME: + return job_id + return None + + +def ungated_jobs(path: str) -> tuple[str | None, list[str], int]: + """(report_job_id, ungated job ids, total job count).""" + with open(path, encoding="utf-8") as handle: + doc = yaml.safe_load(handle) + jobs = (doc or {}).get("jobs") or {} + report = find_report_job(jobs) + if report is None: + return None, [], len(jobs) + needs = jobs[report].get("needs") or [] + if isinstance(needs, str): + needs = [needs] + missing = [ + j for j in jobs + if j != report and j not in needs and j not in ALLOWLIST + ] + return report, missing, len(jobs) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("workflow") + ap.add_argument( + "--positive-control", + action="store_true", + help="drop a job from the needs list in memory; the check must then FAIL.", + ) + args = ap.parse_args() + + report, missing, total = ungated_jobs(args.workflow) + + if report is None: + print( + f"::error file={args.workflow}::no job named '{REPORT_JOB_NAME}' was " + f"found. Either it was renamed — in which case the required status " + f"check `quality / {REPORT_JOB_NAME}` no longer reports and every PR " + f"in the fleet is now permanently PENDING — or this parser stopped " + f"matching. Both are outages." + ) + return 1 + + # ASSERT THE INPUT IS NON-EMPTY. A workflow parsed down to two jobs would + # produce an empty `missing` list and a clean pass that measured nothing. + if total < 5: + print( + f"::error file={args.workflow}::parsed only {total} job(s). This " + f"workflow has well over a dozen; the parse is wrong, so 'no ungated " + f"legs' is not a finding." + ) + return 1 + + if args.positive_control: + # Prove the check has a reachable failure branch, using the real + # document rather than a synthetic one. + with open(args.workflow, encoding="utf-8") as handle: + doc = yaml.safe_load(handle) + jobs = doc["jobs"] + needs = list(jobs[report].get("needs") or []) + if not needs: + print("::error::the report job has no `needs:` at all — it gates nothing.") + return 1 + dropped = needs.pop() + jobs[report]["needs"] = needs + still_missing = [ + j for j in jobs if j != report and j not in needs and j not in ALLOWLIST + ] + if dropped not in still_missing: + print( + f"::error::dropping '{dropped}' from the report job's needs did " + f"NOT make this check flag it. The check is not reading the list " + f"it claims to read, so its clean pass means nothing." + ) + return 1 + print( + f"OK — the check fails when it should: removing '{dropped}' from " + f"`needs:` is detected. Its clean pass is a verdict." + ) + return 0 + + if missing: + for job_id in sorted(missing): + print( + f"::error file={args.workflow}::job '{job_id}' is NOT in the " + f"'{REPORT_JOB_NAME}' job's `needs:` list. That job is the only " + f"meaningful required check on main and beta across the fleet, and " + f"its gating steps read `needs.*.result` — so '{job_id}' can fail " + f"or be cancelled without blocking any merge. Add it to `needs:`, " + f"or add it to ALLOWLIST with a reason it cannot fail meaningfully." + ) + return 1 + + print( + f"OK — all {total - 1} leg(s) are in '{REPORT_JOB_NAME}'s `needs:`; a " + f"failure in any of them fails the fleet's required check." + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main())