Skip to content

test(ci): validate merge-base coverage ratchet before fleet rollout — DO NOT MERGE - #274

Closed
rubenvdlinde wants to merge 2 commits into
developmentfrom
test/coverage-ratchet-e2e
Closed

rubenvdlinde wants to merge 2 commits into
developmentfrom
test/coverage-ratchet-e2e

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Validation PR for ConductionNL/.github#181, opened to answer one question before that change reaches all sixteen apps: does phpunit actually run at the merge base?

The new step checks the app out at the merge base, reinstalls composer deps and re-runs the suite. If that cannot produce a clover report the step hard-fails by design — which is correct as a safety property but would be a fleet-wide false red if the base checkout does not work. That is not something the shared workflow can be trusted on without one live run.

This PR carries no code change, so the expected result on Guard coverage baseline is pass, with "coverage unchanged against the merge base". A second PR will delete a test to confirm the same mechanism goes red on a real drop.

The caller ref is pointed at the PR branch and will be reverted. Do not merge.

.coverage-baseline was read as a floor by the phpunit guard and as an exact
target by the push-side staleness check. Together they demand equality with a
checked-in constant, which against a moving base branch is not satisfiable:
closing "stale" means committing the value the tree will measure after the PR
lands. Measured on openregister — committed 58.93, development advanced
16030->16038 tests, merge result measured 58.88, guard reported a 0.05% drop.

coverage-guard.php gains --against=<clover.xml>, naming a report measured at
the merge base. When present it is the only floor; the committed constant is
reported but not enforced. Both numbers then come from one driver in one job,
so the xdebug/pcov statement-counting difference cancels rather than being
baked in, and the merge base cannot go stale.

Ratios are compared as exact integer cross-products, not rounded percentages:
at two decimals a one-statement regression read as "unchanged" and exited 0.
An empty or zero-statement report is now a hard error rather than 0%, which as
the merge-base side would set the floor to zero and pass every drop.

Verified on real CI clover artifacts: a genuine 1.44% drop fails, an unchanged
tree passes, and adding untested code fails while adding tested code passes.
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/larpingapp @ bdbab83

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-specs ✅
test-l10n ❌
composer ✅ ✅ 101/101
npm ✅ ✅ 537/537
PHPUnit ✅
Newman ❌
Playwright ✅
Hydra gates ❌

Quality workflow — 2026-08-06 06:39 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Validation complete — closing unmerged as intended.

Guard coverage baseline ran and passed on the designated leg (PHP 8.3 / NC stable31), job 92538024679:

head=bdbab83b base=b3ae515f merge-base=b3ae515f
Coverage current:       71.73%  (1020/1422 statements)
Coverage merge base:    71.73%  (1020/1422 statements)
Committed .coverage-baseline: 67.68% — recorded only, NOT enforced on this run.
OK: coverage unchanged against the merge base.

This answered the question it was opened for: phpunit does run at the merge base, so the new step's hard-fail-on-no-clover safety property does not turn into a fleet-wide false red. The companion drop test is #275.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant