Skip to content

The whole fleet pins hydra-gates-ref: v1.0.1, which predates #147/#148/#149 — 16 dead gates are still live in CI everywhere #159

Open

Description

@rubenvdlinde

All 22 repos that set enable-hydra-gates: true also pin:

hydra-gates-ref: v1.0.1

v1.0.1 is f4d9756 (2026-08-03). Everything since is invisible to CI:

commit tag in v1.0.1?
f4d9756 fix(gate-4): audit the LOCK (#135) v1.0.1 ✅
d9645e3 gate-53 ignored the diff scope; gate-22 printed FAIL — 0; gate-33 had never run (#139) v1.1.0 ❌
549ad1a 16 gates reported PASS when their helper never ran (#147) — ❌
0d3e589 produce tests/axe/report.json so gate-33 can actually run (#148) — ❌
fc70ea6 gate-7 follows delegation; gates 6/7 stop passing on an empty scope (#149) — ❌

So every in-CI Hydra Gates run on the fleet today is executing a script in which 16 gates report PASS when their helper never ran, and in which gates 6 and 7 report PASS on an empty scope. That is the dead-gate shape those PRs were written to close, still live everywhere.

Confirmed for: app-versions, decidesk, docudesk, doriath, hermiq, larpingapp, launchpad, nextcloud-app-template, nldesign, openbuild, opencatalogi, openconnector, openregister, petstore, pipelinq, planix, portaliq, procest, scholiq, shillinq, softwarecatalog, zaakafhandelapp.

$ for r in …; do gh api repos/ConductionNL/$r/contents/.github/workflows/code-quality.yml?ref=development \
    -q .content | base64 -d | grep hydra-gates-ref; done
hydra-gates-ref: v1.0.1     # ×22, no exceptions

Two things needed:

  1. Cut a tag containing fix(gates): 16 gates reported PASS when their helper never ran — 2 are authorization gates #147/feat(quality): produce tests/axe/report.json so gate-33 (axe-core) can actually run #148/fix(hydra-gates): gate-7 follows delegation; gates 6/7 stop passing on an empty scope #149 (v1.2.0) and bump the fleet to it. v1.1.0 is not enough — it predates all three.
  2. Decide whether the pin should be a moving minor (v1) rather than an exact patch. A patch pin means every gate fix needs a 22-repo bump PR wave, which is exactly why the fleet is 4 fixes behind after two days.

Related: the full-tree numbers I am reporting elsewhere use main, so they are not comparable to what CI prints today — worth stating explicitly wherever both appear.

Activity

  1. rubenvdlinde commented on Aug 5, 2026

    @rubenvdlinde
    ContributorAuthor

    Rolled. And the answer to question 2 is now forced by evidence rather than preference.

    Status: v1.3.0 PRs open on 17 repos + hrmq enabled for the first time; doriath already merged; openbuild/portaliq/procest/scholiq have their own author's PRs, which I commented on rather than duplicating. Table at the bottom.

    The pin was not just stale — since 13:52Z today it was RED, fleet-wide

    f7eaf2a (#164) flipped hydra-gates-require-full-coverage to default true in quality.yml. That flag requires a gate to declare itself not-applicable. The vocabulary to declare ships in the pinned package:

    pin _skip calls
    v1.0.1 0
    v1.2.0 0
    v1.3.0 36

    A pin with zero _skip calls cannot say "not applicable", so gates 4/24/33 became DID NOT RUN and failed the job everywhere at once. Reproduced byte-for-byte against a real blocked PR (nldesign#220, PHPMD-only):

    v1.0.1  exit 98  COVERAGE: 59 of 61   GATES THAT DID NOT RUN: 24 33
    v1.3.0  exit 0   COVERAGE: 59 of 63   NOT APPLICABLE: 6 7 24 33
    

    Confirmed in real CI on the same repo/branch, both runs after 13:52Z: app-versions 31012647428 at v1.2.0 failure → 31013564400 at v1.3.0 success. This is what was blocking ~14 PHPMD PRs.

    Question 2: the pin must move WITH quality.yml, or not at all

    Three separate desyncs in one day, all the same shape — the shared workflow is @main while the package is pinned:

    1. fix(hydra-gates): a not-applicable gate is not a coverage gap — fix the accounting, then flip the flag #164 flipped a flag whose supporting accounting lives in the package → whole fleet red.
    2. fix(gate-46): resolve anchors the way GitHub actually publishes them #165 added gh_slugify (gate-46 anchor resolution) → in no released tag, so consumers on any tag still run the old resolver. Counts measured with it present are not the counts CI produces.
    3. fix(quality): re-land axe DOM scoping, with the runner in a file — a 22 KB run: step is what made the workflow unresolvable #168 made quality.yml invoke ${GITHUB_WORKSPACE}/gates/hydra-gates/scripts/axe-run.cjs — the workflow now reaches into the pinned package by path. That file exists only on main. Latent today only because no repo sets enable-axe: true; the first one that does, on a tagged pin, gets a file-not-found.

    (3) is the decisive one. A workflow that executes a path inside the pinned package has made the two a single unit. Pinning one and floating the other is not a conservative choice — it is an unversioned interface, and it has now failed three times in a day.

    Recommendation, in preference order:

    1. Pin both together. Consumers reference quality.yml@vX.Y.Z and hydra-gates-ref: vX.Y.Z from the same tag. One bump wave moves both; a desync becomes impossible by construction.
    2. Failing that, float both (@main + hydra-gates-ref: main). Loses reproducibility but at least the halves always match.
    3. What we have now — pin one, float the other — should not survive this issue.

    A moving v1 minor tag does not fix this. It reduces the bump waves but leaves the workflow and package on independent clocks, which is the actual defect.

    I cut two tags

    I deliberately rolled v1.3.0, not v1.4.0, for fleet uniformity with the repo that already landed, and because v1.3.0 is what demonstrably fixes today's red. v1.4.0 is a strict superset and should be a separate, measured follow-up — gh_slugify reduces gate-46 findings (openconnector: 42 with it vs 109 without), so it makes repos greener, but that is a verdict change across 22 repos and deserves its own measurement rather than riding along on an urgent fix.

    One correction to my own earlier reporting in this thread

    My first fallout table showed gate-22 manifest-validation newly failing in 8 repos. That was my harness, not the repos — ajv was absent locally, so the validator ran degraded and correctly refused to report a pass. CI installs ajv. With it installed, gate-22 passes in all of them, and the corrected delta is the opposite of what I first had: 12 repos lose a gate-22 failure and 5 lose gate-7. Flagging it because it is the same class of error this issue is about — a measurement artifact that reads exactly like a finding.

  2. added 20 commits that reference this issue on Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions