Skip to content

Move HAPTIC to GitHub and add the Pod readiness gate - #44

Merged
phihos merged 123 commits into
mainfrom
ci/github-canonical-port-20261013
Oct 14, 2026
Merged

phihos merged 123 commits into
mainfrom
ci/github-canonical-port-20261013

Conversation

@phihos

@phihos phihos commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Moves HAPTIC's development, CI, and publication from GitLab to GitHub, and adds the opt-in Pod readiness gate that lets single-replica rollouts drop no requests. After this merges, no build, test, publication, or documentation path depends on GitLab's CI or container registry.

Size

+32k / −7.5k lines. About half of the additions are generated data: the offline Pod schema for the readiness gate's tests (7.6k), the generated 98-job CI matrix (ci-speed-proof.yml), and per-test duration weights the scheduler balances shards with (4.5k). The deletions are mostly .gitlab-ci.yml and the migration's one-time tooling.

CI on GitHub Actions

  • CI runs the complete matrix on every pull request, merge-queue group, and push: unit shards, chart and docs builds, the e2e matrix (HAProxy 3.0–3.4), minimum Kubernetes, upgrade baselines, Ingress and Gateway API conformance, and the former nightly suites (Gateway API release matrix, churn, scale, upstream canary). Only the canary may fail. No workflow runs on a schedule.
  • Measured earlier: the full matrix finished in 20m53s on GitHub (run 38432655131) against 28m04s on GitLab (pipeline 2940262274).
  • The former nightly suites cost about a minute of lead time (21m42s and 22m12s against 20m53s without them, run 38555150518), traded for one gate instead of a separate schedule.
  • Publication (release.yml, snapshot.yml, pages.yml) verifies the producing CI run, its job inventory, and its artifact digests before it publishes. The website publishes after every successful main CI run. The publication containers set BUILDX_NO_DEFAULT_OCI_ARTIFACT=true, as the GitLab release jobs did (build: add OCI artifact default opt-out docker/buildx#3991).
  • ci-images.yml publishes CI images for changed build inputs from main (ghcr.io/haproxy-haptic/haptic-ci:<input-hash>-hp<series>); jobs pull that tag before building in-job, so a toolchain bump slows down only the PR that makes it.
  • The startup load gate's budget follows the controller's startup probe window (80%, 240 s by default) instead of a fixed 120 s. Two controllers starting together on a 4-vCPU runner spent 104–116 s on ~1,130 validation tests, and e2e-rate-limit ran out of time at 98% with no failure. The reinit grace stays 45 s above the budget.

Registry, docs, and GitLab retirement

  • Chart defaults, install commands, the landing page, and examples use ghcr.io/haproxy-haptic/haptic. Releases up to 0.5.0 are already there with their original digests and signatures (run 38498045175).
  • Dependency updates come from Renovate, self-hosted in haproxy-haptic/renovate-runner with its own GitHub App. Workflow action pins carry their release versions so Renovate can track them.
  • .goreleaser.yml publishes to GitHub directly. Upgrade tests discover chart baselines from GHCR; the Ingress conformance echo image comes from registry.k8s.io at its pinned digest.
  • The documentation site packages the eight preserved playground engines as Brotli, so the complete site fits GitHub Pages (1.32 GB → 478 MB).
  • Deleted .gitlab-ci.yml and its train, budget, nightly-dispatch, and failure-reporting tooling, the one-time registry mirrors, the GitLab comparison baseline, and 23 migration-only proof and diagnosis workflows.
  • Tests that read .gitlab-ci.yml are replaced by checks on the GitHub workflows: job inventory and prose profile (.github/ci-verification.json), nightly selections and failure policy, feature e2e profiles, the bounded benchmark smoke, chart-publication ordering, per-repository image attestation, maint/ tagging, the latest ordering check, and upgrade baseline coverage.
  • CLAUDE.md, agent docs, contributor docs, the Gateway API upgrade playbook, the issue tracker guide, the chart's home/sources, the PR skill, SECURITY.md (private vulnerability reporting), and Renovate (GitLab preset inlined) describe GitHub only. The Go module path stays gitlab.com/haproxy-haptic/haptic for now.

Pod readiness gate

controller.templateLibraries.kubernetesBackends.podReadinessGate.enabled adds the condition haproxy-haptic.org/load-balancer-ready. A pod listing it under readinessGates becomes Ready only after every HAProxy pod runs a configuration that routes to it, so a Deployment keeps the old pod until then. It's chart-only: kubernetes-backends watches pods, routes a gated pod as soon as its containers are ready, and sets the condition in the deployed status phase. See ADR-0034 and docs/site/docs/operations/pod-readiness-gate.md.

The GitHub runs exposed the problem it fixes: an ungated single-replica rollout stopped the old backend 9 ms after HAPTIC saw the new endpoint, and the deploy landed 482 ms later, so two probes failed (run 38483303507).

  • TestIngressRollingRestartZeroDowntime gates its backend and allows no failed request.
  • TestIngressRollingRestartUngatedRecovers keeps the ungated case: only fast 503s, within one window no longer than the scale suite's 3 s convergence budget. This replaces the old "at most one 503" bound for ungated pods, which asserted runner speed rather than a guarantee HAPTIC can give.
  • Enabling the gate grants the controller cluster-wide pods read and pods/status patch; the RBAC golden test pins that this happens only with the gate on.

Local verification of the final commit

  • make lint audit, make test (11,804 tests), make lint-chart, and ./scripts/test-templates.sh (1,144/1,144).
  • Strict MkDocs builds of the docs site and landing page.
  • e2e on a fresh kind cluster with the controller limited to 4 CPUs: TestObservedGenerationIsTruthful|TestHaptic|TestGateway|TestIngress|TestHTTPRoute, 126 passed, 0 failed, including both rolling-restart tests with zero failed requests.

phihos added 30 commits October 12, 2026 18:44
Comment thread .github/ci-verification.json
gitar-bot[bot]

This comment was marked as resolved.

@gitar-bot

gitar-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Gitar review

Review started. Results will appear in the dashboard comment.

Renovate skips SHA-pinned actions without a version comment as
unversioned references, so none of the 73 workflow pins would update.
Each pin now carries its release tag, the scheduler keeps the comments
when it regenerates the matrix, and a test rejects a pin without one.
…uites

Moving these suites to nightly.yml left release tagging and publication
without them: both accept the CI receipt for the tagged commit, and that
receipt no longer covered them. They return to the matrix behind the
full profile. Pull requests and merge groups select a pre-merge profile
that skips exactly the jobs in pre_merge_skipped_jobs, so PR lead time
is unchanged. Pushes to main and maint, manual dispatches, and the weekly
run use full, and publication rejects any receipt that isn't full or an
allowed prose receipt. Nightly keeps only the upstream canary and the
weekly full run. The PR-head fallback in the scale gate is gone, since
scale no longer runs on pull requests.
@phihos

phihos commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Gitar review

@gitar-bot
gitar-bot Bot dismissed their stale review October 14, 2026 12:40

✅ All code review findings resolved.

Configure merge blocking

Comment thread .github/workflows/ci-speed-proof.yml Outdated
A GitHub release download returned HTTP 500 while helm-3.2 built the CI
image in-job, which every job does until main publishes an image for
changed inputs. The checksum-verified downloads now retry like the
kubectl download already did.
…htly.yml

A direct dispatch left verification_profile empty, so the full-only suites were skipped and the gate rejected the run.
@phihos

phihos commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Gitar review

Compatibility, churn, scale, and the upstream canary run in the PR,
merge-queue, and push matrix again, at about a minute of lead time,
instead of through separate profiles and a nightly schedule. This
restores the bd27203 verification design: one full profile, the
canary as the only allowed failure, and merge-queue reuse able to
cover post-merge pushes. nightly.yml and its planner are removed.
@phihos

phihos commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Gitar review

@gitar-bot

gitar-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Gitar review

Starting the review now. Results will appear in the dashboard comment when it finishes.

The goreleaser hook downloaded the complete module graph, including every
tool's test dependencies, on each snapshot and release build: four minutes,
and a proxy.golang.org 502 failed the snapshot job. Lint's go mod tidy -diff
already rejects an untidy tree in the same CI run, so the hook could only
mutate a verified tree or repeat that check over the network.
@phihos

phihos commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Gitar review

@gitar-bot

gitar-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Gitar review

Starting a review of this PR now. Results will appear in the dashboard comment.

GitHub release assets returned HTTP 500 for about a minute in run
38571838388, failing the CI image build (10 s of retries) and the e2e
Gateway API CRD install (12 s) in two jobs. Both now back off
exponentially for about two minutes; the CRD install stops waiting when
the suite is cancelled.
@phihos

phihos commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Gitar review

@gitar-bot

gitar-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Gitar review

Starting a review of this PR now. Results will appear in the dashboard comment.

@gitar-bot

gitar-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 2 closed / 2 findings

🔴 High risk · The opt-in readiness gate expands controller RBAC to cluster-wide pod reads and status patches; a scope error could alter workload readiness and disrupt rollouts.

Moves HAPTIC's CI, publication, and docs from GitLab to GitHub Actions and adds the opt-in Pod readiness gate for zero-downtime single-replica rollouts. Direct dispatch of ci-speed-proof.yml that skipped post-merge suites and then failed has been addressed. No open issues remain.

✅ 2 closed
✅ Edge Case: Direct dispatch of ci-speed-proof.yml skips post-merge suites, then fails

📄 .github/workflows/ci-speed-proof.yml:3 📄 .github/workflows/ci-speed-proof.yml:4239
ci-speed-proof.yml still declares workflow_dispatch: {} without inputs. On a direct dispatch, inputs.verification_profile is empty, so the new inputs.verification_profile == 'full' condition on the five post-merge jobs (compatibility ×3, churn, scale) is false and they're skipped. The verified step still sets CI_VERIFICATION_PROFILE: ${{ inputs.verification_profile || 'full' }}, so verify-run.py requires those jobs and fails every direct dispatch. Under the old != 'prose' condition, empty input ran every job, so this broke in this commit. To fix, treat an empty profile as full in the job conditions (for example inputs.verification_profile == 'full' || inputs.verification_profile == ''), or drop the bare workflow_dispatch trigger now that the docs point to dispatching ci.yml.

Closed: Bug: Releases no longer require Gateway compatibility, churn or scale results

📄 .github/ci-verification.json:49-63 📄 .github/scripts/verify-run.py:13-27 📄 .github/workflows/nightly.yml:39-42 📄 .github/workflows/nightly.yml:106-109 📄 .github/workflows/nightly.yml:147-150 📄 .github/workflows/nightly.yml:276-281 📄 scripts/ci/github_evidence.py:86-88 📄 docs/agents/github-scheduling.md:31-41
This commit drops the gateway-compatibility-*, gateway-churn and scale jobs from required_jobs in .github/ci-verification.json. It also removes them from ci-speed-proof.yml and moves them to nightly.yml. The publication path (verify-run.py verify(), github_evidence.check_direct_receipt, and the release, snapshot and pages workflows) only checks the CI run's job inventory, and nothing in release.yml or scripts/ci/github_release*.py reads nightly results.

So a merge, a v<version> tag and a release can now be published from a tree whose compatibility, churn or scale suite never ran, or failed in a later nightly. Before this commit, those suites were required evidence for the exact merged tree. The repo's CI-verification policy says required validation must not be skipped to save lead time, and the stated reason for the move is to shorten the PR matrix by about 1 minute.

Possible fixes:

  • Keep these suites in the required inventory for release and tag publication only, for example as a release-profile required_jobs list. Release then has to run them, or check a successful nightly.yml run for the same head_sha and tree.
  • Or make github_release_tag refuse to tag until a successful nightly run with all of these suites exists for that commit.
Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

Was this helpful? React with 👍 / 👎 | Gitar

@phihos
phihos added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 6f7285d Oct 14, 2026
108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant