Repository navigation
fix(apps): run homepage and umami Flagger primaries at 3 replicas for HA - #1906
Merged
Merged
Conversation
…available: 1) PR #1882 unified the hand-written PDBs to maxUnavailable: 1, but the chart-shipped PDBs (Helm value toggles) were never swept. Three of them ship minAvailable: 1 over workloads that run a single replica in prod, so the node hosting their pod can never be drained -- the exact #1880 incident class that deadlocks ksail node recycles, Talos OS upgrades and autoscaler scale-down: - hcloud-csi controller (hcloud_csi_controller_replicas: "1") - reloader (reloader_replicas: "1") - trust-manager (trust_manager_replicas: "1") Flip every chart-shipped PDB to the drain-safe house pattern maxUnavailable: 1 (cert-manager x3, cilium relay+operator, oauth2-proxy, metrics-server, dex, vpa-admission, kyverno-admission too -- they run 2 replicas today and were safe, but minAvailable would re-introduce an undrainable node the moment a floor dropped to 1). maxUnavailable scales with the live replica count, so a single pod can always be evicted. Add a Kyverno Audit ClusterPolicy (validate-pdb-drain-safe) that flags any PodDisruptionBudget setting spec.minAvailable, so a new chart or operator can't silently reintroduce the landmine. CloudNativePG-managed PDBs are excluded -- the operator owns drain-safety via switchover and scales minAvailable to instances-1 for multi-instance clusters. Validated: ksail workload validate (local + prod) passes for every changed file; the one prod failure is the pre-existing coroot notificationIntegrations CRD-catalog schema flake (no coroot files touched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The maxUnavailable: 1 PDBs keep nodes drainable, but a PDB only preserves *availability* when the workload has >=2 replicas (3 to also survive a concurrent failure during a drain). Add a Kyverno Audit ClusterPolicy (validate-replica-floor) that flags Deployments/StatefulSets running fewer than 3 replicas, completing the eviction-safety story (drain-safe PDB + topology spread + HA replica floor). Audit, not Enforce: .spec.replicas is owned by KEDA (scale-to-zero), Flagger (canary primaries) and leader-election operators -- mutating it would fight those controllers and drift, and tripling every replica would blow the cluster's memory budget at once. This surfaces gaps in PolicyReports for deliberate, per-workload action. Exemptions (a floor is wrong, not missing): the opt-out label platform.devantler.tech/replica-floor: exempt (workload or pod template); Flagger primaries/sources; KEDA scale-to-zero targets; single-purpose control-plane/operator namespaces and the leader-election singletons that share a namespace with HA workloads. Validated: ksail workload validate passes (300 files); cluster-policies base builds (17 ClusterPolicies). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rather than exempting the leader-election controllers from the
require-three-replicas floor, actually make them HA. Each of these runs
leader election (one active, the rest warm standbys that take over on a
node failure/drain), verified against each chart/upstream:
cloudnative-pg, reloader (enableHA), trust-manager, hcloud-ccm,
hcloud-csi-controller, snapshot-controller, kyverno background/reports/
cleanup -> 3 replicas (prod variables-cluster-config-map).
flagger -> 3 replicas + leaderElection.enabled (required before >1, or
canaries get double-driven). snapshot-controller gains a replica var.
The replica-floor policy exclusions shrink accordingly. What stays exempt
is now only what genuinely can't run 3:
- velero: chart hardcodes replicas:1, no leader election (the repo's
velero_replicas var is a no-op; its "leader-elected" comment is wrong);
- external-dns: chart has no replicaCount; >1 = duplicate Cloudflare
writes + API throttling (project recommends single-instance);
- openbao: HA needs a Raft redesign (ha.raft + auto-unseal + per-node
PVs), not a replica bump -- tracked separately;
- KEDA scale-to-zero apps + Flagger canary sources/primaries: replica
count owned by a controller (the Flagger primary runs the HA replicas);
- VPA recommender/updater: deferred -- --leader-elect support on the
pinned version needs verifying before scaling (wrong = racy
recommendations / double evictions).
CAPACITY: this adds ~14 controller pods of memory requests. The cluster is
capacity-tuned; a baseline worker bump may be needed before this merges.
Validated: ksail workload validate local (300 files) + prod pass for every
changed file (the one prod failure is the pre-existing coroot CRD-catalog
schema flake); new config-map vars substitute; cluster-policies base builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified VPA chart 4.11.0 (appVersion 1.6.0) supports `--leader-elect` on
the recommender and updater binaries (added in VPA 1.3.0); the lease lives
in kube-system and the chart's clusterroles already grant coordination
leases. Add `extraArgs: { leader-elect: true }` to each (map -> --leader-elect=true)
and scale recommender/updater 1->3 and the admission controller 2->3 (it is
all-active, no leader election needed). Drop the vertical-pod-autoscaler
namespace from the replica-floor exemptions.
Without leader election, multiple recommenders produce racy recommendations
and multiple updaters double-evict pods -- which is exactly why this was
deferred from the controller-HA PR until the flag support was confirmed.
Validated: ksail workload validate passes for the changed files; policy base builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flagger copies the source Deployment's replicas onto the primary. The postRenderers previously REMOVED /spec/replicas (so Flux wouldn't fight Flagger's between-rollout scale-to-zero of the source), which left each primary at the chart default of 1 -- a documented "minor HA reduction". Pin the source replicas to 3 instead of removing them, so Flagger runs the primary at 3. This is safe because the platform's Flux drift detection already ignores Deployment /spec/replicas, so Flagger's scale-to-zero of the source is not corrected/fought (the original reason for the strip). opencost is intentionally left at 1: it's a cost *exporter* (singleton by nature -- 3 replicas = duplicate compute + metrics) and is already exempt via the observability namespace.⚠️ homepage is the apex/front page (high blast radius) -- watch the first Flagger rollout after this lands. Validated: ksail workload validate passes for the changed files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onflict) The System Test failed: cert-manager, metrics-server and kyverno are bootstrap-installed by ksail (ksail.yaml certManager/metricsServer/ policyEngine) with their chart-default minAvailable PDBs BEFORE Flux. Flux's HelmRelease then tried to flip them to maxUnavailable, and server-side apply can't drop the other field -- "PodDisruptionBudget ... minAvailable and maxUnavailable cannot be both set" (kyverno's chart also fails to render both). These are the only three charts ksail bootstraps, and the only three that failed; the Flux-only flips (cilium, oauth2-proxy, dex, vpa, reloader, …) reconciled fine. Revert just those three to minAvailable: 1 -- drain-safe at their 2-replica floors (cert_manager/metrics_server/kyverno_admission all "2") and matching ksail's install, so no SSA conflict -- and exempt them from validate-pdb-drain-safe with that rationale. Validated: ksail workload validate passes; cluster-policies base builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onflict) The System Test failed: cert-manager, metrics-server and kyverno are bootstrap-installed by ksail (ksail.yaml certManager/metricsServer/ policyEngine) with their chart-default minAvailable PDBs BEFORE Flux. Flux's HelmRelease then tried to flip them to maxUnavailable, and server-side apply can't drop the other field -- "PodDisruptionBudget ... minAvailable and maxUnavailable cannot be both set" (kyverno's chart also fails to render both). These are the only three charts ksail bootstraps, and the only three that failed; the Flux-only flips (cilium, oauth2-proxy, dex, vpa, reloader, …) reconciled fine. Revert just those three to minAvailable: 1 -- drain-safe at their 2-replica floors (cert_manager/metrics_server/kyverno_admission all "2") and matching ksail's install, so no SSA conflict -- and exempt them from validate-pdb-drain-safe with that rationale. Validated: ksail workload validate passes; cluster-policies base builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onflict) The System Test failed: cert-manager, metrics-server and kyverno are bootstrap-installed by ksail (ksail.yaml certManager/metricsServer/ policyEngine) with their chart-default minAvailable PDBs BEFORE Flux. Flux's HelmRelease then tried to flip them to maxUnavailable, and server-side apply can't drop the other field -- "PodDisruptionBudget ... minAvailable and maxUnavailable cannot be both set" (kyverno's chart also fails to render both). These are the only three charts ksail bootstraps, and the only three that failed; the Flux-only flips (cilium, oauth2-proxy, dex, vpa, reloader, …) reconciled fine. Revert just those three to minAvailable: 1 -- drain-safe at their 2-replica floors (cert_manager/metrics_server/kyverno_admission all "2") and matching ksail's install, so no SSA conflict -- and exempt them from validate-pdb-drain-safe with that rationale. Validated: ksail workload validate passes; cluster-policies base builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onflict) The System Test failed: cert-manager, metrics-server and kyverno are bootstrap-installed by ksail (ksail.yaml certManager/metricsServer/ policyEngine) with their chart-default minAvailable PDBs BEFORE Flux. Flux's HelmRelease then tried to flip them to maxUnavailable, and server-side apply can't drop the other field -- "PodDisruptionBudget ... minAvailable and maxUnavailable cannot be both set" (kyverno's chart also fails to render both). These are the only three charts ksail bootstraps, and the only three that failed; the Flux-only flips (cilium, oauth2-proxy, dex, vpa, reloader, …) reconciled fine. Revert just those three to minAvailable: 1 -- drain-safe at their 2-replica floors (cert_manager/metrics_server/kyverno_admission all "2") and matching ksail's install, so no SSA conflict -- and exempt them from validate-pdb-drain-safe with that rationale. Validated: ksail workload validate passes; cluster-policies base builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…conflict) hcloud-csi is hetzner-only so the docker System Test didn't exercise it, but it hits the same trap as cert-manager/metrics-server/kyverno in the prod deploy: ksail installs the Hetzner CSI controller (chart default PDB, minAvailable: 1) during bootstrap before Flux, so Flux flipping it to maxUnavailable conflicts at server-side apply. Unlike hcloud-ccm (whose chart ships no PDB, so a hand-written maxUnavailable one works), the hcloud-csi chart defaults create: true + minAvailable: 1. Keep minAvailable: 1 (matches ksail, no conflict) and exempt it from validate-pdb-drain-safe. Drain-safety comes from the HA PR's replica bump to 3 (minAvailable: 1 over 3 replicas bounds disruption to 1), not from maxUnavailable. Validated: ksail --config ksail.prod.yaml workload validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…conflict) hcloud-csi is hetzner-only so the docker System Test didn't exercise it, but it hits the same trap as cert-manager/metrics-server/kyverno in the prod deploy: ksail installs the Hetzner CSI controller (chart default PDB, minAvailable: 1) during bootstrap before Flux, so Flux flipping it to maxUnavailable conflicts at server-side apply. Unlike hcloud-ccm (whose chart ships no PDB, so a hand-written maxUnavailable one works), the hcloud-csi chart defaults create: true + minAvailable: 1. Keep minAvailable: 1 (matches ksail, no conflict) and exempt it from validate-pdb-drain-safe. Drain-safety comes from the HA PR's replica bump to 3 (minAvailable: 1 over 3 replicas bounds disruption to 1), not from maxUnavailable. Validated: ksail --config ksail.prod.yaml workload validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…conflict) hcloud-csi is hetzner-only so the docker System Test didn't exercise it, but it hits the same trap as cert-manager/metrics-server/kyverno in the prod deploy: ksail installs the Hetzner CSI controller (chart default PDB, minAvailable: 1) during bootstrap before Flux, so Flux flipping it to maxUnavailable conflicts at server-side apply. Unlike hcloud-ccm (whose chart ships no PDB, so a hand-written maxUnavailable one works), the hcloud-csi chart defaults create: true + minAvailable: 1. Keep minAvailable: 1 (matches ksail, no conflict) and exempt it from validate-pdb-drain-safe. Drain-safety comes from the HA PR's replica bump to 3 (minAvailable: 1 over 3 replicas bounds disruption to 1), not from maxUnavailable. Validated: ksail --config ksail.prod.yaml workload validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…conflict) hcloud-csi is hetzner-only so the docker System Test didn't exercise it, but it hits the same trap as cert-manager/metrics-server/kyverno in the prod deploy: ksail installs the Hetzner CSI controller (chart default PDB, minAvailable: 1) during bootstrap before Flux, so Flux flipping it to maxUnavailable conflicts at server-side apply. Unlike hcloud-ccm (whose chart ships no PDB, so a hand-written maxUnavailable one works), the hcloud-csi chart defaults create: true + minAvailable: 1. Keep minAvailable: 1 (matches ksail, no conflict) and exempt it from validate-pdb-drain-safe. Drain-safety comes from the HA PR's replica bump to 3 (minAvailable: 1 over 3 replicas bounds disruption to 1), not from maxUnavailable. Validated: ksail --config ksail.prod.yaml workload validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o claude/ha-scale-controllers
…ollers # Conflicts: # k8s/bases/infrastructure/cluster-policies/best-practices/validate-replica-floor.yaml
# Conflicts: # k8s/bases/infrastructure/cluster-policies/best-practices/validate-replica-floor.yaml # k8s/clusters/prod/bootstrap/variables-cluster-config-map.yaml
…ection main is broken: #1900 enabled flagger leaderElection and #1922 added a post-renderer doing `op: replace /spec/strategy/type`. With leaderElection ON the flagger chart renders NO spec.strategy block at all (it defaults to RollingUpdate), so the replace op fails — "doc is missing path: /spec/strategy/type" — and the flagger HelmRelease InstallFailed, wedging infrastructure-controllers. (Confirmed by rendering the chart: leaderElection off -> strategy.type: Recreate; on -> no strategy block.) Replace the two ops with a single `op: add /spec/strategy` of the full object. JSON-Patch `add` CREATES the block when the chart omits it (leaderElection on) and REPLACES it when the chart sets Recreate (off), so it is correct in both states and preserves #1922's explicit RollingUpdate params for drift stability. Validated: ksail workload validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
devantler
enabled auto-merge
June 9, 2026 20:24
# Conflicts: # k8s/bases/apps/umami/helm-release.yaml # k8s/clusters/prod/bootstrap/variables-cluster-config-map.yaml
![botantler-1[bot]](https://avatars.githubusercontent.com/in/1025824?s=60&v=4){kind=small}
Contributor
|
🎉 This PR is included in version 1.44.15 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #1903 (clean branch — the dropped KEDA always-on commit is gone). Stacked on #1901.
Flagger copies the source Deployment's
spec.replicasonto the-primary. The postRenderers previously removed/spec/replicas(so Flux wouldn't fight Flagger's scale-to-zero of the source), leaving each primary at the chart default of 1. This pins the source replicas to 3 (op: replace), so Flagger runs the primary at 3 — safe because the platform's Flux drift detection already ignores/spec/replicas.ksail workload validate.🤖 Generated with Claude Code