Repository navigation
feat(ha): zero-downtime hardening for the operator tier (stacked on #1376) - #1377
Merged
Merged
Conversation
…ng apps Make homepage, headlamp, and whoami survive single-node loss and roll without downtime: - replicaCount=2 (variable-overridable per cluster) - PodDisruptionBudget minAvailable=1 - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 For homepage, the upstream chart has no native PDB or topologySpread support, so the PDB is a sibling resource and the spread constraint is injected via the existing postRenderer kustomize patch. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the dex / oauth2-proxy / auth-proxy login path survive single-node loss and roll without downtime, so users don't get logged out during upgrades or node reboots: - replicaCount=2 (variable-overridable per cluster) - PodDisruptionBudget minAvailable=1 - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 - auth-proxy: terminationGracePeriodSeconds=30 + preStop sleep 10 for graceful in-flight request draining oauth2-proxy stays on cookie-based session storage (chart default), so no shared session backend is needed for the second replica. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run the kyverno admission webhook with replicas=2, a PodDisruptionBudget with minAvailable=1, and topology spread across hostname so a single node loss or drain never leaves the cluster without an admission controller. Without this, every Create/Update during a node reboot or roll would fail (or be ignored, depending on failurePolicy). Override the chart's default updateStrategy (maxSurge=1, maxUnavailable=40%) to maxSurge=1/maxUnavailable=0 so a rollout always keeps the existing pod up until the new one is ready. Background, cleanup, and reports controllers stay at the chart default (leader-elected singletons) — they don't sit on the request path, so HA there is unnecessary cost. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cloudflared is the public ingress front-door for dev/prod, so single- node loss or a rolling reboot of one node would interrupt all inbound traffic. Make it survive both: - replicaCount=2 (chart default; now exposed as variable for override) - PodDisruptionBudget minAvailable=1 (chart has no built-in PDB) - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 PDB and topologySpread are added via the existing postRenderer kustomize patch since the cloudflare-tunnel chart exposes neither as a value. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the same HA pattern (replicaCount=2, PDB minAvailable=1, hostname topologySpread, RollingUpdate with maxUnavailable=0) to the base operators that the cluster's day-2 operations depend on: - cert-manager: controller, webhook (request path, blocks all cert issuance), and cainjector each get topologySpread + strategy on top of the existing replicas/PDB - trust-manager: add topologySpread on top of existing replicas/PDB - cloudnative-pg: operator owns the database StatefulSets, runs the validating/mutating webhook for Cluster CRs. Add updateStrategy + topologySpread + sibling PDB (chart has no built-in PDB) - reloader: enable HA (chart requires explicit reloader.enableHA=true for leader election), bump replicas, add topologySpread + PDB - metrics-server: powers HPA + VPA + kubectl top — was running with empty values (replicas=1, no PDB). Add full HA stack - vpa: harden the admission controller (webhook on the request path); recommender + updater stay singletons (leader-elected) - cilium: add operator PDB + updateStrategy and PDBs for hubble relay/ui (replicas were already templated) flux-operator and goldilocks intentionally left as-is. flux-operator's chart does not expose a replica count; it is a leader-elected singleton in the recommended deployment. goldilocks is an advisor-tier dashboard without HA-grade requirements. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply HA pattern to the Hetzner-specific infrastructure that runs only on the Omni clusters (dev/prod). All three are leader-elected, so running 2 replicas means failover is instant instead of waiting on lease expiry. - hcloud-cloud-controller-manager: replicaCount=2, podAntiAffinity across hostname (chart exposes affinity but not topologySpread), RollingUpdate maxSurge=1/maxUnavailable=0, sibling PDB (chart has no native PDB support) - hcloud-csi controller: replicaCount=2, native podDisruptionBudget enabled with minAvailable=1, podAntiAffinity across hostname. CSI node DaemonSet stays at chart defaults (it's already one-per-node) - origin-ca-issuer: replicaCount=2, sibling PDB (chart only exposes replicaCount). Cloudflare origin certs are issued through this on cert renewal — single-replica means a node loss during rotation blocks issuance until reschedule kubelet-serving-cert-approver intentionally skipped: it pulls a standalone-install.yaml from upstream by URL and would need to be restructured to apply patches. Acceptable singleton for now — it only runs on bootstrap and node join. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CoreDNS in providers/docker is a hand-rolled Deployment (replicas=2 already, with podAntiAffinity). Add a PDB minAvailable=1 so a node drain can never evict both replicas at once, and tighten the deployment strategy to maxSurge=1/maxUnavailable=0 so DNS resolution stays continuous across rolls. Local Docker clusters share the same DR/HA test surface as Omni so this stays close to the prod-equivalent kube-system DNS posture. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Apr 18, 2026
Merged
devantler
added a commit
that referenced
this pull request
Apr 18, 2026
#1379) * feat(ha): add PDB, topology spread, and rolling strategy to user-facing apps Make homepage, headlamp, and whoami survive single-node loss and roll without downtime: - replicaCount=2 (variable-overridable per cluster) - PodDisruptionBudget minAvailable=1 - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 For homepage, the upstream chart has no native PDB or topologySpread support, so the PDB is a sibling resource and the spread constraint is injected via the existing postRenderer kustomize patch. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): add PDB, topology spread, and rolling strategy to auth path Make the dex / oauth2-proxy / auth-proxy login path survive single-node loss and roll without downtime, so users don't get logged out during upgrades or node reboots: - replicaCount=2 (variable-overridable per cluster) - PodDisruptionBudget minAvailable=1 - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 - auth-proxy: terminationGracePeriodSeconds=30 + preStop sleep 10 for graceful in-flight request draining oauth2-proxy stays on cookie-based session storage (chart default), so no shared session backend is needed for the second replica. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): make kyverno admission controller HA Run the kyverno admission webhook with replicas=2, a PodDisruptionBudget with minAvailable=1, and topology spread across hostname so a single node loss or drain never leaves the cluster without an admission controller. Without this, every Create/Update during a node reboot or roll would fail (or be ignored, depending on failurePolicy). Override the chart's default updateStrategy (maxSurge=1, maxUnavailable=40%) to maxSurge=1/maxUnavailable=0 so a rollout always keeps the existing pod up until the new one is ready. Background, cleanup, and reports controllers stay at the chart default (leader-elected singletons) — they don't sit on the request path, so HA there is unnecessary cost. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): add PDB and topology spread to cloudflared tunnel cloudflared is the public ingress front-door for dev/prod, so single- node loss or a rolling reboot of one node would interrupt all inbound traffic. Make it survive both: - replicaCount=2 (chart default; now exposed as variable for override) - PodDisruptionBudget minAvailable=1 (chart has no built-in PDB) - topologySpreadConstraints across hostname (ScheduleAnyway, maxSkew=1) - Deployment strategy maxSurge=1, maxUnavailable=0 PDB and topologySpread are added via the existing postRenderer kustomize patch since the cloudflare-tunnel chart exposes neither as a value. Refs: HA & DR plan PR #2a (user-facing critical path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): zero-downtime hardening for base operator tier Apply the same HA pattern (replicaCount=2, PDB minAvailable=1, hostname topologySpread, RollingUpdate with maxUnavailable=0) to the base operators that the cluster's day-2 operations depend on: - cert-manager: controller, webhook (request path, blocks all cert issuance), and cainjector each get topologySpread + strategy on top of the existing replicas/PDB - trust-manager: add topologySpread on top of existing replicas/PDB - cloudnative-pg: operator owns the database StatefulSets, runs the validating/mutating webhook for Cluster CRs. Add updateStrategy + topologySpread + sibling PDB (chart has no built-in PDB) - reloader: enable HA (chart requires explicit reloader.enableHA=true for leader election), bump replicas, add topologySpread + PDB - metrics-server: powers HPA + VPA + kubectl top — was running with empty values (replicas=1, no PDB). Add full HA stack - vpa: harden the admission controller (webhook on the request path); recommender + updater stay singletons (leader-elected) - cilium: add operator PDB + updateStrategy and PDBs for hubble relay/ui (replicas were already templated) flux-operator and goldilocks intentionally left as-is. flux-operator's chart does not expose a replica count; it is a leader-elected singleton in the recommended deployment. goldilocks is an advisor-tier dashboard without HA-grade requirements. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): zero-downtime hardening for omni infrastructure operators Apply HA pattern to the Hetzner-specific infrastructure that runs only on the Omni clusters (dev/prod). All three are leader-elected, so running 2 replicas means failover is instant instead of waiting on lease expiry. - hcloud-cloud-controller-manager: replicaCount=2, podAntiAffinity across hostname (chart exposes affinity but not topologySpread), RollingUpdate maxSurge=1/maxUnavailable=0, sibling PDB (chart has no native PDB support) - hcloud-csi controller: replicaCount=2, native podDisruptionBudget enabled with minAvailable=1, podAntiAffinity across hostname. CSI node DaemonSet stays at chart defaults (it's already one-per-node) - origin-ca-issuer: replicaCount=2, sibling PDB (chart only exposes replicaCount). Cloudflare origin certs are issued through this on cert renewal — single-replica means a node loss during rotation blocks issuance until reschedule kubelet-serving-cert-approver intentionally skipped: it pulls a standalone-install.yaml from upstream by URL and would need to be restructured to apply patches. Acceptable singleton for now — it only runs on bootstrap and node join. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(ha): make local CoreDNS survive node disruption CoreDNS in providers/docker is a hand-rolled Deployment (replicas=2 already, with podAntiAffinity). Add a PDB minAvailable=1 so a node drain can never evict both replicas at once, and tighten the deployment strategy to maxSurge=1/maxUnavailable=0 so DNS resolution stays continuous across rolls. Local Docker clusters share the same DR/HA test surface as Omni so this stays close to the prod-equivalent kube-system DNS posture. Refs: HA & DR plan PR #2b (operator tier) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(dr): wire R2 bucket + Omni etcd backups (PR #3) Configures Cloudflare R2 as the off-cluster backup destination shared by Omni etcd snapshots (this PR), Velero (PR #4), and CloudNativePG (PR #4). Single bucket, separate prefixes per consumer. Repo changes: - k8s/bases/variables/variables-base-config-map.yaml: add r2_endpoint, r2_region, r2_bucket, and per-consumer prefix keys so downstream HelmReleases can reference them via Flux postBuild substitution - k8s/bases/variables/variables-base-secret.enc.yaml: add SOPS-encrypted placeholders r2_access_key_id and r2_secret_access_key. Real values must be rotated in via 'sops --set' before the credentials are used -- documented in the runbook - docs/dr/omni-etcd-backups.md: bucket settings (object lock, versioning, SSE-S3), scoped-token recipe, omnictl set-backup invocation, retention policy (14 daily + 4 weekly long-term), restore overview Why R2: zero egress, S3-compatible (works with Omni's built-in S3 backup target as-is, no proxy), <$0.05/mo for the etcd portion. Why no in-repo manifests: Omni manages the cluster control plane externally and writes snapshots directly to S3 -- there is no in-cluster cron or workload to deploy. This PR is intentionally scoped to the shared R2 wiring + the operator-facing config doc. Local clusters intentionally skipped: ephemeral, single-CP, and fully reconstructed from this repo on every 'ksail cluster create'. Refs: HA & DR plan PR #3 (Omni etcd backups -> R2) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1376 — please merge that one first; this branch's diff against
mainwill shrink to just the operator-tier changes once the base PR lands.This is PR #2b of the HA & DR plan: same zero-downtime pattern applied to the operator/control-plane tier so a node drain or rolling Talos upgrade no longer leaves the cluster's day-2 systems with a window of unavailability. PR #2a (#1376) covered the user-facing path; this covers everything that runs the cluster itself.
What changed
Same pattern as #1376 (
replicaCount=2variable-driven, PDBminAvailable=1,topologySpreadConstraintsonkubernetes.io/hostnamewithScheduleAnyway,RollingUpdatewithmaxSurge=1, maxUnavailable=0).Base operators
cert-manager(controller / webhook / cainjector)trust-managercloudnative-pgClusterCRsreloaderenableHA: truefor leader electionmetrics-serverkubectl top; was running with empty values (replicas=1, no PDB)vpaadmission controllerciliumoperator + hubble relay/uiOmni-only operators
hcloud-cloud-controller-managertopologySpread) + sibling PDBhcloud-csicontrollerpodDisruptionBudget.create=true, minAvailable=1origin-ca-issuerreplicaCount)Local Docker provider
coredns(hand-rolled Deployment): sibling PDB + tighter strategy (maxUnavailable: 0).Intentionally skipped
flux-operator— chart does not expose a replica count; recommended deployment is a leader-elected singleton.kubelet-serving-cert-approver— pulls upstreamstandalone-install.yamlby URL; would need restructuring to apply patches. Acceptable singleton (only runs on bootstrap / node join).goldilocks— advisor-tier dashboard, not request-path; chart has no straightforward HA path.testkube— low criticality.cilium-agent,hcloud-csi-node) — chart-managedupdateStrategy.maxUnavailabledefaults are sane.Validation
kubectl kustomizesucceeds for all 13 component dirs and forclusters/{local,dev,prod}.ksail cluster createand assertkubectl get pdb -Ashows the new PDBs.Type of change
Notes
devantler/plan-ha-dr-clusters(feat(ha): zero-downtime hardening for user-facing critical path #1376). After that merges, GitHub will retarget this PR ontomainautomatically.