Skip to content

PDBs with minAvailable: 1 over ≤1-replica workloads make nodes undrainable (blocks ksail recycles & rolling reboots) #1880

Closed

Description

@devantler

A prod ksail cluster update failed when ksail's autoscaler-node recycle hung
draining cert-manager/origin-ca-issuer for the full 2-minute eviction timeout:

evicting pod cert-manager/origin-ca-issuer-7c6f64969c-m4m59
error when evicting pods/"origin-ca-issuer-..." -n "cert-manager" (will retry after 0s): Cannot evict pod as it would violate the pod's disruption budget.
...
✗ failed to apply updates: ... drain node autoscale-cx33-...: global timeout reached: 2m0s

Root cause

origin-ca-issuer's PodDisruptionBudget sets minAvailable: 1 and is applied
unconditionally, but its Deployment replica count is variable
(replicaCount: ${origin_ca_issuer_replicas:=2}) and is set to 0 in prod
(intentionally disabled — "Non-essential services scaled to 0 … Disabled until
we actually need Cloudflare Origin certs."
).

  • k8s/providers/hetzner/infrastructure/controllers/origin-ca-issuer/pod-disruption-budget.yaml → minAvailable: 1
  • k8s/providers/hetzner/infrastructure/controllers/origin-ca-issuer/helm-release.yaml → replicaCount: ${origin_ca_issuer_replicas:=2}
  • k8s/clusters/prod/bootstrap/variables-cluster-config-map.yaml → origin_ca_issuer_replicas: "0"

A minAvailable: 1 PDB over a workload running fewer than 2 ready replicas
permits zero voluntary disruptions, so any node hosting its pod can never be
drained. The eviction API loops indefinitely until the drain timeout aborts —
deadlocking node recycles and rolling reboots.

The correct pattern already exists in this repo:
k8s/bases/apps/fleetdm/pod-disruption-budget.yaml uses minAvailable: 0
explicitly "to allow KEDA to scale the deployment to zero replicas." That same
reasoning just wasn't applied to origin-ca-issuer or the single-replica
controllers.

Fix options (pick one per workload)

  1. Scale the PDB with the Deployment — template minAvailable (or drop the PDB)
    when replicas ≤ 1, mirroring the existing fleetdm minAvailable: 0
    precedent for scale-to-zero services.
  2. Use maxUnavailable: 1 instead of minAvailable: 1 — always permits
    draining one pod regardless of replica count.
  3. For genuinely-HA controllers, guarantee a ≥2 ready-replica floor (and keep
    minAvailable: 1).

Audit

Sweep all hand-authored PDBs so minAvailable is strictly below the guaranteed
ready-replica floor:

  • ❌ origin-ca-issuer — minAvailable: 1, replicas 0 in prod (this incident)
  • ❌ cloudnative-pg — minAvailable: 1, cloudnative_pg_replicas: "1"
  • ❓ auth-proxy, hcloud-cloud-controller-manager, coredns — verify replica floors
  • ✅ fleetdm — already minAvailable: 0
  • ✅ homepage — minAvailable: 1, homepage_replicas: "2"

Context

Pairs with the ksail-side change that stops recycling autoscaler nodes for
in-place-applicable config changes (so only genuine boot-image / reboot changes
drain). That reduces how often a drain is attempted, but a drain is still
mandatory for a Talos OS bump — so the platform must guarantee its workloads are
drainable for those recycles to ever complete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions