You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PDBs with minAvailable: 1 over ≤1-replica workloads make nodes undrainable (blocks ksail recycles & rolling reboots) #1880
A prod ksail cluster update failed when ksail's autoscaler-node recycle hung
draining cert-manager/origin-ca-issuer for the full 2-minute eviction timeout:
evicting pod cert-manager/origin-ca-issuer-7c6f64969c-m4m59
error when evicting pods/"origin-ca-issuer-..." -n "cert-manager" (will retry after 0s): Cannot evict pod as it would violate the pod's disruption budget.
...
✗ failed to apply updates: ... drain node autoscale-cx33-...: global timeout reached: 2m0s
Root cause
origin-ca-issuer's PodDisruptionBudget sets minAvailable: 1 and is applied unconditionally, but its Deployment replica count is variable
(replicaCount: ${origin_ca_issuer_replicas:=2}) and is set to 0 in prod
(intentionally disabled — "Non-essential services scaled to 0 … Disabled until
we actually need Cloudflare Origin certs.").
A minAvailable: 1 PDB over a workload running fewer than 2 ready replicas
permits zero voluntary disruptions, so any node hosting its pod can never be
drained. The eviction API loops indefinitely until the drain timeout aborts —
deadlocking node recycles and rolling reboots.
The correct pattern already exists in this repo: k8s/bases/apps/fleetdm/pod-disruption-budget.yaml uses minAvailable: 0
explicitly "to allow KEDA to scale the deployment to zero replicas." That same
reasoning just wasn't applied to origin-ca-issuer or the single-replica
controllers.
Fix options (pick one per workload)
Scale the PDB with the Deployment — template minAvailable (or drop the PDB)
when replicas ≤ 1, mirroring the existing fleetdmminAvailable: 0
precedent for scale-to-zero services.
Use maxUnavailable: 1 instead of minAvailable: 1 — always permits
draining one pod regardless of replica count.
For genuinely-HA controllers, guarantee a ≥2 ready-replica floor (and keep minAvailable: 1).
Audit
Sweep all hand-authored PDBs so minAvailable is strictly below the guaranteed
ready-replica floor:
Pairs with the ksail-side change that stops recycling autoscaler nodes for
in-place-applicable config changes (so only genuine boot-image / reboot changes
drain). That reduces how often a drain is attempted, but a drain is still
mandatory for a Talos OS bump — so the platform must guarantee its workloads are
drainable for those recycles to ever complete.
Activity
fix(infra): make the whole PDB fleet uniformly maxUnavailable: 1
botantler-1 commentedon Jun 8, 2026
🎉 This issue has been resolved in version 1.40.2 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀
fix(infra): harden eviction safety — drain-safe PDBs + HA replica-flo…
feat(openbao): switch to 3-node Raft HA (Integrated Storage) (#1907)
feat(ha): add drain-safe PDB to the OpenBao Raft cluster (#2028)
feat(ha): add drain-safe PDBs to the remaining critical controllers a…
fix(external-dns): harden single-replica writer against node failure (#…