feat(autoscaling): add the Kubernetes Descheduler to reclaim Karpenter-style consolidation on Hetzner #2468

🤖 Generated by the Daily AI Assistant

Problem

Karpenter is the tool people reach for when they want active node consolidation — continuously repacking pods onto fewer nodes and retiring the under-utilized ones to cut cost. But Karpenter has no Hetzner Cloud provider (upstream supports AWS, Azure, Alibaba and now OCI; Hetzner is not on the list), so we cannot adopt it here.

Our Hetzner node-lifecycle stack today is:

• Cluster Autoscaler (Hetzner cloud provider) — installed via ksail.prod.yaml (autoscaler.node.*); scales the autoscale-cx* pool up on Pending pods and down on idle nodes.
• CapacityBuffer over-provisioning (k8s/providers/hetzner/infrastructure/overprovisioning/) — keeps one warm spare node so real pods never sit Pending.
• VPA — right-sizes pod requests (k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/).

The gap: Cluster Autoscaler never repacks running pods. It only removes a node once every pod on it happens to be independently reschedulable and the node has sat under the utilization threshold long enough. It has no bin-packing / rebalancing pass. So over time the cluster fragments — pods spread thin across autoscale-cx* nodes that CA then can't drain, and stateful pods land on autoscale nodes and pin them up. We've already hit the concrete symptom: a CNPG primary (wedding-db-10) scheduled onto a cx43 autoscale node instead of a baseline node and effectively blocked that node from scaling down.

Proposed direction

Add the Kubernetes SIG Descheduler (kubernetes-sigs/descheduler) — the cloud-agnostic, Talos-compatible component that provides exactly the missing half. It only evicts pods (respecting PDBs); the existing Cluster Autoscaler then reclaims the emptied nodes. Cluster Autoscaler (provision/deprovision) + Descheduler (rebalance/consolidate) ≈ Karpenter consolidation, on Hetzner.

Deploy it as a GitOps controller in the Hetzner infrastructure layer (Helm chart descheduler from kubernetes-sigs, pinned via HelmRelease, in continuous Deployment mode with --descheduling-interval, or a CronJob). Start conservative and expand policy as we gain confidence:

• HighNodeUtilization — the consolidation/bin-packing strategy explicitly designed to pair with Cluster Autoscaler: evict pods off under-used nodes to concentrate load so CA can scale those nodes down. This is the direct Karpenter-consolidation analog.
• RemovePodsViolatingNodeAffinity / ...TopologySpreadConstraint / ...InterPodAntiAffinity — pull stateful pods (CNPG, etc.) back onto baseline nodes and re-spread after scale events (addresses the wedding-db pin-up above).
• RemoveDuplicates — spread replicas of the same controller across nodes.

Guardrails (must-haves in the implementing PR):

• Respect PodDisruptionBudgets (default) — the platform relies on PDBs (coredns, coroot, …).
• Scope carefully around stateful workloads — CNPG / Longhorn / OpenBao / clickhouse-keeper / SPIRE evictions trigger failovers, so exclude or tightly bound those namespaces; keep evictLocalStoragePods: false.
• Don't fight the CapacityBuffer — exclude the overprovisioning namespace and system-critical priority classes; ensure the descheduler doesn't churn the warm spare.
• Honor node taints/labels so it never evicts from control-plane / baseline-pinned placements it shouldn't.

Rough size

Small–medium. It's one new HelmRelease + values + RBAC in k8s/providers/hetzner/infrastructure/controllers/descheduler/, wired into the controllers kustomization.yaml, with a conservative single-strategy policy to start. No cluster run needed to validate — both overlays must kubectl kustomize build and pass ksail workload validate + the Kubescape scan, per the repo's static-check contract. Roll out one strategy first (HighNodeUtilization), observe, then widen.

Acceptance criteria

• Descheduler runs in the Hetzner cluster as a pinned GitOps HelmRelease, health-gated in the infrastructure layer.
• Policy respects PDBs, excludes the overprovisioning namespace + system-critical priorities, and is scoped to avoid churning stateful workloads.
• At least the HighNodeUtilization (CA-consolidation) strategy is enabled and demonstrably lets Cluster Autoscaler retire an under-utilized autoscale-cx* node it previously couldn't drain.
• Both overlays build and pass CI static checks.

Part of #2043 (resilient, current, well-abstracted GitOps platform). Relevant to the autoscale-node churn seen around #2419 / #2363.