Longhorn: 7 stale node CRs from removed autoscale nodes + autoscaler cooperation gap #2024

🤖 Generated by the Daily AI Assistant

What

kubectl get nodes.longhorn.io -n longhorn-system on prod lists 14 nodes, 7 of which no longer exist in Kubernetes (Ready=False, condition: Kubernetes node missing: node ... has been removed from the cluster, lingering since ~Jun 4):

• autoscale-cx23-201c1b34581712fb, autoscale-cx23-54a5df0e9a38f469 (removed cx23 pool)
• autoscale-cx33-2ec36affa55a95d3, autoscale-cx33-3872b65b0451b9f8, autoscale-cx33-5fa978e3d6f5fc3, autoscale-cx33-6edbbe1a968cc9c, autoscale-cx33-7e115d698f978019

Each also has an orphaned InstanceManager CR. No replicas or engines reference any of them (verified), so they are pure stale state: UI/API noise, scheduler bookkeeping, and they will accumulate forever — the autoscaler creates fresh node names on every scale-up.

Why it needs a human

Longhorn deliberately never auto-deletes node CRs, and the cleanup is a live mutation (the Daily AI Assistant's permission policy blocks prod writes). Per stale node (webhook blocks deletion while schedulable):

for n in autoscale-cx23-201c1b34581712fb autoscale-cx23-54a5df0e9a38f469 \ autoscale-cx33-2ec36affa55a95d3 autoscale-cx33-3872b65b0451b9f8 \ autoscale-cx33-5fa978e3d6f5fc3 autoscale-cx33-6edbbe1a968cc9c \ autoscale-cx33-7e115d698f978019; do
  kubectl -n longhorn-system patch nodes.longhorn.io $n --type=merge -p '{"spec":{"allowScheduling":false}}'
  kubectl -n longhorn-system delete nodes.longhorn.io $n
done

(Longhorn's node controller garbage-collects the orphaned InstanceManager CRs once the node CR is gone.)

Root-cause angle worth deciding deliberately

This cluster runs the KSail-managed Cluster Autoscaler, but Longhorn's kubernetes-cluster-autoscaler-enabled setting is false. Longhorn replicas currently land on autoscale nodes (4–5 replicas per autoscale node right now), so every scale-down degrades volumes and triggers rebuild storms (the FailedRebuilding/FailedSnapshotPurge warning bursts in events; #1967's offline rebuilding mitigates but doesn't prevent). Two levers, both maintainer decisions:

1. Enable kubernetes-cluster-autoscaler-enabled: true (experimental) in the Longhorn HelmRelease defaultSettings so Longhorn annotates its components for CA cooperation instead of being force-evicted.
2. Keep replicas off autoscale nodes via Longhorn node tags / create-default-disk labels on baseline workers only — trades rebuild churn for less replica spread (3 baseline workers = exactly 3-replica capacity, degraded during any worker drain).

Happy to turn either into a PR once a direction is picked.