longhorn stale-node-cleanup CronJob has never run: distroless kubectl image has no /bin/sh #2384

🤖 Generated by the Daily AI Assistant

Problem

k8s/providers/hetzner/infrastructure/controllers/longhorn/cron-job-stale-node-cleanup.yaml (shipped in #2341 for #2024) runs /bin/sh -c <script> in registry.k8s.io/kubectl:v1.36.2 — a distroless image with no shell — so every scheduled run fails with StartError exit 128. The self-healing cleanup of orphaned Longhorn Node CRs has never actually executed; orphaned CRs from autoscaler scale-downs still accumulate. Observed live 2026-07-02 (interactive prod analysis; re-verify before acting).

Proposed direction

Keep the script but give it a runtime that has both a shell and kubectl. Options, preferred first:

1. bitnami/kubectl (shell + kubectl, widely used) pinned by digest;
2. keep registry.k8s.io/kubectl but replace the shell script with a kubectl-only invocation (no sh) if the logic can be expressed as a single kubectl call with jsonpath/go-template — it currently loops, so likely not;
3. an initContainer copying kubectl into a busybox runtime (avoid — complexity).

Whichever image is chosen must pass the verify-app-images/registry policies and be Renovate-trackable.

Rough size

S — one image swap + a validated manifest; confirm the next scheduled run completes and deletes a known orphaned Node CR.