Every prod deploy rolling-reboots the whole fleet: ksail cluster update invalidates the GHCR node proof #3042

🤖 Generated by the Agentic Engineer

Evidence

Since the Cilium rollout gate was released (#3035, merged 2026-08-09T20:00Z) every deploy runs 🔄 Update cluster (Talos machine config) again — and every deploy now rolling-reboots all seven nodes. Deploy durations, newest first:

The gate acted as an accidental A/B toggle: with cluster update skipped the deploy is ~5 min, with it running it is ~an hour.

Root cause, isolated within a single run (run 31333379409)

The same script runs before and after cluster update, so one run contains its own control:

The cluster is fully converged at Stage. ksail cluster update then rewrites flux-system/ksail-registry-credentials away from the Git/SOPS value and invalidates every node's v2 proof, so select_talos_node_targets selects all seven in reboot mode and the reassert performs a serial rolling reboot at ~5–6 min per node.

That the Secret gets rewritten is known and expected — AGENTS.md says the bridge "reasserts it after cluster update in case KSail rewrites its managed Secret". What appears unintended is that the rewrite also costs the node proof, since the proof annotations are Talos machine config (machine.nodeAnnotations) and cluster update re-renders machine config from ksail.prod.yaml and its patches. Two writers, one config: the bridge writes the proof, cluster update drops it, and the bridge then rebuilds it the expensive way.

Impact

Roughly 45–60 minutes added to every merge, on a repo whose main is behind a merge queue, so this serializes all delivery. It also reboots all seven production nodes — including all three control-plane members — on every merge, which is real availability risk taken repeatedly for no state change: the fleet is proven current 5 minutes earlier in the same run.

Expected

A cluster update that changes nothing about the registry credential should leave the node proof intact, so the reassert converges without rebooting anything. Directions worth evaluating (deliberately not pre-judged — this needs a decision, and one of them may be wrong for reasons the evidence here does not show):

1. Carry the proof annotations in the machine config ksail cluster update itself renders, so a re-render preserves rather than drops them.
2. Key node staleness on something cluster update does not rewrite — e.g. reading the node's effective RegistryAuthConfig rather than an annotation the same command owns.
3. Have the bridge re-apply the proof annotations without a reboot when the credential is unchanged and only the annotation was lost — the reboot exists because containerd loads registry auth at process start, which does not apply when the auth did not change.

Acceptance criteria

• A deploy whose credential is unchanged completes without rebooting any node.
• A deploy whose credential did change still reboots the affected nodes and proves an uncached pull (the existing guarantee must not regress) — pinned by a test.
• Before/after deploy durations recorded in the PR.

Rough size: M–L, and it needs a decision between the directions above before implementation. Related: #3039 and #3041 address the fence overhead, which is ~60 s and a different, much smaller cost — they do not address this.