Repository navigation
fix(deploy): hcloud server-limit wedges the ENTIRE merge queue (autoscaler snapshot build needs a free server slot) #2419
Description
🤖 Generated by the Daily AI Assistant
Problem
Every merge_group 🚀 Deploy to Prod run fails while ensuring the autoscaler Talos snapshot:
✗ failed to apply updates: failed to ensure autoscaler config secret: looking up snapshot image for
autoscaler secret: failed to ensure Talos snapshot: talos snapshot build failed: creating the temporary
server failed: server limit reached (resource_limit_exceeded)
The snapshot build spins up a temporary hcloud server, but the project is at its server limit (10 servers: 3 control-plane + 3 static workers + 4 autoscale-cx33). With no free slot the temp server can't be created, so the snapshot (Talos v1.13.5, schematic df884f4f…) never builds.
This wedges the ENTIRE merge queue, not just Talos-config PRs. deploy-prod ensures the autoscaler config secret on every deploy, and that ensure step always looks up (and, when missing, builds) the snapshot image — independent of what the PR changed. Live proof today: it evicted #2416 and #2417 (Talos-config), and also #2418 — a workflows-only pin bump with a zero manifest diff — all with the identical resource_limit_exceeded on schematic df884f4f. So the earlier "non-Talos/workflows-only PRs still merge" assumption was wrong.
Acute impact: main is currently red (the TODOs workflow, healed by #2418→actions v9.0.0) and there is no path to green until a server slot frees, because #2418 itself cannot clear the queue. The autoscale pool has decayed 7→4 nodes (VPA recovery is underway) but is still at the project limit.
Proposed directions (pick one/combine)
- Raise the hcloud project server limit by a couple of slots (maintainer-only account action) — immediate unblock; the v1.13.5 snapshot builds once, then is reused and pressure eases.
- Check for / clean orphaned temp snapshot-build servers from prior failed builds accumulating against the limit (needs hcloud access).
- Pre-build/cache the Talos snapshot out-of-band (scheduled job, or make
deployreuse an existing image and only build when genuinely absent) so the prod hot path never has to create a temp server against a full project — the durable fix (ksail deploy-mechanism hardening). - Shed over-provisioning — the node inflation is the underlying cause; continue letting VPA/autoscaler decay so a slot frees naturally.
Rough size
S–M. Direction 1/2 is a one-off maintainer/account action (immediate); 3 is deploy-mechanism hardening in ksail; 4 is operational and already trending down.
Impact
Blocks all merges to main (queue fully wedged) and keeps main red until a server slot frees. Highest-priority acute blocker.
Activity
[-]fix(deploy): hcloud server-limit blocks the autoscaler Talos snapshot build, wedging the merge queue for Talos-config PRs[/-][+]fix(deploy): hcloud server-limit wedges the ENTIRE merge queue (autoscaler snapshot build needs a free server slot)[/+]devantler commentedon Jul 3, 2026
Live update + remediation linkage. The chosen fix is structural slot reduction, not raising the limit: #2420 (drop cx33 as an overflow node type — cx33/8G nodes can never scale down, ~3Gi of DaemonSets alone exceed the 50% scale-down threshold) plus ksail#5756 (autoscaler scale-down-eligibility knobs). That is already taking effect — the autoscale pool dropped from 4 → 2 cx33 during the last hour, so prod is now 8 servers (3 control-plane + 3 static workers + 2 cx33), freeing hcloud slots.
Deadlock to note: #2420 is a
ksail.prod.yamlchange that only lands via adeploy-prod, which is itself queue-wedged — so the live cluster-autoscaler patch (or an out-of-band apply) carries it until #2420 can merge.Unblock order once a slot is free: re-enqueue #2418 (the promoted
TODOs-red heal, workflows-only) first — a successful deploy also builds the missing v1.13.5 Talos snapshot, unblocking the rest — then #2416/#2417. Re-enqueue of #2418 from the unattended run was classifier-gated (prod-deploy path); leaving it promoted + CLEAN for the maintainer or the next run to enqueue now that slots are freeing.devantler commentedon Jul 3, 2026
Update — the hcloud server-limit wedge has relieved; a different blocker is now on top. The 17:26Z merge_group deploy found the Talos snapshot (ID 404452940), so the temp-server slot is available (prod is down to 8 nodes after the cx33 drop). The queue is now blocked by a server-side-apply field-manager conflict on the live cluster-autoscaler Deployment's
.command— left by the out-of-bandkubectl-patchthat dropped cx33 from--nodesahead of #2420. Deploy-prod'sksailApply can't take.command, so every merge_group deploy (and the heal job) fails.Fix + one-click posted on #2420 (strip the stale
kubectl-patchmanaged-fields entry, then re-enqueue #2420). The hcloud-limit hardening this issue tracks (pre-build snapshot / raise limit / clean orphaned temp servers) is still worth doing for resilience, but is no longer what's holding the queue right now.devantler commentedon Jul 16, 2026
Recurred today (2026-07-16) — identically, for a new Talos version
This is not a one-off. It is back with v1.13.6 (schematic
df884f4f…), same error, same mechanism:Timeline (from
merge_grouprun history, not impression): everymerge_groupCI run succeeded through 10:21Z (#2662). #2662 — the Talos v1.13.5 → v1.13.6 bump — merged at 10:38Z. Everymerge_grouprun since has failed: #2652 (12:33Z), #2635 (12:45Z), #2665 (12:49Z), and again #2665 (14:12Z), which was evicted from the queue a second time.Why the bump is the trigger: while the pinned Talos version's snapshot already existed,
findExistingSnapshothit and no temporary server was ever needed. A version bump invalidates that lookup, so every deploy must now build the snapshot — and the build needs a temporary server the project has no room for. This makes it a standing trap: every future Talos bump re-wedges the queue, and Renovate proposes those regularly.Escalation vs. 07-03:
🩹 Heal Prodis failing too. The heal path re-deploysmainthrough the same ensure step, so it hits the identical failure (run29505546754, both🚀 Deploy to Prodand🩹 Heal Prod= failure). There is currently no working CD path to prod, not merely a blocked queue.Root cause, precisely
The invariant is in the committed config, which is self-documenting about it:
provider.hetzner.serverLimit: 10— the account's hard cap.autoscaler.node.maxNodesTotal: 10— "the Hetzner account cap: 6 static (3 control-planes + 3 workers) + up to 4 autoscaler nodes".maxNodesTotalis set equal to the account server limit, so a fully scaled-out cluster is entitled to consume every slot the account has — leaving zero headroom for the temporary snapshot-build server. The deadlock isn't a flake or a leak; the temp server fails at creation, so nothing is orphaned. It is a budgeting invariant: the cluster's ceiling must sit strictly below the account cap, because the deploy hot path needs one slot the cluster doesn't own.This also explains why direction 4 (“let the pool decay”) appeared to fix it on 07-03 — decay frees a slot, so the symptom clears while the invariant stays broken and waits for the next bump.
Recommendation — one slot of headroom (direction 3/4, made durable)
Reserve one slot: hold
maxNodesTotalone belowserverLimit(9 <= 10). One line, declarative, and it makes the deadlock unreachable rather than waiting for decay. Cost: peak autoscale capacity 4 → 3 nodes. If that peak is genuinely needed, the equivalent is raise the hcloud account limit to 11+ and keepmaxNodesTotalone below it (direction 1, and it preserves capacity).I could not ship this myself:
ksail.prod.yamlis protected — never modify per this repo'sAGENTS.md, so the change is the maintainer's. I prepared and reverted it rather than cross that guardrail. I also can't free a slot directly (the local hcloud token is no longer valid).Unblocking action (any one is sufficient, and immediate): free a single server slot — raise the account limit, or remove one autoscaler node. The v1.13.6 snapshot then builds once and is reused, and the queue drains on its own.
Suggested follow-up once unblocked: add a CI guard asserting
maxNodesTotal < serverLimit, so the invariant can't silently regress.devantler commentedon Jul 16, 2026
Adding the prevention side — filed devantler-tech/ksail#6171.
The immediate unwedge here is a capacity decision (raise the cap, or free a slot). But the reason this configuration was reachable at all is a KSail validation gap:
pkg/apis/cluster/v1alpha1/validation.go:466only rejectsreachableTotal > serverLimit, so a total equal to the cap validates clean — leaving KSail no slot for the temporary server it needs itself to build the Talos snapshot.autoscaler_test.go:830currently pins that as valid ("maxNodesTotal equal to serverLimit is valid despite large pools").So KSail happily blesses a config that deadlocks its own deploy path on the next Talos bump. #6171 proposes reserving that slot at validation time, so the failure surfaces as a config error instead of a mid-deploy
resource_limit_exceeded.Live state as of ~14:35Z: the merge queue is now empty — all three enqueued PRs (#2665, #2652, #2635) were evicted, each by this same failure, and
🩹 Heal Prodfails identically. So there is currently no working CD path to prod, not just a blocked queue.I did not re-queue anything: with the cluster confirmed at exactly the cap, re-queuing re-hits the same failure. I also did not touch
ksail.prod.yaml— platform'sAGENTS.md ## Maintenancelists it as protected, so themaxNodesTotalchange is yours to make.botantler-1 commentedon Jul 17, 2026
🎉 This issue has been resolved in version 1.117.4 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀