Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction

🤖 Generated by the Agentic Engineer

Evidence

The prod deploy's node-safety transaction races the cluster autoscaler and refuses to proceed when
an autoscale node changes underneath it. Three merge-group deploys have now died this way in under
two hours, each evicting its PR from the merge queue:

When (UTC)	Run	Failure
2026-08-16 22:39	#3171	node claimed mid-transaction was replaced — `identity changed … refusing image verification`
2026-08-16 23:02	#3171	evicted again; orphaned `Lease flux-system/ghcr-auth-refresh` held past expiry
2026-08-17 00:16	#3171 (run `31980786395`)	`Talos node autoscale-cx43-2834820e83860317 identity changed, cordon ownership changed, or scheduling safety state changed before revision marker; refusing the mutation.`

The guard itself is correct — refusing to mutate a node whose safety state moved is the right
call. The defect is that nothing reconciles the transaction with the autoscaler, so an ordinary
scale event is indistinguishable from a genuine safety violation.

Residue outlives the run. autoscale-cx43-2834820e83860317 is still cordoned=true with no
ghcr-auth-drain-owner annotation — a cordon with no owner, left behind after the refusal. A
cordoned node with no owner is exactly the state the next deploy will trip over again.

🩹 Heal Prod also failed in this run, which is the free discriminator for whose fault it is:
a base-state fault, not anything in the PR's diff.

Why it matters

Every platform merge is blocked while this recurs. The queue evicts on the failed merge_group,
so the PR bounces back to OPEN and the next re-queue hits the same race. Re-queuing without a fix is
futile by the contract's own merge-queue rule.

Prod itself was never at risk: all three deploys died before publish_platform_manifest, and prod is
currently 11/11 Kustomizations Ready — so this is a delivery-lane outage, not a production
incident.

Expected behaviour

A deploy is not evicted by a routine autoscaler scale event. Either the transaction tolerates an
autoscale node changing state while it holds no claim on that node, or it excludes autoscaler-managed
nodes from the safety marker, or it retries once against the refreshed node set.

Separately, a refused mutation leaves no cordoned node without a drain owner — the refusal path
should release its own cordon, the way the node fence self-releases elsewhere.

Acceptance criteria

A merge-group deploy completes while an autoscale node is added or removed during the run.
After a refused mutation, no node is left cordoned with ghcr-auth-drain-owner absent.
The failure message distinguishes a routine autoscale event from a genuine safety violation.

Rough size: M.

Related: #3178 (stranded cx33 nodes consuming the autoscaler's budget) — a different defect on the
same autoscaler surface, not a duplicate of this one.

🔗 Related PRs

#2985 - fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one [merged]
#3034 - feat(ci): make an orphaned GHCR deploy fence diagnosable and recoverable [merged]
#3037 - fix(cilium): refuse a zero-count gate release and correct the runbook order [merged]
#3077 - fix(ci): reclaim a drain fence whose owning transaction is provably gone [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Root cause located: the taint-set comparison has no exclusion for autoscaler-owned taints

The guard is node_scheduling_state_is_safe_to_reboot in scripts/refresh-flux-ghcr-auth-safety.sh.
It compares the node's entire taint set against the set captured at selection, excluding exactly
one key:

node.kubernetes.io/unschedulable (it mirrors spec.unschedulable, so it is compared separately)

The lifecycle variant node_scheduling_state_is_safe_while_lifecycle_taints_clear excludes three
more — node.kubernetes.io/not-ready, node.kubernetes.io/unreachable,
node.cilium.io/agent-not-ready.

The cluster autoscaler's own taints are excluded nowhere, in either predicate:

DeletionCandidateOfClusterAutoscaler (PreferNoSchedule) — added when a node merely becomes a
candidate for removal, and removed again when it stops being one. No node deletion required.
ToBeDeletedByClusterAutoscaler (NoSchedule) — added when removal actually starts.

So an ordinary autoscaler decision mutates the taint set, the comparison fails, and the guard reports
identity changed, cordon ownership changed, or scheduling safety state changed — which is why three
evictions with three genuinely different proximate causes all produced similar-looking messages.

Why this fires so readily here

cluster-autoscaler-hetzner-cluster-autoscaler (kube-system) re-evaluates scale-down on a ~10
second loop — its log emits Starting scale down at that cadence continuously. Its own counters at
the time of the 00:16Z eviction:

lastScaleUpTime          = 2026-08-17 00:09:41 UTC
lastScaleDownDeleteTime  = 2026-08-17 00:30:47 UTC

The deploy selected and held a node from the autoscaler-managed class and died at 00:16Z — between a
scale-up six minutes earlier and a scale-down delete fourteen minutes later. A transaction that holds
a node for minutes is racing a loop that re-decides every ten seconds, so this is expected to recur
rather than being bad luck.

One correction to the evidence above

The issue reads the leftover cordoned=true node with no ghcr-auth-drain-owner as residue from the
refusal. That is worth re-checking before it drives the fix: cordoning is also what the autoscaler
itself does to a node it is removing, so a cordon with no drain owner is the expected signature of
an in-flight scale-down, not necessarily a leaked cordon. The node in question has since been
reaped, so this cannot now be proven either way — but the acceptance criterion should be written
against a reproduced refusal rather than against that observation.

Implication for the fix

The guard is right to refuse to mutate a node that is being deleted — that is not the part to relax.
The defect is the blast radius: one node leaving the fleet aborts the whole deploy. The correct
behaviour is to deselect that node and continue with the rest, since an autoscaler-managed node
disappearing is a routine event with a specific, recognisable signature, not an unexplained safety
violation.

That also satisfies the third acceptance criterion directly: once autoscaler-owned taints are
recognised by name, the failure message can distinguish "this node is being scaled away" from
"something changed that I cannot explain", instead of collapsing both into one string.

Excluding the two autoscaler keys from the comparison without deselecting the node would be the
wrong fix — it would let the transaction proceed to reboot a node the autoscaler is deleting.

chore: agent-claim #3184 nonce=cfd7bf65981f94495efec7a630e80aee

fix(deploy): normalize both sides of the scheduling-taint comparison

🤖 Generated by the Agentic Engineer

Confirmed from the autoscaler's own log — it was the advisory taint, not a deletion

The cluster-autoscaler log names the exact sequence on autoscale-cx43-2834820e83860317:

00:11:26.656  Successfully added DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317
00:15:43.035  Releasing taint {Key:DeletionCandidateOfClusterAutoscaler Value:1786925486 Effect:PreferNoSchedule}
00:15:43.048  Successfully released DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317
00:20:44.393  Successfully added DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317

The deploy's guard refused at 00:16 — 23 seconds after the release.

Two things this settles:

ToBeDeletedByClusterAutoscaler never appears anywhere in the window. The node was never
marked for actual deletion during the transaction. The eviction was caused entirely by the
advisory DeletionCandidateOfClusterAutoscaler marker (PreferNoSchedule) being released
mid-transaction.
The marker churns by design. It is added, released and re-added on the autoscaler's ~10s
re-evaluation loop, and its value is a regenerated timestamp — so it differs even key-for-key
between two adds. A transaction holding a node for minutes is near-certain to observe it move.

What shipped

#3185 excludes that advisory marker from the scheduling-taint comparison at all six normalization
sites (capture and comparison, both scripts), so the report and the reference predicate cannot
disagree. ToBeDeletedByClusterAutoscaler is deliberately left in the comparison, so a node
genuinely being deleted still fails closed — that is asserted by an existing test which passes
unchanged, and by a new negative-control case.

The PR also normalizes both sides of the comparison. A recovery journal written before this
exclusion existed still carries the marker in its captured initialTaints, so a one-sided
normalization would have fixed the fresh-deploy path and still refused on the recovery path a
killed deploy leaves behind.

Correction to this issue's "residue" claim

The cordoned=true node with no drain-owner annotation is not leaked residue from the refusal:
the autoscaler cordons the nodes it removes, so that is the expected scale-down signature. It has
since cleared on its own — the node is gone and its replacement is uncordoned and untainted.

Still open (not covered by #3185)

A node that is genuinely being deleted (ToBeDeletedByClusterAutoscaler, or a set
deletionTimestamp) still aborts the entire deploy rather than being deselected from the target set.
Failing closed is correct, but aborting is heavier than necessary — the deploy could drop that node
and continue against the rest. Filed separately.

🤖 Generated by the Agentic Engineer

Fresh occurrence — 2026-09-03, `platform#3547` evicted

This bit again today, so recording it as evidence the race is still live rather than filing a
duplicate.

PR: #3547 (renovate[bot], ksail v7.180.4), head cbdfe92279
Run: 33751933050 (merge_group, gh-readonly-queue/main/pr-3547-fe86befc…), 11:51:00Z
Failing job: 🚀 Deploy to Prod → CI - Required Checks
Error: Could not atomically claim and cordon Talos node autoscale-cx43-62cbca80e4c1f54d; refusing to drain it.
Confirmed the mechanism: that node no longer exists — kubectl get node autoscale-cx43-62cbca80e4c1f54d returns NotFound, while a different autoscale node
(autoscale-cx43-59ee3c84869749a0, age 7d23h) is Ready. So the autoscaler removed the node between
selection and the atomic claim, exactly the mid-transaction change this issue describes, and Deselect a node the autoscaler is deleting instead of aborting the whole deploy #3186
proposes deselecting rather than aborting.
Result: the PR was evicted (isInMergeQueue=false, no queue entry) and sat open ~50 minutes
looking merely "still queued".

⚠️ One diagnosis note for whoever picks this up: the same failed log contains two
::error:: lines about KUBE_CONFIG being stale and the cluster unreachable. Those are echoed
script source, not failures — the real error is the single ##[error] line above. Reading them as
the cause would send a run at credential rotation instead of the autoscaler race.

The deploy that merged #3543 at 11:29Z succeeded, so this remains probabilistic rather than a hard
block — which is why it keeps presenting as an unexplained "queued but never merged".

🤖 Generated by the Agentic Engineer

Measurement, 2026-09-30: the race recurred twice after the 09-03 report. It has not recurred since 2026-09-20.

Method. I read the annotations on every failed job of every failed platform merge_group run created on or after 2026-09-04: 37 runs. I matched any annotation mentioning a cordon claim or release. Control: the same query against the 09-03 run 33751933050 returns its Could not atomically claim and cordon … annotation, so the method sees this failure.

Findings

Run	Created (UTC)	Failure
`35503104796` (pr-3927)	2026-09-20 09:43	The node's identity changed before the credential patch, which the bridge correctly refused. The uncordon then failed with `NotFound`: the autoscaler had deleted the node mid-run.
`35505627116` (pr-3927 re-run)	2026-09-20 10:38	`Cordon ownership changed or scheduling safety state changed … refusing to uncordon it` on a different autoscaler node. That node no longer exists either.

The advisory-taint half of the root cause above is fixed on main. Every scheduling-state comparison now normalizes DeletionCandidateOfClusterAutoscaler.
Neither 09-20 failure is that mode. In both, an autoscaler node disappeared while the bridge held it. The first also shows a second, separate problem: after a node is deleted, the release step fails instead of recognizing that there is no fence left to release.
Denominator since then: there were 9 failed merge-group runs after 2026-09-20 10:39 and 4 after fix(deploy): avoid no-op GHCR auth controller restarts #4080 merged (2026-09-24 20:56Z). None matched. fix(deploy): avoid no-op GHCR auth controller restarts #4080 fences a node only when a real image pull needs the new credential, so the bridge now holds autoscaler nodes much less often. That lowers exposure but does not remove the race.

Next step for whoever picks this up: make the release step treat a NotFound node as released, since a deleted Node carries no cordon. Keep the refusal before the credential patch unchanged, because refusing to mutate a node whose identity changed is correct. Then decide whether the bridge should keep selecting autoscaler-pool nodes at all while Cluster Autoscaler can remove them.

chore: agent-claim #3184 nonce=7bcd7b589ee4e2db94d9e67bba298e9b

🤖 Generated by the Agentic Engineer

The remaining autoscaler race is delivered on current main d927e68. #3185 ignores only the advisory candidate taint, #3588 deselects a Node already gone before mutation, #3944 prevents Cluster Autoscaler from selecting a bridge-owned Node, and #4358 treats a successfully confirmed deletion during either fence-release window as no fence left to release while every forbidden, malformed, partial, or ambiguous read still fails closed.

The focused refresh-flux-ghcr-auth suite passes from that fresh main. Its controls retain the owner on a genuine release failure, reject mutation credit for a deleted Node, distinguish the routine autoscaler paths in their diagnostics, and refuse root cutover after a post-mutation disappearance. The earlier measured post-#3944 window recorded about 220 successful merge-group runs and no recurrence; #4358 closes the later release-cleanup diagnostic gap. Closing as completed rather than reopening work already present on main.

Navigation Menu

Repository navigation

Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction #3184

Evidence

Why it matters

Expected behaviour

Acceptance criteria

Root cause located: the taint-set comparison has no exclusion for autoscaler-owned taints

Why this fires so readily here

One correction to the evidence above

Implication for the fix

Confirmed from the autoscaler's own log — it was the advisory taint, not a deletion

What shipped

Correction to this issue's "residue" claim

Still open (not covered by #3185)

Fresh occurrence — 2026-09-03, `platform#3547` evicted

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction #3184

Description

Evidence

Why it matters

Expected behaviour

Acceptance criteria

Activity

coderabbitai commented on Aug 17, 2026

devantler commented on Aug 17, 2026

Root cause located: the taint-set comparison has no exclusion for autoscaler-owned taints

Why this fires so readily here

One correction to the evidence above

Implication for the fix

devantler commented on Aug 17, 2026

Confirmed from the autoscaler's own log — it was the advisory taint, not a deletion

What shipped

Correction to this issue's "residue" claim

Still open (not covered by #3185)

devantler commented on Sep 3, 2026

Fresh occurrence — 2026-09-03, platform#3547 evicted

devantler commented on Sep 30, 2026

devantler commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Fresh occurrence — 2026-09-03, `platform#3547` evicted