fix(deploy): let a deploy through when a failed deploy left one node on a recorded image

.github/workflows/ci.yaml

@@ -987,6 +987,8 @@ jobs:
               - 'scripts/tests/test-talos-render-kernel-args.sh'
               - 'scripts/register-talos-factory-schematic.sh'
               - 'scripts/verify-talos-schematic-readback.sh'
+              # The reviewed record that readback reads, so a record-only change runs its test (#4665).
+              - 'scripts/talos-boot-image-drift-accepted.tsv'
               - 'scripts/tests/test-talos-factory-schematic-registration.sh'
               - 'scripts/tests/test-talos-schematic-readback.sh'
               - 'scripts/verify-kubelet-config-readback.sh'

AGENTS.md

@@ -280,6 +280,8 @@ Production uses **Talos + Hetzner** via KSail's native Hetzner provider. KSail o
 - **Current-main recovery** — healing, manual CD and disaster recovery require their validated checkout to remain the current `main` tip before production changes and artifact promotion. A stale or unreadable revision stops recovery; restart from current `main` with all validation. See the [production recovery runbook](docs/operations/production-recovery.md).
 - **`.github/actions/deploy-prod`** — the composite action both regular deploy paths call (stage/verify all GHCR pull consumers → push → cosign-sign → attest SBOM + SLSA provenance → revalidate published artifact → Flux reconcile → exact published-revision Ready proof → Talos `cluster update` → final reassert → gateway route check), so the merge-queue and manual deploys can never drift. The last step fails the deploy when the gateway has not applied every HTTPRoute at its current generation within five minutes, when it rejected a route or a route names a missing Gateway, or when a running Cilium operator logged that its Gateway API controller failed to start (#4198). DR uses the same exact-revision wait and rollout guard. **No Cilium rollout gate is active**, so every deploy runs its `cluster update` Talos machine-config sync. The guard still runs on both paths and is machinery for the next staged rollout: it reads as active only while the `homogeneous-devices` component is referenced **and** carries `type: OnDelete`. While that holds it suspends Cluster Autoscaler, proves no provider-side node addition is in flight before publish, and skips `cluster update` until a reviewed completion or rollback artifact has applied — a skip that happens inside an otherwise-green deploy, so it is time-bounded against a `platform.devantler.tech/rollout-gate-activated:` marker declared beside the component reference: `scripts/report-cilium-rollout-gate-suppression.sh` **warns** from `warn_after_days` (7) and **fails the deploy** from `fail_after_days` (14). Resolve an expired gate by stepping the remaining Cilium agents onto the current DaemonSet revision or rolling the component back — raising either bound is not a resolution. Secrets are passed as inputs because composite actions cannot read `secrets`.
 
+- **Boot-image readback** — after `cluster update`, a deploy fails unless every node runs the Talos version and Image Factory schematic the checkout expects. A deploy that changes the image and fails after its first node leaves that node on the new image, and no deploy from `main` can pass until it is resolved. A reviewed, dated row in `scripts/talos-boot-image-drift-accepted.tsv` may accept one static node on one named image, reported as a warning; see [the runbook](docs/operations/boot-image-drift.md) (#4665).
+
 **Required GitHub Secrets:**
 
 - `GHCR_TOKEN` — long-lived PAT (owner: `devantler`) with `write:packages` scope, used only for OCI push/signing. It is **not** a pull credential.

docs/operations/boot-image-drift.md

@@ -0,0 +1,58 @@
+# A node left on another boot image by a failed deploy
+
+Every production deploy ends by reading back the Talos boot image of every node
+(`scripts/verify-talos-schematic-readback.sh`). It passes only when every node runs the Talos
+version and Image Factory schematic the checkout expects, and is Ready and schedulable.
+
+## The state this page is about
+
+A deploy that changes the image upgrades nodes one at a time. If it fails after the first node,
+that node keeps the new image while every other node keeps the old one. A deploy from `main` then
+fails the readback on that node, and redeploying does not help: at an unchanged Talos version the
+cluster update applies configuration without a reboot, so the node keeps its image.
+
+The readback prints one line per node when it fails. The node with a different schematic is the one
+left behind. Compare its schematic with the one the failed pull request defines:
+
+```bash
+git show <its-branch>:talos/factory-schematic.yaml | sha256sum
+```
+
+A match confirms where the image came from. No match means the cause is something else, and this
+page does not apply.
+
+## The reviewed record
+
+`scripts/talos-boot-image-drift-accepted.tsv` may accept **one static node on one named image
+until a date**, so that the repair can deploy:
+
+| Column | Value |
+|---|---|
+| `node` | The static node's exact name. An autoscaled node is never accepted: it is replaced, not repaired. |
+| `schematic` | The exact schematic the node reports. It cannot be the expected one. |
+| `issue` | The platform issue that tracks the repair. |
+| `expires` | The last day the row holds. |
+
+With a matching row the readback reports a `::warning::` for the node and ends with
+`PASS-WITH-ACCEPTED-DRIFT`, never with the plain `PASS`. Everything else still binds that node: the
+Talos version, Ready, schedulable and the node counts. The row covers that node and that image only;
+a second node on the same image, or the same node on any other image, fails as before.
+
+The record is refused, before any node is read, unless it is the header and at most one row with
+exactly four well-formed fields. The day after `expires` the row accepts nothing and the readback is
+strict again.
+
+## Adding and removing a row
+
+A row narrows a deploy safety check, so it is added by a pull request the maintainer approves, and
+that pull request is the first deploy that can pass. Set `expires` no further out than the repair
+needs.
+
+Remove the row in the pull request that repairs the node, or right after it. Once the node reports
+the expected image the readback says so with a `::notice::` on every deploy until the row is gone.
+
+## Limits
+
+- The record tolerates the state; it does not repair it. The node is repaired by landing the change
+  that defines its image, or by upgrading the node back.
+- It accepts one node. If a failed deploy left more than one node behind, repair them instead.

scripts/refresh-flux-ghcr-auth.sh

@@ -1716,7 +1716,11 @@ probe_node_runtime_pull() {
           imagePullPolicy: "Always",
           args: ["--version"],
           resources: {
-            requests: {cpu: "10m", memory: "16Mi"},
+            # The Pod names its node, so the kubelet alone admits it, and it refuses a Pod
+            # that asks for more memory than the node has left to request (#4664: a
+            # worker with 15 Mi left refused 16 Mi and stopped the deploy). The probe
+            # only prints a version, so it asks for the least the node can grant.
+            requests: {cpu: "1m", memory: "1Mi"},
             limits: {cpu: "100m", memory: "64Mi"}
           },
           securityContext: {

scripts/talos-boot-image-drift-accepted.tsv

@@ -0,0 +1,2 @@
+node	schematic	issue	expires
+prod-worker-1	93f5f26de33d6534dcd3444c2b244d125f8e72eac7d6b9eddd17e11046cf50d1	devantler-tech/platform#4665	2026-10-22

scripts/tests/refresh-flux-ghcr-auth/fake_kubectl_test.go

@@ -3007,6 +3007,14 @@ func fakeKubectlCreateRuntimeProbe(namespace, manifestFile string) int {
 		container["imagePullPolicy"] != "Always" || securityContext["allowPrivilegeEscalation"] != false {
 		return commandFailure(91, "runtime probe does not prove a private package pull")
 	}
+	// The probe names its node, so the kubelet alone admits it, and refuses it when the node has
+	// less memory left to request than the probe asks for (#4664). The smallest request the
+	// namespace accepts keeps that refusal to a node with nothing left at all.
+	resources, _ := container["resources"].(map[string]any)
+	requests, _ := resources["requests"].(map[string]any)
+	if requests["memory"] != "1Mi" || requests["cpu"] != "1m" {
+		return commandFailure(91, "runtime probe asks its node for more than the minimum request")
+	}
 	probeName, _ := metadata["name"].(string)
 	probeNode, _ := spec["nodeName"].(string)
 	if probeName == "" || probeNode == "" {

scripts/tests/test-talos-schematic-readback.sh

@@ -23,6 +23,10 @@ if ! printf '%s\n' "$filters" | yq -e '.talos[] | select(. == ".github/actions/d
   printf 'changes to the production deploy action must run the Talos readback test in PR CI\n' >&2
   exit 1
 fi
+if ! printf '%s\n' "$filters" | yq -e '.talos[] | select(. == "scripts/talos-boot-image-drift-accepted.tsv")' - >/dev/null; then
+  printf 'a change to the accepted boot-image drift record must run the Talos readback test in PR CI\n' >&2
+  exit 1
+fi
 
 update_line=$(grep -nF './scripts/run-ksail-prod-with-pull-auth.sh cluster update' "$deploy_action" | cut -d: -f1)
 stability_line=$(grep -nF './scripts/wait-for-prod-api-stability.sh' "$deploy_action" | cut -d: -f1)
@@ -34,7 +38,7 @@ if [[ -z "$update_line" || -z "$stability_line" || -z "$readback_line" ]] ||
 fi
 
 tmp_dir=$(mktemp -d)
-trap 'rm -f "$tmp_dir/kubectl" "$tmp_dir/nodes.json" "$tmp_dir/changed.json" "$tmp_dir/first-used"; rmdir "$tmp_dir"' EXIT
+trap 'rm -f "$tmp_dir/kubectl" "$tmp_dir/nodes.json" "$tmp_dir/changed.json" "$tmp_dir/first-used" "$tmp_dir/accepted.tsv"; rmdir "$tmp_dir"' EXIT
 mock_kubectl="$tmp_dir/kubectl"
 cat >"$mock_kubectl" <<'MOCK'
 #!/usr/bin/env bash
@@ -116,4 +120,132 @@ assert_rejected 'a cordoned node'
 jq 'del(.items[5])' "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
 assert_rejected 'a missing static worker'
 
+# A reviewed record may accept ONE static node that an earlier failed deploy left on another
+# image (#4665). Everything below runs against a record file of the test's own.
+other_schematic=$(printf '%064d' 0 | tr 0 a)
+accepted="$tmp_dir/accepted.tsv"
+header=$'node\tschematic\tissue\texpires'
+record() {
+  printf '%s\n' "$header" >"$accepted"
+  (($# == 0)) || printf '%s\n' "$@" >>"$accepted"
+}
+row() {
+  printf '%s\t%s\t%s\t%s' "$1" "$2" "${3:-devantler-tech/platform#4665}" "${4:-2026-10-22}"
+}
+drift_on() {
+  jq --argjson index "$1" --arg schematic "$2" \
+    '.items[$index].metadata.annotations["extensions.talos.dev/schematic"] = $schematic' \
+    "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
+}
+run_with_record() {
+  MOCK_NODES_FILE="$1" KUBECTL_BIN="$mock_kubectl" TALOS_READBACK_ACCEPTED_DRIFT_FILE="$accepted" \
+    TALOS_READBACK_TODAY="${2:-2026-10-08}" \
+    TALOS_READBACK_ATTEMPTS=1 TALOS_READBACK_INTERVAL_SECONDS=0 bash "$readback"
+}
+# A run that must succeed and say one thing, and must not say another. Arguments: description,
+# nodes file, text the output must contain, text it must not contain.
+assert_passes_saying() {
+  local description=$1 output
+  if ! output=$(run_with_record "$2" 2>&1); then
+    printf 'readback refused %s: %s\n' "$description" "$output" >&2
+    exit 1
+  fi
+  if [[ "$output" != *"$3"* || ( -n "$4" && "$output" == *"$4"* ) ]]; then
+    printf 'readback passed %s without saying the right thing: %s\n' "$description" "$output" >&2
+    exit 1
+  fi
+}
+# A run that must be refused for its own reason. Arguments: description, nodes file, the text
+# the refusal must contain, and optionally the date the run believes it is.
+assert_refused_saying() {
+  local description=$1 output status=0
+  output=$(run_with_record "$2" "${4:-2026-10-08}" 2>&1) || status=$?
+  if [[ "$status" != 1 ]]; then
+    printf 'readback returned %s for %s: %s\n' "$status" "$description" "$output" >&2
+    exit 1
+  fi
+  if [[ "$output" != *"$3"* ]]; then
+    printf 'readback refused %s for another reason: %s\n' "$description" "$output" >&2
+    exit 1
+  fi
+}
+
+# The recorded node on the recorded image passes, as a warning and never as the plain pass.
+record "$(row prod-worker-1 "$other_schematic")"
+drift_on 3 "$other_schematic"
+assert_passes_saying 'the recorded node on the recorded image' "$tmp_dir/changed.json" \
+  'PASS-WITH-ACCEPTED-DRIFT: 1 of 7 Talos nodes runs a recorded boot image' 'PASS: all Talos nodes'
+assert_passes_saying 'the recorded node on the recorded image' "$tmp_dir/changed.json" \
+  '::warning::prod-worker-1 runs the recorded boot image' ''
+
+# The record covers that node and that image, nothing else.
+drift_on 4 "$other_schematic"
+assert_refused_saying 'drift on a node the record does not name' "$tmp_dir/changed.json" \
+  'did not converge'
+drift_on 3 "$(printf '%064d' 0 | tr 0 b)"
+assert_refused_saying 'the recorded node on an image the record does not name' "$tmp_dir/changed.json" \
+  'did not converge'
+jq --arg schematic "$other_schematic" \
+  '.items[3].metadata.annotations["extensions.talos.dev/schematic"] = $schematic |
+   .items[4].metadata.annotations["extensions.talos.dev/schematic"] = $schematic' \
+  "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
+assert_refused_saying 'a second node on the recorded image' "$tmp_dir/changed.json" 'did not converge'
+
+# Every other requirement still binds the recorded node.
+jq --arg schematic "$other_schematic" \
+  '.items[3].metadata.annotations["extensions.talos.dev/schematic"] = $schematic |
+   .items[3].status.conditions[0].status = "False"' "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
+assert_refused_saying 'the recorded node while NotReady' "$tmp_dir/changed.json" 'did not converge'
+jq --arg schematic "$other_schematic" \
+  '.items[3].metadata.annotations["extensions.talos.dev/schematic"] = $schematic |
+   .items[3].spec.unschedulable = true' "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
+assert_refused_saying 'the recorded node while cordoned' "$tmp_dir/changed.json" 'did not converge'
+jq --arg schematic "$other_schematic" \
+  '.items[3].metadata.annotations["extensions.talos.dev/schematic"] = $schematic |
+   .items[3].status.nodeInfo.osImage = "Talos (v1.13.9)"' "$tmp_dir/nodes.json" >"$tmp_dir/changed.json"
+assert_refused_saying 'the recorded node on another Talos version' "$tmp_dir/changed.json" \
+  'did not converge'
+
+# The record lapses on its date, and says so.
+drift_on 3 "$other_schematic"
+assert_refused_saying 'an expired record' "$tmp_dir/changed.json" \
+  'the accepted boot-image drift for prod-worker-1 expired on 2026-10-22' 2026-10-23
+assert_passes_saying 'a record on its last day' "$tmp_dir/changed.json" 'PASS-WITH-ACCEPTED-DRIFT' ''
+
+# A record nobody needs any more is reported, and the pass is the plain one.
+assert_passes_saying 'a record the node no longer needs' "$tmp_dir/nodes.json" \
+  '::notice::the accepted boot-image drift for prod-worker-1 is no longer needed' 'PASS-WITH-ACCEPTED-DRIFT'
+assert_passes_saying 'a record the node no longer needs' "$tmp_dir/nodes.json" 'PASS: all Talos nodes' ''
+
+# A record that cannot be read as exactly one reviewed row is refused before any node is read.
+drift_on 3 "$other_schematic"
+invalid='invalid accepted boot-image drift record'
+record "$(row prod-worker-1 "$other_schematic")" "$(row prod-worker-2 "$other_schematic")"
+assert_refused_saying 'a record with two rows' "$tmp_dir/changed.json" "$invalid: more than one row"
+record "$(row autoscale-cx43-example "$other_schematic")"
+assert_refused_saying 'a record for an autoscaled node' "$tmp_dir/changed.json" "$invalid: node"
+record "$(row prod-worker-1 not-a-schematic)"
+assert_refused_saying 'a record with a malformed image' "$tmp_dir/changed.json" "$invalid: schematic"
+record "$(row prod-worker-1 "$desired_schematic")"
+assert_refused_saying 'a record naming the expected image' "$tmp_dir/changed.json" "$invalid: schematic"
+record "$(row prod-worker-1 "$other_schematic" 'see the chat')"
+assert_refused_saying 'a record with no issue' "$tmp_dir/changed.json" "$invalid: issue"
+record "$(row prod-worker-1 "$other_schematic" devantler-tech/platform#4665 someday)"
+assert_refused_saying 'a record with no date' "$tmp_dir/changed.json" "$invalid: expires"
+record "$(row prod-worker-1 "$other_schematic" devantler-tech/platform#4665 2026-10-22)"$'\textra'
+assert_refused_saying 'a record with a fifth field' "$tmp_dir/changed.json" "$invalid: fields"
+printf '%s\n' $'node\tschematic\texpires' "$(row prod-worker-1 "$other_schematic")" >"$accepted"
+assert_refused_saying 'a record with another header' "$tmp_dir/changed.json" "$invalid: header"
+: >"$accepted"
+assert_refused_saying 'an empty record file' "$tmp_dir/changed.json" "$invalid: header"
+
+# A record with a header and no row accepts nothing.
+record
+assert_refused_saying 'drift with an empty record' "$tmp_dir/changed.json" 'did not converge'
+assert_passes_saying 'healthy nodes with an empty record' "$tmp_dir/nodes.json" 'PASS: all Talos nodes' ''
+
+# The record in the repository is one this check can read.
+MOCK_NODES_FILE="$tmp_dir/nodes.json" KUBECTL_BIN="$mock_kubectl" TALOS_READBACK_TODAY=2026-10-08 \
+  TALOS_READBACK_ATTEMPTS=1 TALOS_READBACK_INTERVAL_SECONDS=0 bash "$readback" >/dev/null
+
 printf 'PASS: production Talos version, schematic, and node health are read back fail-closed\n'