Prod deploys and their recovery job both fail: the KUBE_CONFIG secret is malformed YAML #3107

🤖 Generated by the Agentic Engineer

The prod environment's KUBE_CONFIG secret no longer parses as YAML, so every prod deploy fails at credential validation — and so does the recovery job that is supposed to clean up after it. The merge queue therefore cannot merge anything into main.

Evidence

Three independent failures, across two different PRs and both job types, all at the same point:

All three emit:

error: error loading config file "/home/runner/.kube/config": yaml: line 3: did not find expected key

and the two on main's path then hit main's own validation, which already names the remedy:

Restored kubeconfig has no 'admin@prod' context (ksail.prod.yaml's connection.context). KUBE_CONFIG looks malformed or incomplete — refresh it. See docs/dr/runbook.md → 'Refresh CI deploy credentials'.

This is the secret, not #3106

#3106 is a plausible-looking suspect because it adds the first step that touches the kubeconfig, so it surfaces the fault earliest and with the rawest message. It is not the cause: its diff leaves the restore step untouched, and 🩹 Heal Prod — which runs main, without that step — fails identically. #3096, which contains none of #3106's changes, then reproduced it on main's path.

Window

CD on main succeeded at 19:36:29Z (162e3bcce1); the first observed failure is 21:30:36Z. The secret became malformed between those two points. #3104 was filed at 18:24Z about this same credential, so a refresh attempted in response to it is the most likely origin — the value appears to have been pasted with the shape broken around line 3 rather than being merely stale.

Impact

• No PR can merge: the merge_group deploy is a required check.
• Recovery is down too. 🩹 Heal Prod exists to restore main after a failed merge-group deploy, and it dies at the same validation — so one bad credential takes out both the deploy and its own rollback.
• Not currently dangerous to the running cluster: both jobs fail before any cluster mutation (every subsequent step is skipped), so no partial deploy is in flight.

Fix

Refresh the prod environment's KUBE_CONFIG secret per docs/dr/runbook.md → Refresh CI deploy credentials, then re-queue. This is a maintainer action — an agent must not handle credential material, so this issue is the handoff, not something a run can close by itself.

Worth considering alongside it: main's validation already produces a precise, actionable message, but it only runs after the merge queue has committed a PR to a deploy cycle. A cheap pre-flight parse of the restored kubeconfig at the top of the deploy — or in a fast non-deploy check — would fail the PR in seconds instead of consuming a queue slot and a recovery job. Happy to open that as a separate change once the credential is healthy.