Speculative merge-group deploy left the headlamp PVC stuck Terminating, wedging every prod deploy #3167

Activity

1. devantler

   added

   bug

   on Aug 16, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Aug 16, 2026
3. coderabbitai commented on Aug 16, 2026

   

   coderabbitaiBot

   on Aug 16, 2026 – with coderabbitai

   Contributor

   More actions

   🔗 Related PRs

   #2707 - fix(unifi): prevent pruning managed network resources [merged]
   #2985 - fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one [merged]
   #3034 - feat(ci): make an orphaned GHCR deploy fence diagnosable and recoverable [merged]
   #3077 - fix(ci): reclaim a drain fence whose owning transaction is provably gone [merged]
   #3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
4. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Aug 16, 2026
5. devantler commented on Aug 16, 2026

   

   devantler

   on Aug 16, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Update — the stuck deletion has cleared; the remaining state is self-healing. No further prod surgery is needed.

   Live state at 15:02Z:

   • PersistentVolumeClaim/headlamp/headlamp — gone. The Terminating state that jammed the prune has finished, so the original blocker in this issue is resolved.
   • pod/headlamp-79658d4bd5-h7bp6 — recreated, Pending. It cannot schedule yet: 0/9 nodes are available: persistentvolumeclaim "headlamp" not found.
   • Kustomization/apps — still Ready=Unknown, running health checks for latest@sha256:9c10930c with the 20m timeout. That check started 14:47:36Z, i.e. while the PVC was still terminating, so it is doomed and will expire ~15:07Z.

   Why this now recovers on its own, rather than needing another manual step:

   1. main still declares the PVC — k8s/bases/apps/headlamp/persistent-volume-claim.yaml is present. The PR that removes it (fix(headlamp): disable unverified plugin downloads #2741) was evicted, never merged, so the desired state still contains it.
   2. Flux still owns it: headlamp_headlamp__PersistentVolumeClaim is in the apps Kustomization's inventory, and prune: true.
   3. retryInterval: 2m.

   So the sequence is: current health check times out ~15:07:36Z → retry ~15:09:36Z → the PVC is re-applied against a namespace where nothing is terminating any more → it binds → the pod schedules → apps goes Ready=True → the merge queue stops evicting.

   The earlier apply could not do this because at 14:47:36Z the PVC was still Terminating; you cannot recreate an object mid-deletion.

   Queue state: the merge queue is currently empty — platform#2732 was evicted by this, not by any fault of its own, and will need re-enqueuing once apps is green.

   One consequence worth stating plainly: the PV backing the old PVC is gone, so headlamp's plugins-dir volume comes back empty. That is a cache directory repopulated on start, not durable state, so no data loss follows — but it is a real effect of the prune rather than a no-op.

   I am verifying the recovery rather than assuming it, and will post the outcome. If apps has not gone Ready=True by ~15:15Z, the self-heal reasoning above is wrong and this needs a second look.
6. devantler commented on Aug 16, 2026

   

   devantler

   on Aug 16, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Resolved — verified recovered.

   Live state at 15:03:50Z:

   • PersistentVolumeClaim/headlamp/headlamp — Bound, freshly provisioned (pvc-a6fc2814-74d6-4728-84ff-842128840a8c, 256Mi, longhorn-wffc), 47s old.
   • Kustomization/apps — Ready=True, Applied revision: latest@sha256:9c10930c…. It had been Unknown/health-checking for the whole wedge.
   • pod/headlamp-79658d4bd5-h7bp6 — moved Pending → ContainerCreating once the claim bound.

   The recovery mechanism was as described in my previous comment: main still declares the PVC, Flux still held headlamp_headlamp__PersistentVolumeClaim in the apps inventory, so once nothing was mid-deletion the next apply recreated it.

   One correction to that comment: I predicted the doomed health check would have to expire at ~15:07:36Z with recovery following at ~15:09:36Z. It actually recovered at 15:03:14Z, several minutes before that deadline — so the reconcile did not wait out the full 20m timeout. The mechanism was right; the timing prediction was wrong, and the wedge cleared sooner than stated.

   Consequences now unblocked:

   • The merge queue is no longer evicting on the apps health check. platform#3166 (the separate umami ServiceAccount fault) is enqueued at position 1 at its evaluated head.
   • platform#2732 was evicted by this wedge rather than by any fault of its own and needs re-enqueuing; I am confirming the queue genuinely drains before adding it back.

   Worth carrying forward, since the trigger is still present: the underlying hazard is unchanged. platform#2741 removes the headlamp PVC manifest, and its speculative merge_group deploy is what pruned the live PVC while a pod still mounted it — the deletion outlived the evicted PR. Merging #2741 for real will delete that PVC again, and if the headlamp pod is still mounting it the prune can jam the same way. That PR should drain the volume's consumer first, or accept a deliberate pod restart, rather than being merged as an ordinary manifest removal.

   Closing this as resolved.
7. devantler

   closed this as completedon Aug 16, 2026
8. github-project-automation

   moved this from 📥 Backlog to ✅ Done in 🌊 Project Boardon Aug 16, 2026
9. github-project-automation

   moved this from ✅ Done to 👀 In Review in 🌊 Project Boardon Aug 16, 2026
10. devantler commented on Aug 16, 2026

    

    devantler

    on Aug 16, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Resolved end to end.

    Root cause was the speculative #2741 merge-group artifact removing the live headlamp/headlamp PVC while the old two-container Headlamp pod still mounted it. The deletion timestamp survived that candidate’s eviction, so every later apps reconciliation waited on the same pvc-protection finalizer.

    Recovery and permanent controls:

    • Deleted only the mounting Headlamp pod; the old claim completed deletion and main recreated it.
    • fix(ci): protect persistent state from speculative pruning #3168 merged as verified main commit cdababde7bac37d1c5b001195629532e3574bd3b, adding two-stage retirement protection and static plus live production gates before artifact publication.
    • fix(headlamp): disable unverified plugin downloads #2741 was rebased onto that protected baseline and merged as verified main commit 027b9baea1dbb6036ab3e74492bfe470ec9c312c after merge-group CI run 31959207601 passed the live gate and production deploy.
    • Candidate digest sha256:7008993dc3cdc31151aaddd8228edf7c4af2ecc32c45e4993e45708e4ffcf451 is Ready across all Flux layers.
    • Headlamp is now a healthy one-container pod with no plugin sidecar or PVC mount.
    • After proving no Deployment or pod referenced it, explicitly deleted the orphaned headlamp/headlamp claim. Its PV and Longhorn volume are also gone; only the disposable 256Mi plugin cache was retired.

    Final state: Headlamp 1/1 Running, HelmRelease Ready/TestSucceeded, all Flux Kustomizations Ready, and post-merge Validate Main, Release, and CodeQL workflows green.