fix(deploy): emit valid MicroTime timestamps for the GHCR Lease #2835

Activity

1. devantler

   self-assigned this

   on Jul 28, 2026
2. devantler

   added

   bug

   automation

   on Jul 28, 2026
3. github-project-automation

   added this to 🌊 Project Boardon Jul 28, 2026
4. coderabbitai commented on Jul 28, 2026

   

   coderabbitaiBot

   on Jul 28, 2026 – with coderabbitai

   Contributor

   More actions

   🔗 Related PRs

   #2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
   #2339 - docs(agents): record merge-queue mechanics + root-cause-before-requeue [merged]
   #2622 - fix(ci): heal cancelled merge-group deploys [merged]
   #2633 - fix(deploy): verify incoming image on Talos nodes [merged]
   #2793 - feat(cilium): activate homogeneous device selection [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
5. github-project-automation

   moved this to 👀 In Review in 🌊 Project Boardon Jul 28, 2026
6. devantler

   added the

   Bug

   issue type on Jul 28, 2026
7. devantler commented on Jul 28, 2026

   

   devantler

   on Jul 28, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Severity check: this blocks the merge queue but does NOT leave prod degraded. Measured 2026-07-28,
   read-only, right after an eviction.

   #2840 was evicted from the merge queue at 13:06Z with this exact failure in 🚀 Deploy to Prod:

   Could not atomically acquire the GHCR synchronization lease after concurrent updates.
   

   The 🩹 Heal Prod job that exists to restore main after a failed merge-group deploy then failed with
   the same error, so the automated remediation path is down too. That reads alarming, so I verified the
   cluster's actual state rather than inferring it:

   • every control-plane and worker node reports Ready, including the node named in the earlier drain
     failure on #2120;
   • every Flux Kustomization reports Ready;
   • every Flux HelmRelease reports Ready.

   Why it self-corrects: Flux reconciles continuously, so a merge-group deploy that dies partway is
   converged back by the next reconcile without the heal job needing to succeed. The heal job is
   belt-and-braces here, not the only path.

   So the impact is throughput, not availability — which is still significant: no platform PR can merge
   while this stands, and that is currently ~44 open PRs across all lanes. Worth fixing with urgency, but it
   does not need an out-of-hours prod remediation, and nobody should hand-roll a cluster change on the
   strength of the red heal job.

   I am deliberately not touching #2836 — that PR is being actively iterated in another lane (pushed 12:48Z,
   reviewed 12:55Z, 4 unresolved threads at the time of writing), so code changes there belong to it.
8. devantler

   closed this as completedin #2836on Jul 28, 2026
9. github-project-automation

   moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Jul 28, 2026