The GHCR sync lease tells operators to prove a process dead, but ships no way to do it

Evidence

Prod CD was wedged from 2026-08-09 12:52Z. Every deploy failed with:

Another GHCR synchronization transaction holds the synchronization lease; automatic expiry takeover is disabled because Talos writes cannot be fenced. Prove the prior process is dead before explicitly recovering the Lease.

Two operability gaps turned a routine merge-queue eviction into an outage needing log archaeology:

No recovery mechanism exists. The message instructs an operator to "explicitly recover the Lease", but refresh-flux-ghcr-auth.sh has no recovery flag and no runbook section covers it. The only route is a hand-written CAS-guarded kubectl patch against a prod Lease — improvised under pressure, exactly when improvisation is most expensive.
The holder identity cannot be checked for liveness. It is <revision-prefix>-<pid>-<random> (observed: 4384f07cc5a864b0-4297-16649). The PID belongs to a runner that no longer exists, so "prove the prior process is dead" is unanswerable from the identity itself. Establishing it meant correlating the lease's renewTime against job timestamps across several workflow runs.

A GitHub-hosted holder is provably dead via the API — but only if the identity records which run and job took it.

Impact

Every deploy fails until a human intervenes, and the intervention is undocumented. The trigger (a cancelled merge-queue deploy) is routine, not exceptional.

Suggested shape

Embed the run and job id in the holder identity so liveness is decidable against the GitHub API.
Add an explicit, opt-in recovery path that requires the operator to pass the exact holder they observed, so it CAS-fails rather than blind-stealing a live lease.
Cover it in docs/dr/runbook.md, referenced from the error message.

The deliberate absence of automatic expiry takeover is correct and should stay — Talos writes cannot be fenced. This is about making the manual path supported rather than improvised.

Found while fixing the trigger in #3032, which stops a cancelled deploy from taking the lease in the first place. That fix does not make the lease recoverable once orphaned by any other cause.

🔗 Related PRs

#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2625 - fix(deploy): synchronize production GHCR pull auth [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
#2927 - fix(deploy): await pre-handoff kustomize-controller pod termination [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated with Claude Code

The same dead process orphaned a second fence — and this one suspends GitOps

Clearing the Lease unblocked the deploy only as far as the next fence. The re-run then failed with:

Another transaction already owns the image-verification policy handoff; refusing cluster mutation.

The infrastructure Flux Kustomization was left suspend: true, kustomize.toolkit.fluxcd.io/reconcile: disabled, owned by platform.devantler.tech/ghcr-policy-handoff-owner: 4384f07cc5a864b0-4297-16649 — the identical dead holder from the killed 12:52:28Z process. Infrastructure-layer reconciliation had therefore been frozen since then.

This widens the issue in two ways:

The scope is every fence the script takes, not just the Lease. cleanup_refresh_work releases five things in sequence — runtime probe, child policy fence, parent policy fence, bootstrap quarantine, then the Lease last. A kill part-way leaves an arbitrary prefix released and the rest orphaned. Here the parent fence was released and the child was not, so recovery cannot assume "one lease, one recovery": it needs to enumerate every fence the identity still holds.
Severity is higher than the Lease. A held Lease fails deploys loudly. A held policy fence leaves infrastructure suspended, so GitOps delivery silently stops while nothing looks broken until the next deploy.

Recovery is again an improvised CAS-guarded kubectl patch mirroring resume_flux_policy_handoff, with the uid, owner, reconcile annotation and suspend all pinned. Same gap as the Lease: correct procedure, no supported tooling.

A recovery path should therefore sweep by holder identity across all fences (Lease, child and parent policy Kustomizations, node quarantine/cordon annotations) and report what is still held, rather than addressing one resource.

Verified clean this time: no node is unschedulable, no node carries a ghcr-auth-drain-recovery annotation, and no other Lease or Kustomization names the dead identity.

🤖 Generated by the Agentic Engineer

This is happening right now — the deploy lane is wedged on exactly this

The condition this issue describes is live. Every merge_group deploy on this repository
currently fails, and so does the heal job behind it, both at the same step:

##[error]Another GHCR synchronization transaction holds the synchronization lease;
automatic expiry takeover is disabled because Talos writes cannot be fenced.
Prove the prior process is dead before explicitly recovering the Lease.

Observed on the pr-3047 merge-group runs at 00:54:56Z and again at 02:15:32Z. The second
run's 🩹 Heal Prod job hit the identical wall, so main could not be re-deployed either.
#3047 has been evicted twice and the queue is now empty.

The prior process is provably dead


holder	`4384f07cc5a864b0-3904-2045`
acquired	2026-08-10T01:00:18Z
last renewed	2026-08-10T01:35:11Z
`leaseDurationSeconds`	120
read back at	2026-08-10T02:23:48Z — 48 minutes stale, 24× the duration

The holder identity is built as ${desired_revision:0:16}-$$-${RANDOM}, so this one belongs
to PID 3904 of the merge-group CI run created 00:54:56Z. That run is completed with
conclusion failure, and GitHub recorded it finishing at 01:35:49Z — 38 seconds after
the final heartbeat. The renewals stop exactly where the run does, which is what makes this
a dead holder rather than a slow one.

Why nothing recovers on its own

sync_lease_is_available() treats the lease as free only when spec.holderIdentity is the
empty string; expiry is deliberately diagnostic-only, because Talos machine-config writes
expose no fencing token and a resumed zombie could write stale credentials. That reasoning
is right, and it is also precisely why a crashed run leaves the deploy lane permanently
wedged — the state this issue was opened about. Recovery is to clear
spec.holderIdentity on flux-system/ghcr-auth-refresh to "" once the holder is proven
dead, which the table above does.

I could not do it: cluster mutation is not within this agent's authority, so this is
reported rather than fixed.

Note for whoever picks this up

#3034 (ghcr-fence-recovery-3033) is open, green and unmerged. The gap it closes has now
actually fired, which is worth weighing against however it was being prioritised.

Navigation Menu

Repository navigation

The GHCR sync lease tells operators to prove a process dead, but ships no way to do it #3033

Evidence

Impact

Suggested shape

The same dead process orphaned a second fence — and this one suspends GitOps

This is happening right now — the deploy lane is wedged on exactly this

The prior process is provably dead

Why nothing recovers on its own

Note for whoever picks this up

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

The GHCR sync lease tells operators to prove a process dead, but ships no way to do it #3033

Description

Evidence

Impact

Suggested shape

Activity

coderabbitai commented on Aug 9, 2026

devantler commented on Aug 9, 2026

The same dead process orphaned a second fence — and this one suspends GitOps

devantler commented on Aug 10, 2026

This is happening right now — the deploy lane is wedged on exactly this

The prior process is provably dead

Why nothing recovers on its own

Note for whoever picks this up

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions