A bad FluxInstance change cannot be corrected by GitOps, because the Kustomization that applies it waits on its health

🤖 Generated by the Agentic Engineer

Evidence

A bad FluxInstance change cannot be corrected by GitOps, because the Kustomization that applies the
FluxInstance waits on that same FluxInstance becoming healthy. Measured live on prod today, while
#3005 was in progress.

The applying Kustomization is in the loop. kustomization/infrastructure-controllers has 177
inventory entries and one of them is flux-system_flux_fluxcd.controlplane.io_FluxInstance — so it
applies the FluxInstance, and Flux then waits for the applied object to report healthy:

HealthCheckFailed | health check failed after 25m0.034713948s:
  timeout waiting for: [FluxInstance/flux-system/flux status: 'InProgress']

Observed twice in 27 minutes (19:24:56Z, 19:52:01Z), i.e. it re-enters the same wait indefinitely.

Field ownership shows there is no other writer. kustomize-controller owns
spec.kustomize.patches on fluxinstance/flux via server-side apply. Nothing outside the GitOps path
writes that field, so the only route to correct it is the route the fault disables.

The observed chain (from #3005). A FluxInstance patch put an unsatisfiable verify on the root
OCIRepository → the source failed verification → the FluxInstance reported InProgress → the
Kustomization that applies it timed out for 25m cycles → infrastructure and apps went
DependencyNotReady. Three of eleven Kustomizations were stalled for roughly 5.5 hours, with
lastAppliedRev frozen at …82e0d0bf while lastAttemptedRev sat at …e6039d0a.

Affected audience and impact

Anyone operating this cluster. The specific verify misconfiguration in #3005 is one instance of the
class; any FluxInstance change that leaves the instance unhealthy produces the same lock. Recovery
today required direct cluster mutation, which is exactly the capability GitOps exists to remove from
the routine path — and the automated 🩹 Heal Prod fallback runs through the same channel, so it does
not help either.

Expected behaviour

A configuration error delivered through GitOps should be correctable by delivering a corrected
configuration through GitOps. The delivery tree should not require an out-of-band cluster edit to
recover from a change it applied itself.

Acceptance criteria

A FluxInstance change that leaves the instance unhealthy can be reverted by merging the revert,
with no manual kubectl mutation — demonstrated end-to-end against a deliberately bad change.
That demonstration includes a negative control: the same bad change before the fix
reproduces the lock, so the test is not vacuous.
A FluxInstance-level fault surfaces distinguishably rather than as a generic 25m health-check
timeout on infrastructure-controllers (complements The handoff quiesce timeout hides its own cause, which is already in hand #3002, which covers a different message on
the same symptom).
The chosen mechanism is recorded with its trade-off. Candidates, not a decision:
apply the FluxInstance from a Kustomization outside the health-gated dependency chain; drop it
from health gating; or provide a documented break-glass that does not need improvised commands.

Size

Small-to-medium. The diagnosis is settled; the work is choosing the decoupling and proving it with a
reproduction plus negative control.

Related: #3005 (the incident this was measured during), #3002 (the timeout message hiding its cause),
#2996 (a FluxInstance patch matching no target is silently discarded — same "FluxInstance changes fail
quietly" family).

🔗 Related PRs

#2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2919 - fix(security): move Flux cosign verify to the path KSail reads [merged]
#2927 - fix(deploy): await pre-handoff kustomize-controller pod termination [merged]
#2944 - fix(ci): assert Flux verification is in effect, not merely configured [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Root cause confirmed, and the fix designed. Not shipped yet — reason at the bottom.

Today's incident (#3005) rode this exact defect for 7.5 hours, so here is the mechanism pinned to
the manifests, plus the change I propose.

The deadlock, in three facts on `main`

k8s/clusters/base/flux-kustomization-infrastructure-controllers.yaml sets wait: true.
Flux's wait: true health-checks every resource the Kustomization applies — there is no
per-resource exclusion.
k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml lists - flux-instance/,
so FluxInstance/flux is one of those applied resources.
FluxInstance/flux manages the root OCIRepository/flux-system — the very source
infrastructure-controllers reads from (sourceRef.kind: OCIRepository, name: flux-system).

So the layer applies the object that owns its own source, then blocks until that object is healthy.

Why that is a true self-lock, not merely slow

A bad spec.verify on the root source makes the FluxInstance permanently Progressing
(it cannot finish reconciling a source that rejects every artifact). Measured live at 21:57Z:

Resource	State
`ocirepository/flux-system`	`Ready=False VerificationError`, artifact frozen at the 14:28:42Z digest
`fluxinstance/flux`	`Reconciling=True Progressing`, generation 26
`infrastructure-controllers`	`Progressing` — waiting on the above
`infrastructure`	`DependencyNotReady`

The correcting commit cannot arrive, because it would have to arrive through the source the bad
config rejects. GitOps cannot self-heal from this by construction.

Proposed fix — take the FluxInstance out of the wait-gated layer

Give it its own Kustomization with wait: false:

# k8s/clusters/base/flux-kustomization-flux-instance.yaml
spec:
  path: providers/__PROVIDER__/infrastructure/controllers/flux-instance
  sourceRef: { kind: OCIRepository, name: flux-system }
  dependsOn:
    - name: infrastructure-controllers   # flux-operator must be running first
  wait: false                            # <- the whole point
  prune: true

and drop - flux-instance/ from the controllers aggregate.

Why wait: false is correct here specifically, and not a general loosening. Health-gating exists
so dependents do not start against a half-ready layer. Nothing depends on the FluxInstance being
Ready — it is the operator's own desired-state object, and the things that actually matter
(controllers running, CRDs present) are gated by infrastructure-controllers, which keeps wait: true
untouched. What we lose is a health signal nobody consumes; what we gain is that a bad FluxInstance
stalls only itself instead of freezing the whole delivery chain.

Ordering is preserved: dependsOn: infrastructure-controllers keeps flux-operator up before the
FluxInstance is applied, which is the one real ordering constraint today.

Alternatives considered and rejected

wait: false on infrastructure-controllers itself — removes health gating for every
controller (Cilium, Longhorn, Crossplane, …), so infrastructure would start against a half-ready
controller layer. Fixes the deadlock by deleting a control that is doing real work. No.
Explicit healthChecks: listing everything except the FluxInstance — Flux ignores
healthChecks when wait: true, so this means hand-enumerating every resource in a large,
frequently-changing aggregate. It would silently rot the first time someone adds a controller.
Stop managing the FluxInstance through GitOps entirely (bootstrap/ksail path only) — arguably
the most correct long-term answer, since the circularity is inherent, but it is a much larger change
to how the cluster is defined and deserves its own decision rather than being smuggled into a
bug fix. Worth a follow-up if this recurs.

Why no PR in this run

GitHub Actions has been in major_outage since 16:33Z (still major at 21:58Z; webhook triggers
throttled, so pushes are not starting runs). This change restructures the production delivery
topology and its whole value is behavioural — it cannot be justified by reading the diff. Shipping
it unvalidated, while prod's delivery chain is already wedged, is precisely the class of change that
caused #3005 in the first place.

The design above is mechanical to implement; it wants kustomize build + kubeconform + a real
deploy to prove it. I will pick it up when CI can actually run.

🤖 Generated by the Agentic Engineer

This defect is now live again, and it is currently blocking all of `platform`

Recurrence of the exact chain described above, since 2026-08-07T00:49Z (~35h). Adding current
evidence and a concrete remediation; I filed #3013 before finding this issue and have closed it as a
duplicate, porting its unique content here.

Live state. The root flux-system OCIRepository:

Ready=False / SourceVerified=False  reason=VerificationError
  no matching attestations: unsupported: multiple identities are not supported at this time

The live resource carries a three-entry spec.verify.matchOIDCIdentity. cosign rejects a
multi-entry list outright and fails closed for the whole set, so the source refuses every
artifact. infrastructure-controllers is stuck Progressing; infrastructure and apps are
Ready=False. The cluster is healthy on the last good revision — a delivery freeze, not an outage.

main is already correct. Both halves carry the single-entry alternation form (the
flux-instance.yaml kustomize patch since #3001, and ksail.prod.yaml for the bootstrap path). The
fix is pinned in Git behind the source it repairs. Field-manager inspection confirms spec.verify is
owned by flux-operator (Apply) and not written by the routine deploy path, so a redeploy cannot
correct it either.

Blast radius beyond the cluster. Every merge_group deploy times out waiting for the artifact and
the PR is evicted from the merge queue: #3004 evicted twice, #3011 evicted twice. Nothing has
merged to main since 2026-08-07T00:36Z, so this issue's own fix cannot merge either.

Proposed remediation

The deploy already solves this class for the pull credential — a
🔑 Reassert Flux and consumer GHCR pull credential step writes it out-of-band, ahead of the
reconcile that depends on it. The root source's verify block has the same bootstrap circularity
and no equivalent step.

Reassert spec.verify on the root OCIRepository out-of-band in the deploy, before the reconcile,
under the field manager that already owns the resource so there is no second writer.

Acceptance criteria:

The deploy reasserts the root OCIRepository's spec.verify before the reconcile step.
Reverting the live verify block to a multi-entry list is repaired by the next deploy with no
manual intervention — it must fail before the change and pass after.
No field contention with flux-operator.
A green deploy with the step skipped is not mistaken for a pass.

The manual unwedge command for the current freeze is in #3013 and still applies; it only makes the
cluster match main. Related: #3007 (CI does not check that a cosign matcher actually verifies the
artifact it guards) — this incident is exactly the failure #3007 predicted.

@devantler

🤖 Generated by the Agentic Engineer

Resolved — the in-place fix is applied and verified live

@devantler applied the single cluster write. Verified against the live cluster, not inferred:

Check	Value
`ocirepository/flux-system` `matchOIDCIdentity` entries	1 (the alternation form `main` declares)
`SourceVerified`	True — `Succeeded`
`Ready`	True — `Succeeded`

And it self-healed exactly as predicted — all five Kustomizations are now Ready=True at the same
new revision latest@sha256:e996cd9b…, up from the stale e6039d0a… they had been pinned to:

apps                        ready=True   e996cd9b
bootstrap                   ready=True   e996cd9b
flux-system                 ready=True   e996cd9b
infrastructure              ready=True   e996cd9b
infrastructure-controllers  ready=True   e996cd9b

So the divergence is closed in the convergent direction: flux-operator re-rendered the source from
main, and cluster and Git now agree. Delivery is restored, prod is no longer running never-merged
code, and the merge queue is unblocked (#3004 re-queued; its stale DO NOT RE-QUEUE body note has
been cleared).

Closing the incident, not the lesson. The structural loop this exposed is real and unfixed: the
root source's trust config is still delivered through the source it gates, and the heal still shares
the deploy path, so the same deadlock recurs on any future bad value. That recovery gap is now
tracked as a focused, actionable issue — #3014 — with a decisive acceptance test. Prevention
(catching a non-verifying matcher before it ships) remains #3007.

Navigation Menu

Repository navigation

A bad FluxInstance change cannot be corrected by GitOps, because the Kustomization that applies it waits on its health #3006

Evidence

Affected audience and impact

Expected behaviour

Acceptance criteria

Size

Root cause confirmed, and the fix designed. Not shipped yet — reason at the bottom.

The deadlock, in three facts on `main`

Why that is a true self-lock, not merely slow

Proposed fix — take the FluxInstance out of the wait-gated layer

Alternatives considered and rejected

Why no PR in this run

This defect is now live again, and it is currently blocking all of `platform`

Proposed remediation

Resolved — the in-place fix is applied and verified live

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

A bad FluxInstance change cannot be corrected by GitOps, because the Kustomization that applies it waits on its health #3006

Description

Evidence

Affected audience and impact

Expected behaviour

Acceptance criteria

Size

Activity

coderabbitai commented on Aug 6, 2026

devantler commented on Aug 6, 2026

Root cause confirmed, and the fix designed. Not shipped yet — reason at the bottom.

The deadlock, in three facts on main

Why that is a true self-lock, not merely slow

Proposed fix — take the FluxInstance out of the wait-gated layer

Alternatives considered and rejected

Why no PR in this run

devantler commented on Aug 8, 2026

This defect is now live again, and it is currently blocking all of platform

Proposed remediation

devantler commented on Aug 8, 2026

Resolved — the in-place fix is applied and verified live

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

The deadlock, in three facts on `main`

This defect is now live again, and it is currently blocking all of `platform`