Repository navigation
Reassert the root OCI source's verify config out-of-band so a bad value stays recoverable #3014
Description
🤖 Generated by the Agentic Engineer
Evidence
The 2026-08-07 delivery freeze (~35h, cleared by a manual cluster write on 2026-08-08) had a single
structural cause: the trust configuration for the root OCI source is delivered through that
source. ksail.prod.yaml / flux-instance.yaml → Kustomization infrastructure-controllers →
rendered by flux-operator → onto OCIRepository/flux-system — which must verify the artifact
carrying that very config.
Once the live matcher was wrong, main already held the corrected value and could not deliver it.
Both automatic escape routes were void (verified in #3005): cd.yaml shares the deploy-prod
composite and reaches the same reconcile, and ksail cluster update does not own the FluxInstance.
The heal shares the deploy path too, so it failed alongside the deploy (#3005) — there is no second
line of defence. Blast radius reached the repo: every merge_group deploy timed out and evicted its
PR, so nothing could merge either.
The incident is resolved and the loop is not. The same deadlock recurs on any future bad value.
Proposed fix
The deploy already solves this exact class for the pull credential: a
🔑 Reassert Flux and consumer GHCR pull credential step writes it out-of-band, ahead of the
reconcile that depends on it. The root source's spec.verify has the same bootstrap circularity and
no equivalent step.
Add the mirror-image step: reassert spec.verify on the root OCIRepository from the values main
declares, before the reconcile, under the field manager that already owns the resource
(flux-operator) so there is no second writer and no SSA pruning question.
Acceptance criteria
- The deploy reasserts the root OCIRepository's
spec.verifybefore the reconcile step. - Decisive test: with the live verify block set to a multi-entry list, the next deploy repairs
it with no manual intervention — it must fail before the change and pass after. - No field contention: each object is written only under the field manager that already owns it. The FluxInstance is corrected under
kustomize-controller, and the root source is never written directly;flux-operatorre-renders it. (Revised 2026-09-23 after the live re-validation below showed a write to the source alone is reverted by the next flux-operator reconcile.) - A green deploy in which the step was skipped is not mistaken for a pass (a skipped step in a
green deploy is otherwise invisible).
Scope
Recovery only. Prevention — catching a matcher that cannot verify before it ships — is #3007, and
the two are complementary: #3007 stops the bad value being published, this stops a bad value that is
already live from being unrecoverable.
Rough size: S–M. Supersedes the remediation halves of #3005 and #3006, both closed as resolved.
Activity
coderabbitai commentedon Aug 8, 2026
🔗 Related PRs
#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2919 - fix(security): move Flux cosign verify to the path KSail reads [merged]
#2930 - feat(ci): verify published evidence before promoting the prod digest [merged]
#3001 - fix(security): enforce cosign verify on the live root OCIRepository [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Sep 23, 2026
Re-validated 2026-09-23: still needed, and the proposed fix is incomplete as written
I checked this against platform
mainbefore building.spec.verify. The later prevention work (scripts/validate-flux-verify, CI does not check that a cosign matcher actually verifies the artifact it guards #3007) checks the declared value in CI. It does not make a bad live value recoverable.OCIRepository/flux-systemis applied byflux-operator, which rendersspec.verifyfrom the FluxInstance's kustomize patch. TheFluxInstance/fluxitself is applied bykustomize-controller, meaning it arrives through that same root source. During the incident this issue describes, the live FluxInstance still carries the bad patch, so flux-operator's next reconcile writes the bad value back over a source-only reassert.What the step needs so it holds:
kustomize-controller, its existing apply manager), so flux-operator renders the corrected value.spec.verifyon the root source underflux-operatorand trigger the source's reconcile, so the corrected artifact is fetched without waiting for the operator's interval.The decisive test in the acceptance criteria should seed the bad value on both objects, because a test that seeds only the source would pass a fix that reverts in production.
I'm not implementing it this run: it changes the production deploy path, and the two-object ordering deserves its own draft with a fake-kubectl harness like the one the GHCR credential tests use.
chore: agent-claim #3014 nonce=81435e81e09a07f0a18a7bfee5325414
devantler commentedon Sep 30, 2026
Resolved by #4102 and deployed through the protected merge queue in main commit
0d85827b4e5d9b5b13b0a48c81424219f29b6bd4.The behavioral suite covers a bad matcher on both the FluxInstance and root source, nested and mistargeted patches, generation waiting, unreadable state, and incomplete reconciliation. Post-merge production readback through a GET-only wrapper reports
CURRENT, and the platform health gate reportsPLATFORM-HEALTH=OK.