From 1f7a463fd4518558f3d18d358395f884f52a63d2 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 26 Aug 2026 04:21:47 +0200 Subject: [PATCH 1/5] docs(security): record that node image-verification produces decisions The ImageVerificationConfig STATUS block and the liveness script both stated that it was not established whether a matched rule had ever produced a verification decision, on the reasoning that every first-party image is cached so no rule-matching pull occurs. That is no longer accurate. containerd records a per-image `talos.dev/verified` label for images Talos verified, and the label tracks rule MATCH rather than signedness. Measured across all 7 nodes on 2026-08-26: 755 of 755 cached ghcr.io/devantler-tech/* images carry it, and 0 of 1299 cached images with a resolvable non-first-party repository name do. Flux and CloudNativePG publish cosign-signed images to ghcr.io and never carry the label, which is what rules out signedness as the cause and leaves rule match as the discriminator. What remains unproven is the refusal: every first-party image pulled here has passed, so a failing decision has never been observed to block a pull. That is unchanged and remains #3336's job. Also records the pre-activation enumeration the STATUS block asks for: ksail and provider-upjet-unifi verify against the exact keyless identity their rules require; wedding-app and ascoachingogvaner are private packages whose unauthenticated cosign verify returns DENIED, which is an access result and not evidence a signature is absent. Part of #3101 Co-Authored-By: Claude Opus 5 (1M context) --- scripts/validate-image-verifier-liveness.sh | 16 ++++++++----- talos/cluster/verify-first-party-images.yaml | 25 ++++++++++++++++---- 2 files changed, 30 insertions(+), 11 deletions(-) diff --git a/scripts/validate-image-verifier-liveness.sh b/scripts/validate-image-verifier-liveness.sh index abba10d27a..38624d602b 100755 --- a/scripts/validate-image-verifier-liveness.sh +++ b/scripts/validate-image-verifier-liveness.sh @@ -21,12 +21,16 @@ # re-create the exact blind spot it exists to close. # # WHAT THIS DOES NOT PROVE. That the controllers hold running rules and trust -# material is necessary for enforcement, not sufficient: it does not establish -# that a matched rule has ever produced a verification DECISION, because a -# cached image is never re-pulled. That behavioural proof — a known-unsigned -# image demonstrably refused at pull — is devantler-tech/platform#3336's job. -# This check's value is that it needs no rollout and fails loudly when the -# mechanism itself is not running. +# material is necessary for enforcement, not sufficient. A matched rule DOES +# produce a verification decision: containerd records one per image as a +# `talos.dev/verified` label, and that label tracks rule MATCH rather than +# signedness — measured 2026-08-26, 755 of 755 cached first-party images carry +# it and 0 of 1299 cached images with a resolvable non-first-party repository +# name do. What remains unproven is the REFUSAL: every first-party image pulled +# here has PASSED, so a FAILING decision has never been observed to block a +# pull. That behavioural proof — a known-unsigned image demonstrably refused at +# pull — is devantler-tech/platform#3336's job. This check's value is that it +# needs no rollout and fails loudly when the mechanism itself is not running. # # HISTORY, because it changes how a reader should read a green result: until # 2026-08-24 this script asserted a containerd `bin_dir`. No `bin_dir` is diff --git a/talos/cluster/verify-first-party-images.yaml b/talos/cluster/verify-first-party-images.yaml index ff553d4b8c..5eeb28f044 100644 --- a/talos/cluster/verify-first-party-images.yaml +++ b/talos/cluster/verify-first-party-images.yaml @@ -8,10 +8,17 @@ # every node, and `security.TUFTrustedRootController` maintains the Sigstore TUF # trust root that keyless verification needs. # -# What is NOT established is that a matched rule has ever produced a -# verification DECISION: every first-party image is already cached, so no -# rule-matching pull occurs. Proving a refusal at pull is #3336. The original -# finding is #2856. +# A matched rule DOES produce a verification decision, and containerd records +# it per image as a `talos.dev/verified` label. The label tracks rule MATCH, +# not signedness: `ghcr.io/fluxcd/*` and `ghcr.io/cloudnative-pg/*` are +# cosign-signed upstream and never carry it, because no rule below names them. +# Measured 2026-08-26 across all 7 nodes: 755 of 755 cached +# `ghcr.io/devantler-tech/*` images carry the label, and 0 of 1299 cached +# images with a resolvable non-first-party repository name do — no exceptions. +# +# What is NOT established is the REFUSAL. Every first-party image pulled here +# has PASSED, so a FAILING decision has never been observed to block a pull. +# Proving that refusal is #3336. The original finding is #2856. # # Check this, do not assume it. The repository cannot answer it — the rules read # as correct whether or not anything evaluates them: @@ -20,11 +27,19 @@ # # BEFORE ACTIVATING (#3101): an image that matches a rule but fails # verification is refused, so any first-party image that is running yet -# unsigned goes ImagePullBackOff the moment the verifier starts working. +# unsigned goes ImagePullBackOff the moment it is next pulled. # Enumerate the running first-party digests and confirm each carries a cosign # signature — attestations alone do not satisfy the catch-all rule below — then # sign or scope the rule off any that do not. # +# Enumerated 2026-08-26 — of the four first-party images running on the +# cluster, ksail and provider-upjet-unifi verify against the exact keyless +# identity their rules require. wedding-app and ascoachingogvaner are PRIVATE +# packages: an unauthenticated cosign verify returns DENIED, which is an ACCESS +# result and NOT evidence that a signature is absent. Settling those two needs +# a token carrying `read:packages`; until then their signature state is +# UNKNOWN — and they are the two images the catch-all rule governs. +# # Declares the cosign signatures required at the containerd PULL layer on # every node — a defense-in-depth complement to the GitOps-source # verification Flux already performs on OCI *manifest* artifacts (apps' From ca9267d9bdbc73abad55d2a7de7b5b1b17ff1414 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 26 Aug 2026 05:08:35 +0200 Subject: [PATCH 2/5] docs(security): keep the pull-refusal statement conditional The BEFORE ACTIVATING note asserted that a matched-but-failing image is refused and that an unsigned first-party image goes ImagePullBackOff, while the same file records twenty lines earlier that a failing decision has never been observed to block a pull. Stating the refusal as established is the exact defect this change set exists to correct, so the note now reads as expected behaviour and points at the issue that would prove it. Co-Authored-By: Claude Opus 5 (1M context) --- talos/cluster/verify-first-party-images.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/talos/cluster/verify-first-party-images.yaml b/talos/cluster/verify-first-party-images.yaml index 5eeb28f044..bc187f5c11 100644 --- a/talos/cluster/verify-first-party-images.yaml +++ b/talos/cluster/verify-first-party-images.yaml @@ -25,9 +25,11 @@ # scripts/validate-image-verifier-liveness.sh # Measured 2026-08-24: 7 of 7 nodes hold running rules and a running trust root. # -# BEFORE ACTIVATING (#3101): an image that matches a rule but fails -# verification is refused, so any first-party image that is running yet -# unsigned goes ImagePullBackOff the moment it is next pulled. +# BEFORE ACTIVATING (#3101): a matched image that fails verification is +# EXPECTED to be refused, so any first-party image that is running yet unsigned +# should enter ImagePullBackOff the moment it is next pulled. That refusal is +# exactly what #3336 exists to prove and has not been observed here, so treat it +# as the risk to design around rather than as a settled outcome. # Enumerate the running first-party digests and confirm each carries a cosign # signature — attestations alone do not satisfy the catch-all rule below — then # sign or scope the rule off any that do not. From 590c4dbe72906b06cdb9de357a9256e6322d6762 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 26 Aug 2026 06:22:56 +0200 Subject: [PATCH 3/5] docs(security): correct the authenticated-read premise in both places The status block said settling the two private images needs a token carrying read:packages. It does not: validate-image-verifier-liveness.yaml already decrypts the cluster's own Flux GHCR pull credential into DOCKER_CONFIG before running the inventory. Record that, while still declining to claim a PASS result no run has produced. The inventory script also still said the rules have never produced a decision, which is the exact claim this PR disproves. Align it with the header's framing: decisions are observed, the refusal is not. --- scripts/inventory-first-party-image-signatures.sh | 10 ++++++---- talos/cluster/verify-first-party-images.yaml | 12 ++++++++++-- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/scripts/inventory-first-party-image-signatures.sh b/scripts/inventory-first-party-image-signatures.sh index 19f9cdd426..fe7a6d9222 100755 --- a/scripts/inventory-first-party-image-signatures.sh +++ b/scripts/inventory-first-party-image-signatures.sh @@ -4,10 +4,12 @@ # # WHY THIS EXISTS (#3334, under #3101) # Talos verifies first-party images itself, and an image that MATCHES a rule but fails verification -# is refused at pull. Every such image is already cached on the fleet, so no rule-matching pull -# happens today and the rules have never produced a decision. The moment one does, every latent -# violation becomes a simultaneous ImagePullBackOff. That blast radius is measurable now, without -# touching a node — and until it is measured, enabling the control is a guess. +# is refused at pull. The rules DO produce decisions — containerd records each one per image as a +# `talos.dev/verified` label, and every first-party image pulled here has PASSED. What has NOT been +# observed is the REFUSAL: every such image is already cached on the fleet, so no rule-matching pull +# of an unsigned first-party image happens today. The moment one does, every latent violation +# becomes a simultaneous ImagePullBackOff. That blast radius is measurable now, without touching a +# node — and until it is measured, enabling the control is a guess. # # 🔴 THE ENUMERATION IS THE HARD HALF, AND A HAND-WRITTEN LIST CANNOT DO IT. # The apps that run here are not declared in this repository: they arrive as Flux OCIRepository diff --git a/talos/cluster/verify-first-party-images.yaml b/talos/cluster/verify-first-party-images.yaml index bc187f5c11..df0e32245c 100644 --- a/talos/cluster/verify-first-party-images.yaml +++ b/talos/cluster/verify-first-party-images.yaml @@ -39,8 +39,16 @@ # identity their rules require. wedding-app and ascoachingogvaner are PRIVATE # packages: an unauthenticated cosign verify returns DENIED, which is an ACCESS # result and NOT evidence that a signature is absent. Settling those two needs -# a token carrying `read:packages`; until then their signature state is -# UNKNOWN — and they are the two images the catch-all rule governs. +# an AUTHENTICATED read, and this repository already stages one: the +# `🔑 Stage the GHCR pull credential` step in +# .github/workflows/validate-image-verifier-liveness.yaml decrypts the cluster's +# own Flux GHCR pull credential into DOCKER_CONFIG before it runs +# scripts/inventory-first-party-image-signatures.sh, so no ADDITIONAL +# `read:packages` token has to be minted. What is missing is a recorded RUN of +# that authenticated inventory against the fleet — the newest run of that +# workflow (2026-08-11) executed only its unit-test job — so until one is on +# record their signature state stays UNKNOWN, and they are the two images the +# catch-all rule governs. # # Declares the cosign signatures required at the containerd PULL layer on # every node — a defense-in-depth complement to the GitOps-source From 8e285054146905addbb6a7bc37710b34d13fef67 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 26 Aug 2026 07:15:59 +0200 Subject: [PATCH 4/5] docs(security): describe image-verification failure scope per image Enforcement happens at the pull, so an unsigned matching image is refused when that image is next pulled while other cached images keep running. The comment claimed one pull turns every latent violation into a simultaneous ImagePullBackOff, which overstates the blast radius and would mislead the enablement decision. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/inventory-first-party-image-signatures.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/inventory-first-party-image-signatures.sh b/scripts/inventory-first-party-image-signatures.sh index fe7a6d9222..5764b0027d 100755 --- a/scripts/inventory-first-party-image-signatures.sh +++ b/scripts/inventory-first-party-image-signatures.sh @@ -7,9 +7,12 @@ # is refused at pull. The rules DO produce decisions — containerd records each one per image as a # `talos.dev/verified` label, and every first-party image pulled here has PASSED. What has NOT been # observed is the REFUSAL: every such image is already cached on the fleet, so no rule-matching pull -# of an unsigned first-party image happens today. The moment one does, every latent violation -# becomes a simultaneous ImagePullBackOff. That blast radius is measurable now, without touching a -# node — and until it is measured, enabling the control is a guess. +# of an unsigned first-party image happens today. Enforcement is per image and happens AT THE PULL, +# so a violation surfaces only when that particular image is next pulled — on node replacement, +# cache eviction or a tag bump — while every other cached image keeps running. The blast radius is +# therefore a count of latent violations surfacing one at a time, not one simultaneous outage. It +# is measurable now, without touching a node — and until it is measured, enabling the control is a +# guess. # # 🔴 THE ENUMERATION IS THE HARD HALF, AND A HAND-WRITTEN LIST CANNOT DO IT. # The apps that run here are not declared in this repository: they arrive as Flux OCIRepository From 8479b6f5a11cc57f8aa9e3650d18710a0ded8533 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 26 Aug 2026 08:09:54 +0200 Subject: [PATCH 5/5] docs(security): scope the blast radius to the image, not to a serial order "Surfacing one at a time" claims a serialization the mechanism does not guarantee. Enforcement is per image at the pull, so a violation is scoped to the workloads that need THAT image pulled -- but a tag bump rolls replicas and nodes together, so one failing image can stop several of them concurrently. Comment-only; the 269 code lines are byte-identical. --- scripts/inventory-first-party-image-signatures.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/scripts/inventory-first-party-image-signatures.sh b/scripts/inventory-first-party-image-signatures.sh index 5764b0027d..c75cb18109 100755 --- a/scripts/inventory-first-party-image-signatures.sh +++ b/scripts/inventory-first-party-image-signatures.sh @@ -10,9 +10,10 @@ # of an unsigned first-party image happens today. Enforcement is per image and happens AT THE PULL, # so a violation surfaces only when that particular image is next pulled — on node replacement, # cache eviction or a tag bump — while every other cached image keeps running. The blast radius is -# therefore a count of latent violations surfacing one at a time, not one simultaneous outage. It -# is measurable now, without touching a node — and until it is measured, enabling the control is a -# guess. +# therefore scoped to the workloads that need that image pulled, not the whole fleet at once; a +# single failing image can still stop several replicas or nodes together, since a tag bump rolls +# them concurrently. It is measurable now, without touching a node — and until it is measured, +# enabling the control is a guess. # # 🔴 THE ENUMERATION IS THE HARD HALF, AND A HAND-WRITTEN LIST CANNOT DO IT. # The apps that run here are not declared in this repository: they arrive as Flux OCIRepository