diff --git a/scripts/inventory-first-party-image-signatures.sh b/scripts/inventory-first-party-image-signatures.sh index 19f9cdd426..c75cb18109 100755 --- a/scripts/inventory-first-party-image-signatures.sh +++ b/scripts/inventory-first-party-image-signatures.sh @@ -4,10 +4,16 @@ # # WHY THIS EXISTS (#3334, under #3101) # Talos verifies first-party images itself, and an image that MATCHES a rule but fails verification -# is refused at pull. Every such image is already cached on the fleet, so no rule-matching pull -# happens today and the rules have never produced a decision. The moment one does, every latent -# violation becomes a simultaneous ImagePullBackOff. That blast radius is measurable now, without -# touching a node — and until it is measured, enabling the control is a guess. +# is refused at pull. The rules DO produce decisions — containerd records each one per image as a +# `talos.dev/verified` label, and every first-party image pulled here has PASSED. What has NOT been +# observed is the REFUSAL: every such image is already cached on the fleet, so no rule-matching pull +# of an unsigned first-party image happens today. Enforcement is per image and happens AT THE PULL, +# so a violation surfaces only when that particular image is next pulled — on node replacement, +# cache eviction or a tag bump — while every other cached image keeps running. The blast radius is +# therefore scoped to the workloads that need that image pulled, not the whole fleet at once; a +# single failing image can still stop several replicas or nodes together, since a tag bump rolls +# them concurrently. It is measurable now, without touching a node — and until it is measured, +# enabling the control is a guess. # # 🔴 THE ENUMERATION IS THE HARD HALF, AND A HAND-WRITTEN LIST CANNOT DO IT. # The apps that run here are not declared in this repository: they arrive as Flux OCIRepository diff --git a/scripts/validate-image-verifier-liveness.sh b/scripts/validate-image-verifier-liveness.sh index abba10d27a..38624d602b 100755 --- a/scripts/validate-image-verifier-liveness.sh +++ b/scripts/validate-image-verifier-liveness.sh @@ -21,12 +21,16 @@ # re-create the exact blind spot it exists to close. # # WHAT THIS DOES NOT PROVE. That the controllers hold running rules and trust -# material is necessary for enforcement, not sufficient: it does not establish -# that a matched rule has ever produced a verification DECISION, because a -# cached image is never re-pulled. That behavioural proof — a known-unsigned -# image demonstrably refused at pull — is devantler-tech/platform#3336's job. -# This check's value is that it needs no rollout and fails loudly when the -# mechanism itself is not running. +# material is necessary for enforcement, not sufficient. A matched rule DOES +# produce a verification decision: containerd records one per image as a +# `talos.dev/verified` label, and that label tracks rule MATCH rather than +# signedness — measured 2026-08-26, 755 of 755 cached first-party images carry +# it and 0 of 1299 cached images with a resolvable non-first-party repository +# name do. What remains unproven is the REFUSAL: every first-party image pulled +# here has PASSED, so a FAILING decision has never been observed to block a +# pull. That behavioural proof — a known-unsigned image demonstrably refused at +# pull — is devantler-tech/platform#3336's job. This check's value is that it +# needs no rollout and fails loudly when the mechanism itself is not running. # # HISTORY, because it changes how a reader should read a green result: until # 2026-08-24 this script asserted a containerd `bin_dir`. No `bin_dir` is diff --git a/talos/cluster/verify-first-party-images.yaml b/talos/cluster/verify-first-party-images.yaml index ff553d4b8c..df0e32245c 100644 --- a/talos/cluster/verify-first-party-images.yaml +++ b/talos/cluster/verify-first-party-images.yaml @@ -8,23 +8,48 @@ # every node, and `security.TUFTrustedRootController` maintains the Sigstore TUF # trust root that keyless verification needs. # -# What is NOT established is that a matched rule has ever produced a -# verification DECISION: every first-party image is already cached, so no -# rule-matching pull occurs. Proving a refusal at pull is #3336. The original -# finding is #2856. +# A matched rule DOES produce a verification decision, and containerd records +# it per image as a `talos.dev/verified` label. The label tracks rule MATCH, +# not signedness: `ghcr.io/fluxcd/*` and `ghcr.io/cloudnative-pg/*` are +# cosign-signed upstream and never carry it, because no rule below names them. +# Measured 2026-08-26 across all 7 nodes: 755 of 755 cached +# `ghcr.io/devantler-tech/*` images carry the label, and 0 of 1299 cached +# images with a resolvable non-first-party repository name do — no exceptions. +# +# What is NOT established is the REFUSAL. Every first-party image pulled here +# has PASSED, so a FAILING decision has never been observed to block a pull. +# Proving that refusal is #3336. The original finding is #2856. # # Check this, do not assume it. The repository cannot answer it — the rules read # as correct whether or not anything evaluates them: # scripts/validate-image-verifier-liveness.sh # Measured 2026-08-24: 7 of 7 nodes hold running rules and a running trust root. # -# BEFORE ACTIVATING (#3101): an image that matches a rule but fails -# verification is refused, so any first-party image that is running yet -# unsigned goes ImagePullBackOff the moment the verifier starts working. +# BEFORE ACTIVATING (#3101): a matched image that fails verification is +# EXPECTED to be refused, so any first-party image that is running yet unsigned +# should enter ImagePullBackOff the moment it is next pulled. That refusal is +# exactly what #3336 exists to prove and has not been observed here, so treat it +# as the risk to design around rather than as a settled outcome. # Enumerate the running first-party digests and confirm each carries a cosign # signature — attestations alone do not satisfy the catch-all rule below — then # sign or scope the rule off any that do not. # +# Enumerated 2026-08-26 — of the four first-party images running on the +# cluster, ksail and provider-upjet-unifi verify against the exact keyless +# identity their rules require. wedding-app and ascoachingogvaner are PRIVATE +# packages: an unauthenticated cosign verify returns DENIED, which is an ACCESS +# result and NOT evidence that a signature is absent. Settling those two needs +# an AUTHENTICATED read, and this repository already stages one: the +# `🔑 Stage the GHCR pull credential` step in +# .github/workflows/validate-image-verifier-liveness.yaml decrypts the cluster's +# own Flux GHCR pull credential into DOCKER_CONFIG before it runs +# scripts/inventory-first-party-image-signatures.sh, so no ADDITIONAL +# `read:packages` token has to be minted. What is missing is a recorded RUN of +# that authenticated inventory against the fleet — the newest run of that +# workflow (2026-08-11) executed only its unit-test job — so until one is on +# record their signature state stays UNKNOWN, and they are the two images the +# catch-all rule governs. +# # Declares the cosign signatures required at the containerd PULL layer on # every node — a defense-in-depth complement to the GitOps-source # verification Flux already performs on OCI *manifest* artifacts (apps'