Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions scripts/inventory-first-party-image-signatures.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,16 @@
#
# WHY THIS EXISTS (#3334, under #3101)
# Talos verifies first-party images itself, and an image that MATCHES a rule but fails verification
# is refused at pull. Every such image is already cached on the fleet, so no rule-matching pull
# happens today and the rules have never produced a decision. The moment one does, every latent
# violation becomes a simultaneous ImagePullBackOff. That blast radius is measurable now, without
# touching a node β€” and until it is measured, enabling the control is a guess.
# is refused at pull. The rules DO produce decisions β€” containerd records each one per image as a
# `talos.dev/verified` label, and every first-party image pulled here has PASSED. What has NOT been
# observed is the REFUSAL: every such image is already cached on the fleet, so no rule-matching pull
# of an unsigned first-party image happens today. Enforcement is per image and happens AT THE PULL,
# so a violation surfaces only when that particular image is next pulled β€” on node replacement,
# cache eviction or a tag bump β€” while every other cached image keeps running. The blast radius is
# therefore scoped to the workloads that need that image pulled, not the whole fleet at once; a
# single failing image can still stop several replicas or nodes together, since a tag bump rolls
# them concurrently. It is measurable now, without touching a node β€” and until it is measured,
# enabling the control is a guess.
#
# πŸ”΄ THE ENUMERATION IS THE HARD HALF, AND A HAND-WRITTEN LIST CANNOT DO IT.
# The apps that run here are not declared in this repository: they arrive as Flux OCIRepository
Expand Down
16 changes: 10 additions & 6 deletions scripts/validate-image-verifier-liveness.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,16 @@
# re-create the exact blind spot it exists to close.
#
# WHAT THIS DOES NOT PROVE. That the controllers hold running rules and trust
# material is necessary for enforcement, not sufficient: it does not establish
# that a matched rule has ever produced a verification DECISION, because a
# cached image is never re-pulled. That behavioural proof β€” a known-unsigned
# image demonstrably refused at pull β€” is devantler-tech/platform#3336's job.
# This check's value is that it needs no rollout and fails loudly when the
# mechanism itself is not running.
# material is necessary for enforcement, not sufficient. A matched rule DOES
# produce a verification decision: containerd records one per image as a
# `talos.dev/verified` label, and that label tracks rule MATCH rather than
# signedness β€” measured 2026-08-26, 755 of 755 cached first-party images carry
# it and 0 of 1299 cached images with a resolvable non-first-party repository
# name do. What remains unproven is the REFUSAL: every first-party image pulled
Comment thread
devantler marked this conversation as resolved.
Resolved
# here has PASSED, so a FAILING decision has never been observed to block a
# pull. That behavioural proof β€” a known-unsigned image demonstrably refused at
# pull β€” is devantler-tech/platform#3336's job. This check's value is that it
# needs no rollout and fails loudly when the mechanism itself is not running.
#
# HISTORY, because it changes how a reader should read a green result: until
# 2026-08-24 this script asserted a containerd `bin_dir`. No `bin_dir` is
Expand Down
39 changes: 32 additions & 7 deletions talos/cluster/verify-first-party-images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,48 @@
# every node, and `security.TUFTrustedRootController` maintains the Sigstore TUF
# trust root that keyless verification needs.
#
# What is NOT established is that a matched rule has ever produced a
# verification DECISION: every first-party image is already cached, so no
# rule-matching pull occurs. Proving a refusal at pull is #3336. The original
# finding is #2856.
# A matched rule DOES produce a verification decision, and containerd records
# it per image as a `talos.dev/verified` label. The label tracks rule MATCH,
# not signedness: `ghcr.io/fluxcd/*` and `ghcr.io/cloudnative-pg/*` are
# cosign-signed upstream and never carry it, because no rule below names them.
# Measured 2026-08-26 across all 7 nodes: 755 of 755 cached
# `ghcr.io/devantler-tech/*` images carry the label, and 0 of 1299 cached
# images with a resolvable non-first-party repository name do β€” no exceptions.
#
# What is NOT established is the REFUSAL. Every first-party image pulled here
# has PASSED, so a FAILING decision has never been observed to block a pull.
# Proving that refusal is #3336. The original finding is #2856.
#
# Check this, do not assume it. The repository cannot answer it β€” the rules read
# as correct whether or not anything evaluates them:
# scripts/validate-image-verifier-liveness.sh
# Measured 2026-08-24: 7 of 7 nodes hold running rules and a running trust root.
#
# BEFORE ACTIVATING (#3101): an image that matches a rule but fails
# verification is refused, so any first-party image that is running yet
# unsigned goes ImagePullBackOff the moment the verifier starts working.
# BEFORE ACTIVATING (#3101): a matched image that fails verification is
# EXPECTED to be refused, so any first-party image that is running yet unsigned
# should enter ImagePullBackOff the moment it is next pulled. That refusal is
# exactly what #3336 exists to prove and has not been observed here, so treat it
# as the risk to design around rather than as a settled outcome.
# Enumerate the running first-party digests and confirm each carries a cosign
# signature β€” attestations alone do not satisfy the catch-all rule below β€” then
# sign or scope the rule off any that do not.
#
# Enumerated 2026-08-26 β€” of the four first-party images running on the
# cluster, ksail and provider-upjet-unifi verify against the exact keyless
# identity their rules require. wedding-app and ascoachingogvaner are PRIVATE
# packages: an unauthenticated cosign verify returns DENIED, which is an ACCESS
# result and NOT evidence that a signature is absent. Settling those two needs
# an AUTHENTICATED read, and this repository already stages one: the
# `πŸ”‘ Stage the GHCR pull credential` step in
# .github/workflows/validate-image-verifier-liveness.yaml decrypts the cluster's
# own Flux GHCR pull credential into DOCKER_CONFIG before it runs
# scripts/inventory-first-party-image-signatures.sh, so no ADDITIONAL
# `read:packages` token has to be minted. What is missing is a recorded RUN of
# that authenticated inventory against the fleet β€” the newest run of that
# workflow (2026-08-11) executed only its unit-test job β€” so until one is on
# record their signature state stays UNKNOWN, and they are the two images the
# catch-all rule governs.
#
# Declares the cosign signatures required at the containerd PULL layer on
# every node β€” a defense-in-depth complement to the GitOps-source
# verification Flux already performs on OCI *manifest* artifacts (apps'
Expand Down
Loading