Repository navigation
Prove an unsigned image is refused at pull, then activate the liveness check #3336
Description
🤖 Generated by the Agentic Engineer
Evidence
#3101's whole premise is that this cluster's supply-chain posture is asserted but not enforced. A wired verifier is not proof it enforces — that is exactly the inference #3101 was filed to stop being made.
Expected outcome
Refusal is observed, not reasoned about, and the daily liveness signal is switched on so the state cannot silently regress.
Acceptance criteria
- A known-unsigned image matching a rule is pulled on a real node and observed to be refused (
ImagePullBackOff), with the containerd-side reason captured. This is the "verify it actually works" step: static validation and a configuredbin_dirprove neither. - The probe uses a deliberately unsigned throwaway image, never a production workload, so the proof cannot itself cause an outage.
- A correctly signed image is pulled in the same session and succeeds, so the refusal is attributable to the signature and not to a broken verifier refusing everything.
.github/workflows/validate-image-verifier-liveness.yamlhas its dailyscheduletrigger restored and theworkflow_dispatch-only guard dropped from the fleet job, and its first scheduled run is green.
Why activation is last
The liveness check ships dormant in #3103 because its verdict is red until enforcement lands, and a permanently-red main spends the "something broke, fix it now" signal on a state this epic already tracks. It is switched on once the verdict is expected to be green.
Part of #3101.
Current prerequisite and remaining work
The inventory prerequisite #3334 is CLOSED. Authenticated workflow run 34045065606 completed successfully on 2026-09-06 at ac65b4a7163b50b988fe09835b760f09917b9d8e; both inventory and verifier-liveness jobs passed, and inventory reported PASS=5 FAIL=0 UNKNOWN=0. Its earlier workflow-dispatch blocker under devantler-tech/monorepo#3073 is superseded by that run. Keep the native dependency as the record of the prerequisite, but do not treat the closed issue as outstanding work.
The next implementation must provide a deliberately unsigned throwaway image and a correctly signed positive control, exercise the node pull layer directly so admission rejection cannot masquerade as node refusal, and define cleanup. Revalidate the inventory against the selected production state before the probe. Restore the daily schedule only after observed refusal and the signed control succeed, then verify the first scheduled run.
Blocker: production workflow_dispatch re-run of the signed-control image pull probe after #3783, which scheduled runs are not permitted to trigger | authority | last-verified 2026-09-24: unsigned refusal observed 2026-09-13, signed control still needs a clean uncached observation | asked slack 2026-09-14
Metadata
Metadata
Assignees
Type
Fields
Projects
- Status
Activity
chore: agent-claim #3336 nonce=cf266fb4a083355158236c6cfce67a49
devantler commentedon Aug 26, 2026
Findings from an attempt at this issue. Two of them change how the probe has to be built, so
recording them before the next attempt spends effort on a path that cannot work.
1. The premise has moved: a matched rule demonstrably DOES produce a decision.
containerd records a per-image
talos.dev/verifiedlabel for images Talos verified, and thelabel tracks rule MATCH rather than signedness. Measured across all 7 nodes: 755 of 755 cached
ghcr.io/devantler-tech/*images carry it; 0 of 1299 cached images with a resolvablenon-first-party repository name do — no exceptions.
ghcr.io/fluxcd/flagger-loadtesterandghcr.io/cloudnative-pg/cloudnative-pgare both keyless-cosign signed and both unlabelled,which is what rules out signedness and leaves rule match as the discriminator. Recorded in
#3378. This narrows this issue to what it always really was: proving the refusal.
2. A Pod-API probe can never test this layer. The
verify-app-imagesImageValidatingPolicy is
validationActions: [Deny]on pod CREATE cluster-wide, and it mirrorsthese same three rules. An unsigned first-party image submitted as a Pod is therefore rejected
at admission and never reaches the node pull layer this issue is about. The probe has to
bypass Kubernetes —
talosctl image pullagainst a node is the right instrument.3. There is no unsigned first-party image available to probe with.
doggy-countdown— theunsigned example #3101 cites in its Evidence — is now signed on every tag (
1.0.0,1.0.1,1.0.2,latest, and the threesha-*builds) and passes the catch-all rule identity. Thatevidence is stale. The only unsigned thing under a first-party path is a cosign signature
artifact (
:sha256-<digest>), which is unsigned by construction and matches the catch-all —usable, but it is an OCI artifact rather than a runnable image, so a pull failure would need its
reason read carefully to rule out a media-type confound.
Cleanest path is the one the acceptance criteria already describe: publish a deliberately
unsigned throwaway image under a first-party path, pull it on a node, and remove it after
(
talosctl image remove). That is a registry publish plus a node-level pull, and the node pullis a production mutation — outside what this agent may do unattended, so it needs the
maintainer, or an explicitly granted narrow permission.
Also worth carrying into the activation decision: of the four first-party images running today,
ksailandprovider-upjet-unifiverify against their exact rule identities.wedding-appandascoachingogvanerare private packages — an unauthenticatedcosign verifyreturnsDENIED,which is an access result, not evidence a signature is missing. Their state is genuinely
UNKNOWN until something with
read:packageschecks them, and they are the two images thecatch-all rule governs. Activating before that is settled risks exactly the ImagePullBackOff the
BEFORE ACTIVATING note warns about.
chore: agent-claim #3336 nonce=fad007f81d26cb2d1c2cd7d212694262
5 remaining items
devantler commentedon Sep 13, 2026
Reclassified: this is not blocked on maintainer authority, and the
blockedlabel is removed.The maintainer told me on 2026-09-13, in an interactive session, to stop parking work on him and resolve it myself. Re-reading this issue with that in mind, the authority framing does not hold:
probe-image-signature-enforcement.yaml(feat(security): add the dormant image-signature enforcement probe #3732). It runs on one node, needs an exact confirmation phrase, serializes onprod-deploy, and cleans up after itself.Plan, in order:
ghcr.io/devantler-tech/that matches the catch-all rule and is referenced by nothing else (draft PR, in progress).PASS, restore the liveness schedule per the acceptance criteria. OnFAILorINCONCLUSIVE, record the verdict here and fix the cause.chore: agent-claim #3336 nonce=bcfc3531de62cd0cb51829e376c20c5f
ci(security): verify the unsigned probe image with the job token
ci(security): refuse to publish the unsigned probe image from any ref…
ci(security): re-verify the probe's unsigned control by digest and ke…
devantler commentedon Sep 13, 2026
The refusal probe is running now.
ghcr.io/devantler-tech/unsigned-probe-throwaway@sha256:8c8bc0b3…and confirmed the image has no signature, attestation, SBOM or referrers.The probe runs behind any production deploy. If either image is already cached on the node, it reports INCONCLUSIVE rather than a false result. I'll record the verdict here.
devantler commentedon Sep 13, 2026
Verdict: PASS. A matched unsigned image is refused at the node pull layer.
Run: https://github.com/devantler-tech/platform/actions/runs/34777101451 (one production worker, 19:15Z):
ghcr.io/devantler-tech/*rule and were confirmed absent from the node's content store beforehand, so the pulls were real, not cache hits.This is the first acceptance criterion, observed rather than reasoned about.
Remaining: restore the daily
scheduleonvalidate-image-verifier-liveness.yaml, drop the dispatch-only guard from the fleet job, and see its first scheduled run green. A draft PR for that is being prepared; this issue closes once that first scheduled run is green.Noticed, not blocking: cleanup warned it "could not remove" the unsigned ref from the node. The pull was refused, so nothing should have been stored, and the warning is most likely the probe treating "not present" as a removal failure. There was also a harmless
printf: Broken pipemessage. Both are tracked separately.chore: agent-claim #3336 nonce=8e1f3a144ee58ed026a2950ec06cb7e2
docs(security): state the observed image refusal in verifier comments
devantler commentedon Sep 13, 2026
The 2026-09-13 probe pass is weaker than it looked.
Consequence for this issue: the signed control still needs a clean observation. After #3783 merges, re-run the probe with a signed image the probed node does not already hold. If the node holds it, the probe now stops as inconclusive instead of passing. The daily liveness schedule (#3782) is unaffected, because it doesn't depend on this check.
devantler commentedon Sep 30, 2026
All acceptance criteria are now observed. Production probe 36783104809 passed on the reviewed current main revision:
The daily liveness schedule is already active from the merged activation change. The latest three scheduled runs are green, including 36670771496 on September 30. The old cached-control evidence and workflow-dispatch blocker are superseded by this run.