Repository navigation
Description
🤖 Generated by the Agentic Engineer
Evidence
talos/cluster/verify-first-party-images.yaml ends with a catch-all rule over ghcr.io/devantler-tech/* that requires a keyless cosign signature issued by the shared publish-app.yaml workflow. Any image matching a rule but failing verification is supposed to be blocked at the containerd pull layer.
The public first-party package doggy-countdown is currently deployed from this org and carries no cosign signature at all. Read anonymously from GHCR (the package is public, so no credentials were involved):
- the image manifest for the running digest resolves — HTTP 200;
- the cosign signature tag for that same digest — HTTP 404;
- the repository's tag listing contains a
sha256-<digest>referrer-style tag (anactions/attestartifact,artifactType: application/vnd.oci.empty.v1+json) but nosha256-<digest>.sigentry.
So the artifact has attestations and no signature. Despite that, the workload is running, and one of its replicas is on a node that was provisioned after the image was published — meaning a fresh pull of an unsigned first-party image succeeded on a node that had no cached copy.
Why it matters
This is the node-pull layer — the control that decides whether image bytes may run at all, and the backstop beneath the Kyverno admission layer. Two readings, and both are worth knowing:
- The rule is not being enforced as believed. Then every conclusion drawn from "first-party images are signature-gated at pull" is unfounded, and the control is providing assurance it does not deliver.
- The rule is enforced and this image is an exception for some reason not visible in the file. Then the reason is undocumented, and the next person to reason about this policy will reach the wrong conclusion too.
Either way the file and reality disagree, and the file is what people are trusting.
There is also a latent availability edge: if enforcement is working and this image simply predates it on the nodes that hold it, then the next node to pull it fresh gets ImagePullBackOff. The in-file comment records exactly that failure mode happening once before.
Scope note — this blocks tightening
#2815 proposes narrowing these same matchers. That work assumes the current rules are enforcing and that every running first-party image satisfies them. This finding contradicts that assumption for at least one image family, so #2815's evidence step should establish enforcement state first — narrowing a rule that is not enforcing changes nothing except the confidence people place in it.
Acceptance criteria
- Determine, from the live machine configuration rather than from the repository, whether
ImageVerificationConfigis active on nodes and which rules it carries. - Explain why an unsigned first-party image pulls successfully today.
- Either the image family becomes signed by the expected workflow, or the rule is amended to describe what is actually true — with the reason recorded in the file.
- A signal exists that would catch this disagreeing again, rather than it being found by hand.
Rough size: S–M (mostly investigation).
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- StatusShow more project fields✅ Done
Activity
coderabbitai commentedon Jul 29, 2026
🔗 Related PRs
#2382 - fix(policies): accept the provider publish-workflow cosign identity in verify-app-images [merged]
#2416 - fix(talos): trust provider-upjet-* publish-provider-package.yml signing identity [merged]
#2429 - fix(policy): trust the actions publish-app identity for app-image verification [merged]
#2457 - fix(talos): accept the actions publish-app identity at node image verification [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Aug 11, 2026
Re-checked the premise 13 days on — it holds, but my probe needs a positive control before anyone acts on it
Not claiming this issue; this is a read-only revalidation so whoever picks it up starts from current
facts rather than 07-29 ones.
doggy-countdownis still unsigned by this issue's own method. Anonymous GHCR read, 2026-08-11:latestresolves tosha256:6a4285ef…, and the correspondingsha256-6a4285ef….sigtag returnsHTTP 404. So the disagreement between
talos/cluster/verify-first-party-images.yamland realityhas not closed on its own.
But the same probe returns 404 for
ksailtoo, and that is the part worth flagging. Two readings,and I could not separate them:
.sigsibling tags(the referrers API and
actions/attestartifacts are both in play here, and this issue alreadyrecords seeing a referrer-style tag on
doggy-countdown).I have no image I know to be signed by
publish-app.yamlreturning HTTP 200, so I cannot tell agenuinely unsigned image from a probe looking in the wrong place. A 404 with no positive control is a
claim about the probe, not about the artifact.
Suggested addition to the acceptance criteria: establish a positive control first — one image
whose signature this method does find — and record it in the issue. Every later measurement here,
including the "is it signed now" check that closes this issue, is uninterpretable without it. That is
cheap and it protects the criterion 3 decision (sign the family vs. amend the rule) from being made
on a measurement artifact.
Criterion 1 still needs the live Talos
ImageVerificationConfigread, which I did not attempt.devantler commentedon Aug 11, 2026
Evidence refreshed 2026-08-11 — the finding still holds, and one of its supporting arguments needs correcting
The image is still unsigned. Read anonymously from GHCR today (public package, no credentials):
latestresolves tosha256:6a4285efcbdae0a8d69867ca92378cc5bb7547a621afe47ce1a96f90c33603d0sha256-6a4285ef….sig→ HTTP 404referrersAPI for that digest → emptyThat exact digest is what is running. Both replicas report
imageID=ghcr.io/devantler-tech/doggy-countdown@sha256:6a4285ef…, onprod-worker-1andautoscale-cx33-27e0aeaa0c6be36.Correction: the "node provisioned after the image" argument does not hold — but the conclusion is stronger without it
The original evidence argued that one replica sits on a node provisioned after the image was published. Checked directly, that is not true of any current node:
prod-worker-1autoscale-cx33-12ac9ae5b0b9effeautoscale-cx33-27e0aeaa0c6be36autoscale-cx43-d63a60203d535b1Every node predates the image, whose config blob records
created: 2026-07-26T17:06:19Z.That inverts the argument in a way that makes it stronger, not weaker. The original framing invited the rebuttal "the other nodes just have it cached from before enforcement". They cannot: the digest did not exist until 2026-07-26, so every node now running it necessarily pulled it fresh after that date. A warm containerd cache cannot explain any of these pulls.
And the rule predates the pull:
talos/cluster/verify-first-party-images.yamlhas been at this path since 2026-07-06 (4ee654ab), with the cosign subject pinning tightened on 2026-08-05 (a8e6404c) and 2026-08-10 (2126815b).doggy-countdownmatches no earlier rule, so it falls to theghcr.io/devantler-tech/*catch-all requiring a keylesspublish-app.yamlsignature — which it does not have.So the disagreement is real and current: an image that matches a rule, fails that rule, and runs anyway.
What remains unverified — and why
Acceptance criterion 1 asks for the live machine configuration rather than the repository. I could not satisfy it:
talosctlis not installed on the agent host (~/.talos/clustersexists, but the binary does not), so I could not read whetherImageVerificationConfigis actually active on the nodes or which rules it carries.That is the one question that discriminates between the issue's two readings, and I am deliberately not guessing at it. The likeliest mechanism worth testing first is that the patch reaches nodes only through a Talos
cluster update, so the config on disk may lag the repository — but that is a hypothesis, not a finding, andksail.prod.yamldoes not reference this file explicitly (whether KSail auto-discoverstalos/cluster/was not confirmed here).Next step for whoever picks this up: install
talosctlon the runner and read the appliedImageVerificationConfigfrom a node — ideallyprod-worker-1and one autoscaler node, since a divergence between them would itself be the answer.Leaving this unassigned; the remaining work is gated on tooling I do not have rather than on analysis.
devantler commentedon Aug 11, 2026
Acceptance criterion 1 answered — read from the live machine configuration
talosctlis now available on the runner, so I could do the read that was blocked earlier. TheImageVerificationConfigIS active on the nodes, and it carries the current rules.talosctl get imageverificationruleson both nodes hosting the workload —prod-worker-1(10.0.1.4) and the autoscaler nodeautoscale-cx33-27e0aeaa0c6be36(10.0.1.11) — returns identical output:ghcr.io/devantler-tech/ksail*ghcr.io/devantler-tech/provider-upjet-*ghcr.io/devantler-tech/*There is no node divergence — the hypothesis that the autoscaler node might be missing the config is refuted, even though it runs an older Talos (v1.13.4 vs v1.13.7 on the static nodes).
Rule
0002— the onedoggy-countdownfalls to — carries the current, tightened verifier, matchingmainafter2126815b:And containerd has the verifier plugin loaded:
io.containerd.image-verifier.v1.bindir — loading plugin.So this is not the "rule is not deployed" reading. The config on the nodes and the file in the repository agree. The disagreement is between the config and observed behaviour.
A hypothesis I tested and discarded — do not reuse it
The rule resources report
created: 2026-08-11T15:14:53Z, which looks like "the rules only landed minutes ago". That reading is wrong.nodename(15:14:53Z) andimagecacheconfigs(15:14:39Z) carry the same stamp, andmachinestatusshows the node enteringrunningat15:14:39Z— the node had just rebooted under the in-flightcluster update. Talos resourcecreatedtimestamps reset on every config reload, so they say nothing about when a rule first reached a node. Recording this so the next investigation does not spend the same effort or draw the false conclusion.Where that leaves the "why"
The remaining explanation consistent with all the evidence is that verification happens at pull time only, and the pull already happened. The running pods have not re-pulled since enforcement reached these nodes, and containerd serves the cached image. That is consistent with the in-file comment recording an
ImagePullBackOffthe one time a fresh pull did hit the rule.I am labelling that a hypothesis, not a finding — proving it needs a fresh pull of the unsigned digest on one of these nodes, which is a cluster mutation I must not perform. What would settle it without a mutation is the first natural reschedule of this workload onto a node with a cold cache: under these rules it should fail with
ImagePullBackOff.Updated criteria status
ImageVerificationConfigis active on nodes; all three rules present, current subject pinning, no divergence between static and autoscaler nodes.doggy-countdowngets signed bypublish-app.yaml, or the rule records what is actually true. Signing it is the better fix — it is the only first-party app image not covered, and the rule is correct as written.Still leaving this unassigned. Criterion 3 is a
doggy-countdownCI change and criterion 4 is a detection control; they are separable and neither is blocked any more.devantler commentedon Aug 11, 2026
Acceptance criteria 1 and 2 are now answered, and the answer is the first of the two readings in the
issue body: the rule is not enforcing. The cache explanation is refuted by direct observation
rather than by inference.
Criterion 1 — the live rules (read from the node, not the repo)
All three rules are present and active on
prod-worker-1, carrying the current tightenedsubjectRegexvalues, withskip: falseanddeny: false. The catch-all is:So the matcher does cover
ghcr.io/devantler-tech/doggy-countdown, and nothing is skipping it.Criterion 2 — why the unsigned image pulls: it just did, on a node with the rule active
The obvious defence was that the image predates enforcement on the nodes holding it and is simply
cached, since containerd verifies at pull. That is now ruled out — the image was pulled fresh from
the registry today, minutes after the rule was confirmed in place:
0002present onprod-worker-1(resource stamp; see caveat below)Pulling image "ghcr.io/devantler-tech/doggy-countdown@sha256:6a4285ef…"Successfully pulled image … in 3.021s. Image size: 23108742 bytesA
Pullingevent followed bySuccessfully pulled … in 3.021swith a byte count is a real registryfetch. A cache hit reports
Container image … already present on machineand emits noPulling— sothis was not a cached layer being reused. That image carries no cosign signature (the evidence in the
issue body still holds), the active rule requires one, and the pull succeeded anyway.
createdstamps reset on everymachine-config reload, so
15:49:52Zis the most recent reload, not when the rule was introduced.It bounds the claim in the direction that matters — the rule was in place before the pull — and the
rules were independently observed active on this node earlier in the day as well.
Conclusion: the pull-layer control is configured as intended and is not blocking an image that
fails it. Any reasoning that treats "first-party images are signature-gated at pull" as an enforced
property is currently unfounded, and #2815's narrowing work should not assume otherwise.
What this leaves
Criterion 2 is answered. Criteria 3 and 4 are unchanged and now separable:
doggy-countdownstarts being signed by the expected workflow, or the rule isamended to describe what is actually true, with the reason recorded in the file. Given the finding
above, amending the file alone would be recording a control that does not work; the underlying
non-enforcement needs its own answer first (Talos-side behaviour vs. our configuration).
found by pulling node state by hand, and the same class of gap (a control believed active, silently
evaluating nothing) is the one flagged in the Kubescape notes — a check that asserts a known-bad
image is actually refused would catch it, whereas asserting the config exists would not have.
I have not claimed this issue — this is investigation recorded for whoever picks up 3 and 4.
devantler commentedon Aug 11, 2026
Criterion 3 unblocked: the root cause is ours, and the control has never enforced
Claiming this issue. The previous comment left criterion 3 blocked on a fork — "the underlying
non-enforcement needs its own answer first (Talos-side behaviour vs. our configuration)". That fork
is now resolved, from live node state: it is our node image. The verifier component that would
consume these rules is not installed, so
ImageVerificationConfigis producing rules that nothingreads.
The mechanism
containerd does not verify images itself. It loads an image-verifier plugin, and the only one
present on these nodes is the
bindirplugin, which works by executing every binary in a configuredbin_dir. Both containerd instances onprod-worker-1log it loading and nothing further:containerd's own documentation states the behaviour for an unconfigured directory verbatim:
That is exactly the state of these nodes.
Evidence, each with its control
bin_dirconfigured?verif|bin_dir|bindiracross every/etc/cri/conf.d/*file and/etc/containerd/config.tomlreturns nothing. The renderedcri.tomlconfiguresimages,runtimeandnrionly./opt/containerd/image-verifierlstat … no such file or directory./opt/containerd/bin,/opt/containerd/lib.). Control:ls -l /opton the same node correctly listscniandcontainerd, so the listing does show children when there are any.verif|cosign|sigstore|notationin/usr/local/bin,/usr/local/lib/containerd,/usr/bin,/sbin,/opt. Control:/usr/binand/sbinenumerate 140 entries each, so the probe reads real directories;/usr/local/bincontains exactly one binary,nsenter.iscsi-tools,qemu-guest-agent,util-linux-tools,schematic. Nothing signature-related.So the chain is complete and consistent with the observed pull: the rules are correct and active
(criterion 1), no verifier binary exists to evaluate them, containerd's documented response to that
state is to permit the pull, and an unsigned matching image duly pulled fresh and ran (criterion 2).
What this rules out
This is not a rule-authoring error, not first-match-wins ordering, not a stale node
config, and not a containerd cache artifact. Every one of those was a live hypothesis on this
issue at some point. The rules are fine; nothing is reading them.
It also means amending
verify-first-party-images.yamlwould be the wrong fix. The previouscomment already suspected this — "amending the file alone would be recording a control that does not
work" — and that now has a mechanism behind it. Rewriting the rules to match observed behaviour
would document a control that cannot enforce regardless of what the rules say.
Consequence for #2815
#2815 proposes narrowing these matchers.
Narrowing a rule set that nothing evaluates changes nothing except confidence. #2815 should be
sequenced after enforcement is real, and its evidence step can now be satisfied by pointing here
rather than re-deriving it.
The one thing I could not establish — stated as an open question, not a guess
Which mechanism is supposed to supply the verifier binary. The Talos v1.13
ImageVerificationConfigreference page documents the rule schema — ordering,skip,deny,keyless and public-key verifiers — but says nothing about what enforces them, whether a system
extension is required, or what
bin_dirshould be. I am deliberately not asserting "a cosignextension is missing": I have positive evidence that no verifier is installed and none is wired, and
no evidence about which supported mechanism is meant to install one. That is the next question, and
it decides the shape of the fix (schematic/extension change vs. containerd config vs. both).
Criterion 4 gets a cheaper form as a result
The signal this issue asks for now has an obvious low-cost shape alongside the gold-standard one,
because we know precisely which condition failed:
bin_diris configured and contains atleast one executable, on every node. This exact check would have caught this on day one, and it is
a plain node read.
real proof and stays the target; the liveness check is what makes its absence visible in the
meantime.
Asserting the rules exist would still not have caught this — that was already true and remains the
trap. This is the same failure class as the Kubescape notes: a control believed active while
evaluating nothing, reading identically to a compliant one.
Next
I will split the remedy and the signal into their own issues under this one rather than growing this
thread further, and take the liveness check first since it is shippable without a node rollout.
fix(security): check each containerd independently and never pass an …
fix(security): stop the liveness checker passing a fleet it never ful…
chore: agent-claim #2856 nonce=f90da85947d56f7c26e5e01cfa2f0499
devantler commentedon Aug 18, 2026
Live re-measurement — the disagreement is still open, and the cause is settled
Re-verified against the cluster today rather than inferred from the repository.
Criterion 1 — is
ImageVerificationConfigactive on nodes? The rules are present and syntactically valid (talosctl validatepasses), but they are not evaluated.scripts/validate-image-verifier-liveness.shagainst prod, 2026-08-18:Every node, on both
/etc/cri/conf.d/cri.tomland/etc/containerd/config.toml: nobin_dirin theio.containerd.image-verifier.v1.bindirtable.Criterion 2 — why does an unsigned first-party image pull successfully? Because containerd does not verify images itself. It delegates to that plugin, and its documented behaviour when
bin_diris unset, absent, or empty is to permit the pull. No verifier binary is installed fleet-wide, so the rules below it never run. The image is not an exception to an enforcing rule — there is no enforcement at all. Installing and wiring the verifier is #3101.The unsigned image is still unsigned. Confirmed anonymously against public GHCR for the digest currently running (
doggy-countdown, 2 replicas):sha256-<digest>.sig(cosign signature)So criterion 3's first branch ("the image family becomes signed") is not met.
What this PR does — and deliberately does not do
#3219 takes criterion 3's second branch: it amends the file to describe what is actually true, with the reason recorded in it. Until then the file asserted the opposite — that a matching-but-unverified image is blocked — which is the "the file and reality disagree, and the file is what people are trusting" complaint in this issue's own body.
It does not install the verifier. That is #3101, and it is a node-config rollout rather than a comment change.
A hazard #3101 should know about before it activates
These two findings compose into something neither shows alone:
ghcr.io/devantler-tech/doggy-countdownis running, unsigned, and matches theghcr.io/devantler-tech/*catch-all rule. Enforcement is the only thing currently absent. So the moment #3101 makes the verifier work, that workload's next fresh pull is refused and it goes ImagePullBackOff — the "latent availability edge" this issue predicted, now with a named image.That wants resolving before activation, not during it: either sign the image family through
publish-app.yaml, or scope the rule off it with a recorded reason. I have written the warning into the policy file itself, since that is where the person doing the activation will be looking.Criteria status
bin_dir⇒ containerd permits every pull.validate-image-verifier-liveness.sh+ its workflow) but is deliberately dormant:workflow_dispatchonly, because it would report red fleet-wide every day while Establish that first-party image signature verification actually enforces on the fleet #3101 is open. Activation is one line, and belongs to Establish that first-party image signature verification actually enforces on the fleet #3101. Worth noting the signal has therefore never run on a schedule — today's run was by hand, which is the same way this issue was found.chore: agent-claim #2856 nonce=5e634c614a5a211e3b6f78ee5a32345b
devantler commentedon Aug 19, 2026
Closing — all four acceptance criteria are met
Re-verified live against the prod cluster today (2026-08-19) rather than inferred from the
repository, per this issue's own instruction to determine enforcement state from the live machine
configuration.
Live measurement:
scripts/validate-image-verifier-liveness.shexits 1 —7 of 7 nodes cannot enforce image verification, on both
/etc/cri/conf.d/cri.tomland/etc/containerd/config.toml. This matches the 2026-08-18 measurement, so the state is unchangedand the documented status is accurate. (Per-node detail stays in the private operator notes.)
Criteria
io.containerd.image-verifier.v1.bindirtable carries no
bin_diron any node, so theImageVerificationConfigrules are neverevaluated.
delegates to the bindir plugin and, per its documented behaviour, permits the pull when that
directory is unset, absent, or empty. The rules were correct throughout — they were simply
never consulted.
talos/cluster/verify-first-party-images.yamlleads with a
STATUS — THESE RULES ARE INERT ON THIS CLUSTER TODAYblock recording themechanism, the measurement, and the pre-activation hazard that
ghcr.io/devantler-tech/doggy-countdownis running unsigned and matches the catch-all rule.scripts/validate-image-verifier-liveness.shasserts the enforcementpath rather than the configuration — deliberately, since asserting the rules exist would not
have caught this, the rules having been correct the whole time. Its own tests run in
ci.yaml'svalidatejob, whichCI - Required Checksdepends on, so a regression in the checker blocks amerge rather than merely reporting.
Why the fleet check is dormant, and why that is not a gap
The fleet sweep is
workflow_dispatch-only on purpose. The verifier is absent fleet-wide, so a dailyrun would publish RED every day for a condition #3101 already tracks — spending the repository's
"something broke, fix it now" channel on known, open work. Turning on the cadence is one reversible
line, and it belongs with the activation.
What remains, and where
Installing and wiring the verifier is #3101, which also carries the blocker that must be cleared
first: an image matching a rule but failing verification is refused, so
doggy-countdowngoesImagePullBackOff the moment the verifier starts working unless it is signed or the rule is scoped off
it beforehand.
This issue asked what the disagreement was, why it existed, and whether we would notice it again.
All three are answered and recorded, so it closes here; the remaining work is activation, not
investigation.