Sign or scope every first-party image that a verifier would refuse, before enforcement goes live

🤖 Generated by the Agentic Engineer

Evidence

#3101 measured that doggy-countdown is unsigned today and pulls and runs fresh, because nothing evaluates the ImageVerificationConfig rules. The moment a working verifier is wired, every image matching a rule but lacking a valid signature starts failing to pull at once.

Impact

This is the outage in #3101. Turning on a control that has never enforced converts every latent violation into a simultaneous ImagePullBackOff across the fleet. The blast radius is unknown until it is measured, and it is measurable now, without touching a node.

Expected outcome

A complete, current inventory of which first-party images match a rule in talos/cluster/verify-first-party-images.yaml and whether each one carries a signature that rule would accept — followed by signing them or narrowing the rule, so that enabling enforcement is a no-op for every workload.

Acceptance criteria

Every image matching a rule is enumerated from the rendered production manifests, not from a hand-written list, so a workload added later cannot be silently missed. Report the enumeration method.
Each is checked against the rule's current subjectRegex with cosign verify, and the pass/fail result recorded. A check that errors is recorded as UNKNOWN, never as a pass.
Every failing image is either signed through the existing publish path, or its rule is narrowed with the reason stated — no image is left in a state where enforcement would refuse it.
A re-run of the inventory reports zero failures and zero UNKNOWNs, and that re-run is what closes this issue.
The inventory runs in CI so a newly added unsigned first-party image fails closed rather than waiting for the next manual pass.

Why this is separate from wiring the verifier

It is the only part of #3101 that can be done safely today, it needs no node change, and it must land before the verifier so the rollout is uneventful. It is independent of the Spike's answer: whatever supplies the binary, the set of images that would be refused is the same.

Part of #3101.

Blocker: devantler-tech/monorepo#3073 | last-verified 2026-09-03: agent runtime denies workflow dispatch (classifier), so the authenticated inventory run this issue needs cannot be started by any agent lane; needs a maintainer dispatch of validate-image-verifier-liveness.yaml or the permission grant in that issue. Adding a schedule: is NOT an acceptable substitute — that workflow gates its daily cadence behind #3336 establishing refusal first.

chore: agent-claim #3334 nonce=fb5c3f21d56bbc7fab32a11bd93cdf2d

🤖 Generated by the Agentic Engineer

Revalidated this issue's premise before implementing it, and two things changed that affect how
AC1 should be built. Recording them here rather than implementing an enumeration that would report
clean while having checked the wrong set.

The stated blast radius is gone

doggy-countdown — the one image this issue measured as unsigned and running — was deregistered and
its repository archived on 2026-08-24 (archived=true, re-read today). The known latent violation
this issue was filed around is no longer on the fleet, so the inventory has to be rebuilt from
current state rather than reusing that finding.

AC1's enumeration source needs deciding first — a literal reading of it misses the images that pull

Two classes of ghcr.io/devantler-tech/* reference match the catch-all rule textually but do not
behave alike:

Flux OCI artifacts — the oci://ghcr.io/devantler-tech/{wedding-app,ascoachingogvaner,aws,github-config}/manifests
OCIRepository URLs are pulled by Flux's own client, not by containerd, so Talos
ImageVerificationConfig never evaluates them. They already carry their own verify blocks.
Inventorying them would measure something this control cannot act on.
HelmRelease-derived container images — ksail and provider-upjet-unifi are the two
first-party images with dedicated rules, and neither appears as an image: field in the manifests.
They come from chart values, so an enumeration that scans rendered kustomize output finds exactly
one ref (wedding-app) and silently misses both images that have their own rules.

That second point is the one that matters: it is precisely the failure AC1 exists to prevent — "a
workload added later cannot be silently missed" — reached by following AC1 literally. The
enumeration has to resolve to what containerd actually pulls, which means rendering HelmReleases
too (flux build / helm template), reading live pod specs and accepting that it lags a newly added
workload, or a hybrid that fails closed on anything it cannot resolve. Worth settling explicitly
before code is written.

Measured signature state today

Against each rule's own identity, unauthenticated, from the published registry:

Image	Rule	Result
`ksail:v7.41.2`	`ksail*` → `cd.yaml@refs/tags/v.+`	PASS
`provider-upjet-unifi:v0.1.0`	`provider-upjet-*` → `publish-provider-package.yml@refs/heads/main`	PASS
`wedding-app:latest`	catch-all → `publish-app.yaml@<sha>`	UNKNOWN — registry denied the read without credentials

Recorded as UNKNOWN, never as a pass, per AC2. It also adds a requirement the ACs do not yet state:
the inventory must perform an authenticated GHCR read, or it will return UNKNOWN for every private
first-party package and a re-run can never reach "zero UNKNOWNs" — which is the condition AC4 makes
the closing criterion.

What this changes

Nothing about the goal, and no AC is dropped. AC1 gains a prerequisite decision on the enumeration
source, and AC2 gains an authentication requirement. Both are cheap to settle and expensive to get
wrong, because every failure mode here is the same one: reporting clean while having checked nothing.

No claim is made that these three are the complete set — this was a repository scan, not a full
render, which is exactly the gap AC1 has to close.

chore: agent-claim #3334 nonce=9a7dbed2811517d16b3ebc29df060944

chore: agent-claim #3334 nonce=db49024d92315a10d37ecfaa79d73862

chore: agent-claim #3334 nonce=d3bf59a235ec010c75f5f48afc222f47

chore: agent-claim #3334 nonce=4f0e3ffd910ccf700a907264602960ed

🤖 Generated by the Agentic Engineer

Inventory measured against the live fleet — 3 of 5 confirmed, 2 unmeasurable, and the premise has moved

Measured 2026-08-25 on admin@prod. Every running first-party image was resolved to its digest from containerStatuses[].imageID, then verified with cosign against the exact rule it would match in talos/cluster/verify-first-party-images.yaml — same issuer, same subjectRegex, first-match-wins order — so this reports what the verifier would decide, not whether a signature exists.

Control: 129 distinct images run on the fleet; 5 match ghcr.io/devantler-tech/*. Every workload spec (Deployment/StatefulSet/DaemonSet/CronJob) was swept as well as running pods, and the two lists agree.

image	rule	verifier would
`ksail`	1 — ksail `cd.yaml` @ `refs/tags/v*`	pull ✅
`provider-upjet-unifi`	2 — `publish-provider-package.yml` @ `refs/heads/main`	pull ✅
`doggy-countdown`	3 — `publish-app.yaml` @ 40-hex SHA	pull ✅
`wedding-app`	3	UNDETERMINED ⚠️
`ascoachingogvaner`	3	UNDETERMINED ⚠️

The premise of this issue is now stale

It opens with "#3101 measured that doggy-countdown is unsigned today". It is not unsigned now — it verifies keylessly against the catch-all's publish-app.yaml identity at a 40-hex ref. Whatever was true when #3101 was written, the blast-radius argument can no longer rest on that image.

The two undetermined ones are a MEASUREMENT gap, not a finding

cosign returns DENIED: requested access to the resource is denied on the .sig tag for both, anonymously and with this agent's token. That is not evidence they are unsigned — GHCR answers DENIED for "no access" and "no such tag" alike, so the result is genuinely unknown in both directions.

Two controls establish that it is access, not absence:

doggy-countdown verified anonymously, so the method works and those packages are readable.
A direct registry probe was run to separate the cases and its own positive control failed — doggy-countdown's .sig returned 404 through that path while cosign had just verified it, and a running image's own manifest 404'd too. So those HTTP codes prove nothing and are discarded rather than reported.

The blocker is a token scope: gh api orgs/devantler-tech/packages/container/<pkg> returns 403 — You need at least read:packages scope for all five packages, including the three that verify. Granting read:packages to the agent's token is the one step that closes this, and it is a maintainer action.

Blocker: agent token lacks read:packages (maintainer authority — token scope) | last-verified 2026-08-25: 403 on all 5 packages, while 3 of 5 verified anonymously via cosign, so the gap is scope-only and not a cluster or manifest problem.

What this changes for #3101

Enforcement's blast radius is now bounded rather than unknown: three of five running first-party images are confirmed to pass their own rule. The remaining exposure is exactly two app images, both under the same catch-all rule, and both resolvable by one scope grant — not by any manifest change.

Reproducing

kubectl --context admin@prod get pods -A -o json \
  | jq -r '[.items[] | (.status.containerStatuses[]?, .status.initContainerStatuses[]?) | .imageID]
           | map(select(startswith("ghcr.io/devantler-tech/"))) | unique | .[]'
# then, per image, cosign verify --certificate-oidc-issuer <issuer> \
#   --certificate-identity-regexp <the subjectRegex of the FIRST matching rule> <image@digest>

Read-only throughout; nothing on the cluster or in the registry was mutated.

🤖 Generated by the Agentic Engineer

`blocked` label removed — this issue has no blocker

Checked live before touching it:

repos/devantler-tech/platform/issues/3334/dependencies/blocked_by → empty. The only
dependency edge is blocking Install and wire the containerd image verifier on every node #3335, which is closed.
The body carries no **Blocker:** line at all, so skip clause (b) of Drain oldest-first
was never satisfiable — the label was doing the skipping on its own.
The body says the opposite of the label: "it is measurable now, without touching a node"
and "It is the only part of Establish that first-party image signature verification actually enforces on the fleet #3101 that can be done safely today".

Every other blocked security issue ahead of this one was re-checked in the same pass and each
carries a proper live-verified blocker line naming maintainer authority (#2867/#3251, #2889,
ksail#6428, .github#132). Those skips stand. This one did not.

First blast-radius measurement (AC 1 + AC 2)

Enumerated from the live prod fleet rather than a hand-written list — every
containers/initContainers/ephemeralContainers image across all pods, resolved to the digest
the node actually pulled (.status.containerStatuses[].imageID), then matched against the rules in
talos/cluster/verify-first-party-images.yaml in order, first match wins, exactly as Talos
evaluates them.

Image (as pulled)	Rule	Verdict
`ksail:v7.178.36`	1 — `ksail*`	PASS
`provider-upjet-unifi:v0.1.0`	2 — `provider-upjet-*`	PASS
`doggy-countdown@sha256:15fbea57…`	3 — catch-all	PASS
`ascoachingogvaner@sha256:7be66c68…`	3 — catch-all	UNKNOWN
`wedding-app@sha256:c2c4b786…`	3 — catch-all	UNKNOWN

Confirmed failures: 0. UNKNOWNs: 2.

The two UNKNOWNs are an authorization gap, not a missing signature

cosign verify returns DENIED on the .sig for both. That alone cannot tell "unsigned" from
"unreadable", so it was discriminated with an anonymous registry probe of the image manifest
itself:

ascoachingogvaner, wedding-app → 401 on the image manifest ⇒ the packages are private and
the local credential lacks read:packages. Recorded UNKNOWN, never a pass, per AC 2.
Control: doggy-countdown's .sig tag returns 404 anonymously while its image manifest reads
fine — a public package whose signature is stored as an OCI referrer rather than a .sig tag,
which cosign resolves. So the probe genuinely separates 401 (unreadable) from 404 (absent), rather
than reporting one shape for both.

CI already has the credential these two need — scripts/ghcr-auth-lib.sh is the established bridge
(ci.yaml, run-ksail-prod-with-pull-auth.sh) — so the credentialed re-run that clears both is the
closing condition in AC 4, not new design work.

🔴 The stale-example warning in this issue understates the correction

The body warns that doggy-countdown is a stale example because the product was deregistered and
its repository archived on 2026-08-24. Two corrections, both measured just now:

It is still running on prod. It is in the live pod set today, so it is in the blast radius
whatever happened to its repository.
It verifies. It PASSES rule 3 against the current subjectRegex. It is the image Establish that first-party image signature verification actually enforces on the fleet #3101
and this issue both name as the known-unsigned first-party workload, and the premise that the
blast radius is non-empty rests on it. On today's evidence that premise does not hold.

That materially de-risks #3101: on current evidence, activating enforcement refuses nothing
that is confirmed-readable, and the residual risk is confined to the two private packages.

What remains

AC 3 (sign or narrow) currently has nothing to act on — no confirmed failure exists. AC 4/AC 5
(a re-run reporting zero failures and zero UNKNOWNs, running in CI) are the real remaining work,
and that is what the follow-up child ships.

chore: agent-claim #3334 nonce=9390f99cdabc52a0f1f460f83db6e2f1

fix(security): simplify the probe credential file and drop an unused …

fix(ci): follow the sourced library when shellchecking the inventory …

🤖 Generated by the Agentic Engineer

The machinery this issue needs already exists and has never been run. Naming the exact next action so it stops being re-derived.

Investigating this while finishing #3378, I checked whether the inventory could be produced at all, and found the gap is narrower than "build the inventory":

.github/workflows/validate-image-verifier-liveness.yaml already implements this issue's CI criterion. Its 🔏 Inventory First-Party Image Signatures job enumerates from the rendered production manifests (not a hand-written list), stages the cluster's own Flux GHCR pull credential into DOCKER_CONFIG, and runs scripts/inventory-first-party-image-signatures.sh.
It has never been dispatched. The workflow is workflow_dispatch-only, and its newest runs are three pull_request events from 2026-08-11, which predate that trigger change. So there is no fleet inventory on record — which is exactly why the two private images below are still UNKNOWN.
Nothing gates the dispatch except the branch. The prod environment carries a single branch_policy protection rule (protected branches) and no required reviewers, so a workflow_dispatch against main is the whole action.

The checkers themselves are sound — it really is only the run that is missing. At 8479b6f5a1 both unit suites pass locally: scripts/tests/test-validate-image-verifier-liveness.sh (all cases passed) and scripts/tests/test-inventory-first-party-image-signatures.sh (23 named cases, all checks passed) — including the cases that pin this issue's own acceptance criteria, notably a 2xx read of an unsigned image is still FAIL, HTTP 404/429/500 -> UNKNOWN (not FAIL), one UNKNOWN withholds exit 0, and empty enumeration -> exit 2, never 0.

What the run would settle. Of the four first-party images running, ksail and provider-upjet-unifi already verify against the exact keyless identity their rules require. wedding-app and ascoachingogvaner are private packages: an unauthenticated cosign verify returns DENIED, which is an access result and not evidence that a signature is absent. Those two are the images the catch-all rule governs, and the authenticated job is what turns them from UNKNOWN into a pass or a fail.

Note for whoever picks this up: an unattended agent session on the current host cannot produce this evidence locally — talosctl there resolves to an access path that is not present in that context, so scripts/validate-image-verifier-liveness.sh fails closed with refusing to report a fleet that was not checked. That is the script behaving correctly, not a defect, and it is the reason this belongs in the workflow rather than in a local run. Dispatching that workflow is currently outside what this agent may trigger, so I am recording the action rather than performing it.

Suggested next step, in one line:

gh workflow run validate-image-verifier-liveness.yaml --repo devantler-tech/platform --ref main

Then record the per-image PASS / FAIL / UNKNOWN result here, and update the enumerated note in talos/cluster/verify-first-party-images.yaml so the repository stops carrying an UNKNOWN it can now answer.

chore: agent-claim #3334 nonce=90cec9a9ccc0eaafbdaf7ffb6208436e

🤖 Generated by the Agentic Engineer

The authenticated verification now passes on current main ac65b4a7163b50b988fe09835b760f09917b9d8e: workflow run 34045065606 executed both the image-signature inventory and verifier-liveness jobs successfully. The published inventory result is PASS=5 FAIL=0 UNKNOWN=0.

This satisfies the remaining authenticated rerun criterion. The workflow and its regression checks already exist. The dispatch blocker in the issue body is superseded by this successful run; no new credentials or permission grant was required. Refusal testing and activation remain separate work under #3336; this result does not claim they are complete.

Navigation Menu

Repository navigation

Sign or scope every first-party image that a verifier would refuse, before enforcement goes live #3334

Evidence

Impact

Expected outcome

Acceptance criteria

Why this is separate from wiring the verifier

The stated blast radius is gone

AC1's enumeration source needs deciding first — a literal reading of it misses the images that pull

Measured signature state today

What this changes

Inventory measured against the live fleet — 3 of 5 confirmed, 2 unmeasurable, and the premise has moved

The premise of this issue is now stale

The two undetermined ones are a MEASUREMENT gap, not a finding

What this changes for #3101

Reproducing

`blocked` label removed — this issue has no blocker

First blast-radius measurement (AC 1 + AC 2)

The two UNKNOWNs are an authorization gap, not a missing signature

🔴 The stale-example warning in this issue understates the correction

What remains

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Sign or scope every first-party image that a verifier would refuse, before enforcement goes live #3334

Description

Evidence

Impact

Expected outcome

Acceptance criteria

Why this is separate from wiring the verifier

Activity

devantler commented on Aug 24, 2026

The stated blast radius is gone

AC1's enumeration source needs deciding first — a literal reading of it misses the images that pull

Measured signature state today

What this changes

devantler commented on Aug 25, 2026

Inventory measured against the live fleet — 3 of 5 confirmed, 2 unmeasurable, and the premise has moved

The premise of this issue is now stale

The two undetermined ones are a MEASUREMENT gap, not a finding

What this changes for #3101

Reproducing

devantler commented on Aug 25, 2026

blocked label removed — this issue has no blocker

First blast-radius measurement (AC 1 + AC 2)

The two UNKNOWNs are an authorization gap, not a missing signature

🔴 The stale-example warning in this issue understates the correction

What remains

devantler commented on Aug 26, 2026

devantler commented on Sep 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

`blocked` label removed — this issue has no blocker