Skip to content

security: tighten cosign subject matchers at the image-verification layer #2815

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

#2402 tightens the workflow-ref part of the cosign subject matchers on the Flux OCIRepository trust rules (artifact layer) from @.+ to a 40-hex SHA or a release tag, and drops the now-dead (reusable-workflows|actions) alternation. Doing that surfaced the same two weaknesses, unfixed, at the image layer:

File Matcher Weakness
talos/cluster/verify-first-party-images.yaml (ghcr.io/devantler-tech/* rule) .../(actions|reusable-workflows)/.../publish-app\.yaml@.+$ floating ref and dead alternation
talos/cluster/verify-first-party-images.yaml (provider-upjet-* rule) .../publish-provider-package\.yml@.+$ floating ref
k8s/bases/infrastructure/cluster-policies/best-practices/verify-app-images.yaml .../(actions|reu…)/…@.+$ floating ref and dead alternation

reusable-workflows was archived on 2026-07-10, so the alternation can no longer match anything new; @.+ accepts an image signed from any ref of the workflow repo, including @main.

Note verify-first-party-images.yaml's ksail rule already uses the tight form (cd\.yaml@refs/tags/v.+$), so the tight shape is established in this very file — these rules are simply the ones that never got narrowed.

Problem / audience

Cluster operators. This is the node-pull and admission layer, so it is the control that decides whether a workload image may run at all. It is currently one step looser than the artifact layer it is supposed to backstop.

Why this is NOT part of #2402

Different layer, materially different blast radius. A non-matching regex here does not merely stop a source reconciling — it causes ImagePullBackOff on every affected workload (the in-file comment records exactly that happening during the actions#425 repoint). Narrowing safely therefore requires per-image evidence that every currently running first-party image is actions-signed from a pinned ref, which is a distinct verification job from the four artifacts #2402 covered.

Smallest useful change

  1. For each first-party image currently running in the cluster, read the signing certificate's SAN and record the repo and ref form (the same GHCR-manifest → sigstore-bundle → openssl x509 route used to evidence Tighten cosign OIDC subject ref matchers (@.+) on OCIRepository verify rules #2402; note this needs a token with read:packages for private packages, which the current host PAT does not have).
  2. Only for image families where every running image is actions-signed from a pinned ref, narrow that rule to @([0-9a-f]{40}|refs/tags/v.+)$ and drop the alternation.
  3. Leave any family with a surviving pre-repoint image untouched, and record why.
  4. Roll out verify-first-party-images.yaml (Talos node layer) and verify-app-images.yaml (Kyverno admission layer) in that order, verifying pulls still succeed between them.

Acceptance criteria

  • Every first-party image rule that can be narrowed rejects a floating ref (@main, @refs/heads/*) and accepts the ref forms actually in use.
  • The (actions|reusable-workflows) alternation is gone wherever no running image still needs it; where it survives, the reason is recorded in the file.
  • Evidence is per-image-family and read from real signatures, not inferred.
  • No workload enters ImagePullBackOff during or after rollout (checked live).

Notes

The provider-upjet-* rule is signed by each provider repo's own publish-provider-package.yml, so its pinned-ref convention must be confirmed separately from the shared publish-app path.

Part of #2402's follow-up. Rough size: M.

Activity

  1. coderabbitai commented on Jul 26, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #2382 - fix(policies): accept the provider publish-workflow cosign identity in verify-app-images [merged]
    #2399 - feat: accept devantler-tech/actions cosign subjects during the publish-workflow repoint [merged]
    #2404 - fix: tenant artifact verification rejects the dual signing identities from #2399 [merged]
    #2457 - fix(talos): accept the actions publish-app identity at node image verification [merged]
    #2633 - fix(deploy): verify incoming image on Talos nodes [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. moved this to 📥 Backlog in 🌊 Project Boardon Jul 26, 2026
  3. self-assigned this
    on Jul 29, 2026
  4. devantler commented on Jul 29, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Evidence gathered, and a blocker found — releasing this claim

    I picked this up, ran the evidence step, and am handing it back unstarted because step 1 cannot be completed with the access this host has, and because a finding surfaced that changes the premise.

    Which first-party images are actually running (five families):

    Family Rule it matches Current matcher
    ksail ksail rule already tight (cd.yaml@refs/tags/v.+)
    provider-upjet-unifi provider rule floating @.+
    wedding-app, ascoachingogvaner, doggy-countdown catch-all floating @.+ and the dead (actions|reusable-workflows) alternation

    Step 1 is blocked for two of the three app families. wedding-app and ascoachingogvaner are private packages: GHCR refuses an anonymous pull token, and this host's PAT has no read:packages, exactly as this issue predicted. Their signing certificates therefore cannot be read, so the per-image evidence acceptance criterion cannot be met for them. This is a credential-scope blocker, not a design question — the narrowing itself is straightforward once the evidence exists.

    And the premise needs re-checking first. The one app package that is public, doggy-countdown, turns out to carry no cosign signature at all while running normally — filed as #2856, now linked as blocking this issue. That matters here specifically: this issue's plan assumes every running first-party image satisfies the current rule and that the only question is how much to narrow it. For at least one family that assumption does not hold, so narrowing would tighten a rule whose enforcement is itself unconfirmed.

    Suggested sequencing, unchanged in substance from the plan above but with one step in front of it:

    1. First-party image pull verification does not block an unsigned first-party image #2856 — establish whether the rules enforce at all, and why an unsigned image pulls.
    2. Obtain read:packages (or run the evidence step somewhere that has it) for the private families.
    3. Then narrow, in the order this issue already sets out.

    Unassigning myself and deleting the claim branch so this is not parked behind a lease I am not working. The provider-upjet-unifi rule is the one family whose narrowing might be separable, since its signing workflow is per-repo — but it is a Crossplane package rather than an app, and its pinned-ref convention still needs confirming, which the same access limit blocks.

  5. removed their assignment
    on Jul 29, 2026
  6. devantler commented on Aug 4, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Re-verified this issue's blocker against live state rather than inheriting it, and it still stands.

    Criterion 3 (per-image evidence read from real signatures) is blocked on a token permission. The
    host credential cannot reach the packages API at all:

    GET /orgs/devantler-tech/packages?package_type=container
    403 Resource not accessible by personal access token
    

    So the GHCR-manifest → sigstore-bundle → openssl x509 route this issue specifies cannot be run from
    an unattended tick. Narrowing the matchers without that evidence is exactly what the issue forbids —
    and the in-file comment records that guessing here already caused ImagePullBackOff once.

    Criterion 4 (no workload enters ImagePullBackOff, checked live) additionally needs a watched
    window.
    A scheduled run is not one, for the same reason the probe activation on #2651 is held back:
    the failure mode is cluster-wide and needs someone watching it.

    Unblocking is a maintainer act — granting the host credential read:packages (or an equivalent
    fine-grained packages permission). I have not self-widened anything.

    Leaving this unassigned and actionable-once-unblocked rather than parking it silently.

  7. self-assigned this
    on Aug 5, 2026
  8. moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Aug 5, 2026
  9. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Security.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions