Skip to content

security(supply-chain): the computed approved revisions are not consumed β€” matchers still accept any fixed revisionΒ #3308

Closed
8 of 9 issues completed

Description

@devantler

πŸ€– Generated by the Agentic Engineer

Evidence

scripts/guard-shared-publish-workflow-pin.sh is explicit that it stops short of the property this
family of work is actually after:

Pinning a fixed revision is not the same as pinning an APPROVED one: a superseded actions SHA
still matches [0-9a-f]{40}. Restricting to approved revisions needs a generated allow-list and
is tracked separately on #2818.

That deferral points at #2818, which has been CLOSED as COMPLETED since 2026-08-10 β€” so today
nothing tracks the enforcing half, and the guard's own note directs the next reader to a closed
issue.

Meanwhile #3048 (in flight) lands the generator: a report that computes, per consumer, which
publish-workflow revisions must stay trusted. Run live against the portfolio it currently reports
5 consumers examined, 2 diverged β€” wedding-app and ascoachingogvaner each have a deployed
artifact signed by one revision while their workflow is pinned to another. For those two, a matcher
that accepts only the current pin would stop verifying an artifact that is running right now.

So after #3048 merges the numbers exist and nothing reads them.

Problem

The cosign matchers still accept any fixed 40-hex revision of the shared publish workflows. A
superseded β€” or simply wrong β€” revision of a devantler-tech/actions publish workflow satisfies
every check we have, which is the substitution the whole #2818 line of work was meant to close.

Expected

The approved-revision set is generated from the report rather than hand-maintained, and the matchers
accept only revisions in that set.

Note the ordering trap the guard already documents: it matches the literal pattern text
[0-9a-f]{40}, because these subjects are regexes. A subject naming one concrete commit β€” exactly
what a generated approved-revision list emits β€” is therefore rejected today, despite being
strictly narrower than what is accepted. The guard's allow-list has to be widened deliberately in
the same change, or the correct generated output reads as a guard failure.

Acceptance criteria

  • The approved-revision set is derived from the security(supply-chain): nothing computes which publish-workflow revisions must stay trustedΒ #3048 report, not written by hand.
  • Every diverged consumer's set contains both revisions the report names, so a currently
    deployed artifact never stops verifying as a side effect of narrowing.
  • guard-shared-publish-workflow-pin.sh accepts a concrete-commit subject, with the widening
    covered by a test that fails if it is reverted.
  • A negative case proves an out-of-set revision is rejected β€” the guard fails by passing, so a
    widened matcher still verifies and nothing else in CI notices.

Size

Small-to-medium. The generator is done; this is consumption plus one deliberate guard widening.

Also worth a look

#2818 reads COMPLETED while this half is deferred and its child #3048 is still open. Flagging rather
than reopening it β€” the closure was a deliberate call, and this issue restores the missing tracking
either way.

Implementation delivered; production verification pending (verified 2026-09-06): the scheduled regeneration prerequisite is satisfied. Scheduled run 33950394952 completed successfully on 2026-09-05 at 06:39:43Z, and #3611 merged on 2026-09-05 at 11:38:03Z with the narrowed matchers and enforcing guards. The remaining work is tracked in #3569: observe every consumer's OCIRepository artifact revision advance on its next publish after the merge. Keep this issue open until that production evidence is recorded; the scheduled-run wait and narrowing implementation are no longer blockers.

Activity

  1. moved this to πŸ“₯ Backlog in 🌊 Project Boardon Aug 22, 2026
  2. self-assigned this
    on Sep 2, 2026
  3. added a commit that references this issue on Sep 2, 2026
  4. devantler commented on Sep 2, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    The enabling half is done; three findings change the shape of the rest

    The guard widening this issue names as a prerequisite is split out as #3531 and delivered in #3532 β€” a concrete commit and an alternation of two concrete commits are now accepted, proven by a GREEN case that fails against the previous guard and RED cases that fail if the widening is done without a whole-line anchor. No matcher changed. Three things found while building it argue against narrowing the live subjects in the same pass.

    1. The report has moved since filing: 4 consumers, and now all 4 diverged

    Run against merged main (060686b7) today, report-publish-workflow-signing-revisions.sh reports 4 consumer(s) examined, 4 diverged, 0 unresolved β€” not the "5 examined, 2 diverged" recorded above on 2026-08-22. Every consumer pins the same revision today (3ee6cb9b) while each was signed by a different one:

    consumer signed by pins today
    .github (github-config) b3783a51 3ee6cb9b
    ascoachingogvaner 625b7c0c 3ee6cb9b
    aws b089a1b0 3ee6cb9b
    wedding-app 6ae5d87b 3ee6cb9b

    So AC2 now applies to every consumer rather than two: each needs a two-revision set, and a set generated from current pins alone would stop verifying all four deployed artifacts rather than two.

    2. Only 4 of the 7 matched subjects are per-consumer β€” the other 3 have no consumer to derive a set from

    The guard matches 7 subjects. The four OCIRepositories above are per-consumer. The remaining three are generic:

    • talos/cluster/verify-first-party-images.yaml β€” first-party images generally
    • k8s/bases/infrastructure/resource-graph-definitions/tenant/resource-graph-definition.yaml β€” the tenant template, instantiated for tenants that do not exist yet
    • k8s/bases/infrastructure/cluster-policies/best-practices/verify-app-images.yaml β€” a Kyverno policy over app images generally

    This issue's Expected ("the matchers accept only revisions in that set") is under-specified for these. The report is keyed by consumer and cannot produce a set for a matcher that covers an open population. Narrowing the tenant template would bake today's revisions into every future tenant; narrowing the Kyverno policy needs the union across all apps, which grows on every publish and is a different (and much more brittle) artifact than a per-consumer set. Worth deciding explicitly rather than discovering during implementation β€” my read is these three should stay on the pattern form, with that stated as a deliberate scope boundary.

    3. Evidence quality is not uniform across the four, and the report says so

    Two consumers carry deployed version inferred from newest PUBLISHED tag; not applied-revision evidence (.github, aws); the other two carry is what the manifest PINS and was published. For the inferred pair the signing revision is a reasonable inference, not an observation. Narrowing on it has a real failure mode β€” an OCIRepository that quietly stops reconciling while CI stays green, which is the exact shape #3048's header warns about.

    That is why #3532 changes no matchers. Prod access here is read-only, so the applied revision cannot be confirmed directly this run; closing that gap (or accepting the inference deliberately) belongs with the narrowing.

    One AC has no home yet

    AC4 β€” "a negative case proves an out-of-set revision is rejected" β€” is a property of the set-checking mechanism, not of the guard: the guard validates ref shape and cannot know which revisions are approved. #3532 covers the guard-level negatives (near-miss SHAs, a set widened back with a branch). The set-level negative needs whatever ends up deriving and enforcing the committed sets.

    Leaving this issue open for the narrowing half.

  5. devantler commented on Sep 2, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Addendum to finding 3 β€” the inferred-evidence gap is being narrowed independently.

    codex/registry-tag-check-3331 (#3533, opened 17:41Z against #3331) makes the report require each historically published candidate to still exist in its current GHCR package before attributing it as deployed, and to refuse by name when a version was deleted rather than silently walking backward.

    That does not turn an inference into an observation β€” it still infers the deployed version from the newest published tag β€” but it removes one way the inference can be confidently wrong. Worth landing before the narrowing is derived, so the two consumers marked not applied-revision evidence (.github, aws) are attributed against tags that still exist. No file overlap with #3532, which touches only the guard and its test.

  6. devantler commented on Sep 3, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Finding 3 is closed by observation, and the narrowing needs a paved road before it is safe

    The applied revision is observable after all β€” read prod's OCIRepositories, not the registry's tag list. Each consumer's OCIRepository on prod carries status.artifact.revision = <tag>@sha256:<digest>, which is exactly the artifact Flux has verified and applied. Resolving the signing identity of that digest (read-only, 2026-09-03 ~15:30Z):

    consumer applied signed by (observed) how observed
    aws 1.1.3@sha256:6ab28956… publish-manifests.yaml@b089a1b041cb86af22cdc57de58a4d7d258dcc32 signing certificate SAN, decoded from the .sig bundle
    .github (github-config) 1.23.1@sha256:73ef6a3c… publish-manifests.yaml@b3783a5148b844b3f1f4011cdc84871ad66348c1 signing certificate SAN, decoded from the .sig bundle
    wedding-app 1.15.10@sha256:9d1feaca… publish-app.yaml@6ae5d87b342984cdaf34740e60366b9d570d8fc5 the πŸš€ CD run that published exactly tag v1.15.10, referenced_workflows[].sha
    ascoachingogvaner 1.13.4@sha256:ec90c928… publish-app.yaml@625b7c0cd5ad4a04f9eb7494298c6d0fa44521ef the πŸš€ CD run that published exactly tag v1.13.4, referenced_workflows[].sha

    All four agree with the report's inferred values, so the inference is corroborated by observation rather than replaced by it. The two publish-app consumers publish to private packages, which this run's credential cannot read from the registry; for those the run's referenced_workflows is the same fact the OIDC job_workflow_ref claim is minted from, keyed on the exact applied tag rather than the newest published one.

    The design hazard the narrowing has to solve first. Every consumer pins the shared workflow from its own cd.yaml, and those pins move on every devantler-tech/actions bump β€” by dependency automation, several times a month. A matcher narrowed to (<applied-signer>|<current-pin>) is therefore stale the moment a consumer merges a bump: the next artifact it publishes is signed by a revision the platform no longer accepts, Flux refuses it, and the OCIRepository stops reconciling β€” the exact failure mode the report's header warns about, now caused by the fix. Hand-regenerating the set after each bump is a two-repository ordered rollout on every dependency update, which is the DevEx tax the hardening rule forbids. So the narrowing is only safe with automated regeneration: something that recomputes the four sets when either input moves and opens the platform PR itself. Degradation in the regeneration window is bounded (Flux retries; the next artifact verifies once the PR merges) and acceptable; an unbounded manual gap is not.

    Decision.

    1. The three generic subjects (talos/cluster/verify-first-party-images.yaml, the tenant template, the Kyverno app-image policy) stay on the pattern form, stated as a deliberate scope boundary β€” they cover open populations and have no consumer to derive a set from.
    2. The four per-consumer OCIRepository matchers are narrowed to @(<applied-signer-sha>|<consumer-main-pin-sha>) β€” never to a single revision β€” so a deployed artifact never stops verifying as a side effect (AC2).
    3. The set is generated, committed as a data file, and a CI guard asserts the four matchers equal the generated file; the same guard carries the set-level negative case AC4 asks for (an out-of-set revision is rejected by the generated matcher).
    4. Regeneration is a scheduled platform workflow that recomputes the sets from the report plus the applied-revision observation above and opens a draft PR when they move; the narrowing is not flipped on until that automation has run green at least once (feature-flag-first: generated file lands first, matchers switch second).

    Filed as sub-issues below so each ships as its own increment; this issue stays the parent until the matchers are switched.

  7. 2 remaining items

  8. devantler commented on Sep 4, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Status, so the next run does not re-derive it: this is 5 of 6 delivered, and the remainder is blocked β€” not untouched work.

    Selected this as the oldest actionable Security issue, then found on inspection that most of it has shipped since it was filed on 2026-08-22:

    Child State What landed
    #3531 CLOSED the pin guard accepts concrete revisions (parent AC3)
    #3550 CLOSED generate-publish-workflow-approved-revisions.sh writes the per-consumer pair (AC1)
    #3551 CLOSED guard-publish-workflow-approved-revisions.sh holds each matcher to the set; refuses an out-of-set revision in every mode (AC4)
    #3552 CLOSED daily regeneration workflow (#3568)
    #3558 CLOSED cosign verification required on every devantler-tech OCIRepository
    #3569 OPEN, now labelled blocked the narrowing switch itself

    The parent's headline claim β€” "the matchers still accept any fixed revision" β€” is still true today, and deliberately so: guard-publish-workflow-approved-revisions.sh is feature-flag-first, and with APPROVED_REVISIONS_ENFORCE unset the pattern form is still accepted. All four consumers now diverge between applied_signer_sha and main_pin_sha in the generated set, which is exactly why the narrowing has to emit the pair rather than the current pin.

    So the whole remainder is #3569, and #3569 is blocked on one green scheduled run of the regeneration workflow, which merged only this morning (10:19:54Z) and has never run β€” total_count=0. I repaired that blocker record there, in the conforming shape with a one-call re-verification, because it had been recorded as prose with no blocked label.

    Claim on this issue retired: the work is not startable here, and the actionable unit is the child.

  9. added 2 commits that reference this issue on Sep 6, 2026
  10. added a commit that references this issue on Sep 6, 2026
  11. devantler commented on Sep 13, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Scope change, recorded so the next run reads this issue correctly: its problem statement is true again for two of its four consumers, on purpose.

    On 2026-09-12, #3726 (from the model decision recorded on #3699) made ascoachingogvaner and wedding-app trusted application release streams. Their matchers accept any immutable 40-hex commit of the platform-owned publish-app.yaml workflow again, and the guard now requires that form for them. Both matchers applied on prod carry it today. .github and aws remain bound to their exact generated pairs.

    What this means for this issue:

  12. devantler commented on Sep 20, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Delivered β€” verified against main at cf2f6248

    Every acceptance criterion is met, and the enforcing half this issue was opened about is live in CI.

    The set is generated, not hand-written. scripts/publish-workflow-approved-revisions.tsv carries
    four consumers with their applied_signer_sha and main_pin_sha;
    generate-publish-workflow-approved-revisions.sh produces it and
    .github/workflows/regenerate-publish-workflow-approved-revisions.yaml keeps it current β€” ten
    regeneration PRs have merged since.

    The matchers consume it. write-publish-workflow-matchers.sh derives each subject from that set,
    and both configuration consumers now name concrete revisions rather than the wide shape:

    consumer subject revision
    .github @(a147ead8…|3c3a6348…)
    aws @(b089a1b0…|3c3a6348…)

    Each carries both the revision that signed the deployed artifact and the current pin, so narrowing
    did not stop a running artifact verifying.

    The shape guard was widened in the same line of work. guard-shared-publish-workflow-pin.sh now
    recognises two forms β€” the literal pattern text and a concrete 40-hex commit β€” with the reasoning and
    the #3308 reference in place at the recognition site, and a test behind it.

    An out-of-set revision is rejected, and the rejection runs. CI's πŸ” Enforce approved publication signers step runs guard-publish-workflow-approved-revisions.sh with APPROVED_REVISIONS_ENFORCE=1
    against the repository, alongside test-publish-workflow-approved-revisions-guard.sh and
    test-write-publish-workflow-matchers.sh.

    One scope change, made deliberately and worth stating plainly

    This issue named wedding-app and ascoachingogvaner as the diverged consumers whose sets should carry
    both revisions. They were instead excluded from narrowing as trusted release streams, and they still
    carry @[0-9a-f]{40}. The rationale is recorded at TRUSTED_RELEASE_STREAM_CONSUMERS in
    publish-workflow-approved-revisions.lib.sh: an application tenant releases on its own cadence, so
    pinning it to the currently observed workflow revisions would make an ordinary tenant release wait for a
    platform change to land first.

    That is a real availability argument and it is not the same question this issue was asking, so closing
    here rather than leaving it open indefinitely. The residual risk β€” a superseded publish-app.yaml
    revision still satisfies both app tenants' matchers β€” is carried forward on its own issue with the
    trade-off stated, rather than left implicit in a closed one.

  13. devantler commented on Sep 20, 2026

    @devantler
    ContributorAuthor

    πŸ€– Generated by the Agentic Engineer

    Closing as delivered. The residual β€” both app tenants still accepting any workflow revision β€” is carried forward with its trade-off on #3917.

  14. moved this from πŸ“₯ Backlog to βœ… Done in 🌊 Project Boardon Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Security.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions