security(render): prod render fetches third-party manifests unpinned, one from a moving branch

🤖 Generated by the Agentic Engineer

Historical evidence — 2026-08-17

Measured 2026-08-17 while running the required prod authorization validator locally.

Two entries in the prod controllers render fetch third-party manifests over the network at render time:

k8s/providers/hetzner/infrastructure/controllers/origin-ca-issuer/kustomization.yaml:8-9 — two CRDs from Cloudflare's trunk branch (a moving branch, not a pin)
k8s/providers/hetzner/infrastructure/controllers/kubelet-serving-cert-approver/kustomization.yaml:21 — an install bundle pinned to tag v0.11.0 (a tag, no digest)

Both are active in the prod render (controllers/kustomization.yaml:43 and :45).

Observed at that measurement: raw.githubusercontent.com returned HTTP 429 on repeated attempts, and kubectl kustomize k8s/providers/hetzner/infrastructure/controllers aborted with accumulating resources ... 429: Too Many Requests. Four render-dependent tests in scripts/validate-eks-ci-role-policy failed identically on clean main and on a feature branch, which is what establishes the dependency as the cause rather than any change.

Problem

The render is not reproducible. trunk can move between two renders of the same commit, so one commit can produce two different expectedRenderedSurfaceSHA values with nothing in this repository having changed. That is directly relevant to the aggregate-fingerprint churn tracked in The EKS authorization surface hash goes stale on every HelmRelease dependency bump #2768.
Unreviewed third-party content reaches the prod authorization surface. The cert-approver install bundle contributes ServiceAccount / ClusterRole / (Cluster)RoleBinding documents to the very surface validate-eks-ci-role-policy exists to pin. A trunk move lands with no PR, no review, and no digest.
A required security gate is hostage to a third-party CDN. While the fetch rate-limits, the authorization measurement cannot be produced at all — which is how this was found.

Expected

Prod renders resolve entirely from reviewed, digest-pinned content in this repository.

Precedent — the pattern already exists here

scripts/update-vendored-operators.sh vendors pinned upstream bundles with SHA-256 verification and reviewed Checkov dispositions (CDI, KubeVirt), and CI re-validates the committed bytes against the pinned digest via --validate-committed. These two remotes are the outliers, not a new problem needing a new mechanism.

Acceptance criteria

No kustomization.yaml reachable from a prod render path references a remote URL.
Both dependencies are vendored under version control, pinned by digest, and refreshed through a reviewed updater path, with CI validating the committed bytes against that pin.
The five authorization roots and k8s/clusters/prod render with no network access.
A regression check fails if a new remote URL is introduced into a render path.

Suggested decomposition

A guard that fails when a render path references a remote URL — small, immediately valuable, and prevents a third outlier.
Vendor the origin-ca-issuer CRDs — most urgent, because trunk is unpinned today.
Vendor the kubelet-serving-cert-approver bundle — pinned to a tag, so lower risk than (2), but it is the one contributing authorization documents.

Rough size: M overall; child (1) is S.

Decomposed 2026-08-29 into three sequenced sub-issues (real sub-issue links):

security(render): guard against a render path acquiring a third network fetch #3442 — a guard that fails when a render path acquires a remote URL — DONE and merged
(scripts/guard-render-remote-resources.sh, wired into ci.yaml, with
scripts/render-remote-resource-exceptions.tsv recording the three known remotes as tracked,
not approved, and a test at scripts/tests/test-guard-render-remote-resources.sh).
security(render): vendor the origin-ca-issuer CRDs instead of fetching them from the trunk branch #3445 — vendor the origin-ca-issuer CRDs (the moving trunk branch) — DONE and merged in fix(render): vendor immutable controller artifacts #3635
security(render): vendor the kubelet-serving-cert-approver install bundle instead of fetching it at render time #3446 — vendor the kubelet-serving-cert-approver bundle — DONE and merged in fix(render): vendor immutable controller artifacts #3635

All three children are complete. The current interactive request superseded the former upstream-read authority restriction, and #3635 completed the remaining implementation and delivery.

Completed 2026-09-06: #3635 merged at 45841c15cd44d1a4eda8497bbea0995faf9951b7 after current-head review and successful merge-group CI and deployment. The remaining vendoring work is delivered; no maintainer action is pending.

⚠️ Possible Duplicate Issue(s)

Rendering manifests depends on a third-party host, so a network transient fails a required check #3354

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

The failure reaches CI too, and the mechanism is sharper than the filing states

The required pipeline on #3181 (head 07ec2b5f) failed at 🔍 Detect Changes → 💾 Validate two-stage PVC retirement, with 🔐 Validate EKS Authorization and every other k8s job skipped as a result. So this is not only a local-render inconvenience: while the fetch is failing, the required authorization gate does not run at all.

The mechanism. kustomize does not fail on a plain HTTP error. It reclassifies the unreachable raw URL as a git repository and then fetches that:

accumulating resources from 'https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/trunk/.../clusteroriginissuers.yaml': URL is a git repository
failed to run '/usr/bin/git fetch --depth=1 https://raw.githubusercontent.com/cloudflare/origin-ca-issuer HEAD'

raw.githubusercontent.com/cloudflare/origin-ca-issuer is not a git repository, so that second fetch fails on its own terms. That is why the same root cause surfaces with two different terminal codes — 429 on my local render (kubelet-serving-cert-approver, which is accumulated first) and 404 in CI (origin-ca-issuer). The 404 is the misclassified git fetch, not evidence that Cloudflare deleted trunk.

Explicitly not claimed: that trunk is gone. I could not test it — curl returned 429 for both trunk and main from here, so the rate limit blocks the very check that would distinguish them. Whether trunk still exists is unresolved and does not need resolving, because vendoring removes the question.

Not a repo defect on its own. The immediate trigger is a GitHub-side CDN rate limit (main was green at 13:18Z, before the incident), and re-running once it clears is the sanctioned response. What is ours is the exposure this reveals: a required security gate and the prod render both fail closed on a third-party CDN, and one of the two dependencies is not pinned at all. The reproducibility argument in the filing stands independently of any incident.

🤖 Generated by the Agentic Engineer

New evidence — this now blocks merge-critical work, not just local renders

Filed yesterday on reproducibility grounds. Today it took a required check out on an open PR, so the
impact is larger than originally recorded.

platform#3181's CI run 32041832988 failed at 🔍 Detect Changes → 💾 Validate two-stage PVC retirement, and the cause is this dependency, not the PR:

accumulating resources from 'https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/trunk/deploy/crds/cert-manager.k8s.cloudflare.com_clusteroriginissuers.yaml': URL is a git repository':
failed to run '/usr/bin/git fetch --depth=1 https://raw.githubusercontent.com/cloudflare/origin-ca-issuer HEAD': remote: 404: Not Found

The PR touches no manifests, so its diff cannot reach that step — the render pulls the CRDs live on
every validation, and an unavailable CDN fails the job regardless of the change under test. Note the
404 is still the misclassification described in the issue body (kustomize reclassifies the
unreachable raw URL as a git repo and reports its git fetch failure), not evidence that trunk
was deleted.

The "is `trunk` gone?" question is still open — and my probe cannot answer it

Direct probes today returned 429 for trunk, for main, and for a known-good unrelated file on
another repository. Since the control failed identically, the 429 is host-wide rate limiting on this
network rather than anything about this dependency, so it distinguishes nothing. Recording that as
unresolved rather than guessing — it does not change the fix either way.

Why this raises the priority

A third-party outage should degrade one optional job, not fail a required check on an unrelated PR.
Two independent failure modes are now observed:

Availability — an unreachable CDN fails required validation for changes that touch no manifests.
Reproducibility — trunk is a moving branch, so one commit can render two different
authorization surfaces and therefore two different expectedRenderedSurfaceSHA values. That is a
plausible contributor to the fingerprint churn on fix(dex): restrict GitHub auth to maintainers #2709/fix(apps): pin tenant OCI manifest tags #2737/fix(longhorn): reclaim orphaned replica directories automatically #3181.

Suggested first child, unchanged in shape but firmer in motivation

A guard that fails on any remote URL in a render path. It should ship with the two known remotes
recorded as explicit, dated exceptions rather than failing main on day one — the fix-vs-except
ladder — so the guard lands immediately and each exception is then retired by its own vendoring
change (scripts/update-vendored-operators.sh already establishes the vendored + SHA-256-pinned +
CI-revalidated pattern this should follow).

🤖 Generated by the Agentic Engineer

New evidence, 2026-08-18: this now evicts PRs from the merge queue

The same kubelet-serving-cert-approver fetch named above took a ready PR out of the merge queue
today — a failure mode a step beyond the local-test failures recorded when this was filed.

#3225 was enqueued at 21:04:21Z and evicted when the merge_group CI run failed in
🔍 Detect Changes → 💾 Validate two-stage PVC retirement:

error: accumulating resources from
'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml':
URL is a git repository
fatal: repository 'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/' not found

Note the shape: kustomize's HTTP fetch failed, so it fell back to treating the URL as a git
remote, and the reported error is a git fetch 404 rather than the underlying HTTP failure. The
message therefore names neither the real cause nor this issue — it reads like a broken repository
reference, which is a misdirection cost on top of the outage.

That it was the dependency and not the change is established the same way #3196 established it
originally: the identical step succeeded on the same commit's own pull_request run
(15826dfef0, run 32181427032, 💾 Validate two-stage PVC retirement -> success) minutes earlier,
and main was green throughout. A clean re-queue then went through.

Why this raises the priority rather than just adding a datapoint. A local test failure is loud
and stops the person in front of it. A merge-queue eviction is quiet: the PR silently leaves the
queue and stays OPEN and CLEAN, so it reads exactly like one still draining. Any run that
re-queues without pulling the merge_group logs will keep paying for it without ever learning why —
and an unattended lane that does not check will simply leave finished work parked.

🤖 Generated by the Agentic Engineer

A measured instance of the availability half of this, today

This issue argues the integrity case (unpinned/moving-branch fetches). The same render-time fetch is
also an availability dependency, and it failed for real today.

🔐 Validate EKS Authorization failed on an unrelated PR (#3244, a shell-test change that touches no
manifest) at 2026-08-19T11:15:20Z:

render k8s/providers/hetzner/infrastructure/controllers:
  accumulating resources from 'kubelet-serving-cert-approver/':
  fatal: repository 'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/' not found
  : exit status 128

The reference is k8s/providers/hetzner/infrastructure/controllers/kubelet-serving-cert-approver/kustomization.yaml
→ https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml.

It was transient, and that is the point. Re-checked at ~11:40Z: the pinned URL returns 200 and the
upstream repository resolves 200. Nothing changed upstream — the fetch simply failed once.

Two things this pins down:

The blast radius is wider than CI. Flux renders this same layer, so a sustained outage of the
remote host does not merely fail a PR check, it stalls reconciliation of the prod controllers layer.
The failure is indistinguishable from a real removal at the point of failure. The error says
"not found", which reads as "upstream deleted it" and invites exactly the wrong repair. Confirming it
was transient took a separate live fetch.

Vendoring or mirroring these manifests — the fix this issue already proposes — removes the availability
exposure at the same time as the integrity one, so the two cases fold into one change rather than
competing for it.

(No action taken on #3244 beyond re-running its own checks; that PR does not touch this path.)

chore: agent-claim #3196 nonce=1eab8cb976dfa397befead2609df95f6

@devantler

🤖 Generated by the Agentic Engineer

Status check — child (1) is DONE and wired; (2) and (3) are the remainder

Verified at main today rather than assumed:

scripts/guard-render-remote-resources.sh exists (14.3K, executable).
scripts/tests/test-guard-render-remote-resources.sh pins its verdict in all three
directions, including the schemeless-remote, host-named-local-directory, stale-exception-row,
glob-injection and both anti-vacuity cases.
Both are wired into 🧪 Validate Manifests in ci.yaml (guard ... k8s + the test), with
matching changes path filters.
scripts/render-remote-resource-exceptions.tsv carries exactly the three live rows, each
pointing back here.

So the "prevents a third outlier" outcome is already delivered. I also checked for render paths
the guard's k8s root would miss: the only kustomization.yaml outside k8s/ is
tests/cilium-bandwidth-manager-bbr/kustomization.yaml, and it references no remote. No coverage
gap.

New evidence — the cert-approver remote broke CI today

The reproducibility argument in this issue is no longer only about trunk moving. On
2026-08-30T09:47:21Z, an unrelated PR's 🧪 Validate Manifests job failed like this:

accumulating resources from 'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml':
  read tcp ...:52162->185.199.110.133:443: read: connection reset by peer

and the render then aborted, failing a test that has nothing to do with cert approval
("FAIL: the opt-in fixture render has no Cilium HelmRelease"). A plain re-run cleared it.

That is the third failure mode for the same root cause, and the most expensive one in practice:
it is indistinguishable from a real regression at a glance, it lands on whichever PR happens
to be in CI when the CDN blips, and it costs a diagnosis plus a re-run every time. The earlier
observations were HTTP 429 (2026-08-17) and the unreviewed-content/trunk-drift risk. Worth
noting the tag-pinned entry — child (3), rated "lower risk" above — is the one that actually
caused this, because availability does not care about pinning.

Constraint on children (2) and (3): an unattended run cannot fetch the upstream bytes

Both remaining children mean vendoring content from repositories outside devantler-tech
(cloudflare/origin-ca-issuer, alex1989hu/kubelet-serving-cert-approver). The engineering
contract's professional-work repository boundary requires the maintainer's current, explicit
confirmation that a named non-portfolio repository is unrelated to professional duties before
even a read-only action — and a scheduled run has no channel to obtain it. This is missing
authority, not an external dependency, so I am deliberately not adding a blocked label:
that would route this issue into skip clause (b) and hide it from every future tick, which is
worse than leaving it visible.

Recording it so each tick does not re-derive it. Unblocking is one sentence from @devantler
confirming those two upstream projects are unrelated to his professional work — after which
children (2) and (3) are straightforward, and scripts/update-vendored-operators.sh is the
existing mechanism to extend (checksum + Checkov disposition + --validate-committed).

I acquired the cross-lane claim to do this work and have retired it, so the issue is free for
whoever can act on it.

🤖 Generated by the Agentic Engineer

New evidence — this now also evicts PRs from the merge queue

Adding a second, distinct failure mode observed today. The original evidence was an HTTP 429
breaking a local render; this one is a different network error, in a different context, with a
materially worse cost.

PR #3530 was evicted from the merge queue when its merge_group run
33671787384 failed in
🔐 Validate Production Authorization:

✗ kustomization validation failed: ... accumulating resources from
'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml':
read tcp 10.1.1.166:41196->185.199.109.133:443: read: connection reset by peer
✔ 116 kustomizations validated  ✗ 1 failed

The control is unusually clean: the same job passed on the same commit c119981c at PR level
51 minutes earlier (18:20:12Z, 🔐 Validate Production Authorization SUCCESS). Nothing in the change
differed between the two runs — only the network. 115 of 116 kustomizations resolved offline; one
remote read decided the outcome.

Why this raises the priority rather than just adding a data point

The 429 case blocked a measurement. This case evicts a PR from the merge queue, which costs a
full re-queue and re-run — and on this repository the merge queue runs a real production deploy, so
queue cycles are not cheap. It also lands as an apparently unrelated failure: the follow-on error is
remote: 404: Not Found / repository ... not found, because kustomize falls back to a git fetch
after the HTTP read fails. That reads like a deleted upstream rather than a blip, and sends whoever
triages it in the wrong direction.

So the exposure is no longer only "renders are not reproducible" — it is also "any PR can be
randomly evicted from the queue by a third-party CDN".

Note on the blocker

I have not re-notified the maintainer, per the standing note on #3445. Recording this here as
evidence only. Both children remain blocked on the same out-of-org artifact-fetch clearance, and the
v0.11.0 remote above is the pinned-tag one — so even the tag-pinned entry, not just Cloudflare's
moving trunk, is a live availability dependency.

I filed #3537 for this before finding this issue; it is a duplicate and I have closed it, pointing
here.

chore(deps): move the tracked render exception to v0.12.0 with the URL

Navigation Menu

Repository navigation

security(render): prod render fetches third-party manifests unpinned, one from a moving branch #3196

Historical evidence — 2026-08-17

Problem

Expected

Precedent — the pattern already exists here

Acceptance criteria

Suggested decomposition

The failure reaches CI too, and the mechanism is sharper than the filing states

New evidence — this now blocks merge-critical work, not just local renders

The "is `trunk` gone?" question is still open — and my probe cannot answer it

Why this raises the priority

Suggested first child, unchanged in shape but firmer in motivation

New evidence, 2026-08-18: this now evicts PRs from the merge queue

A measured instance of the availability half of this, today

2 remaining items

Status check — child (1) is DONE and wired; (2) and (3) are the remainder

New evidence — the cert-approver remote broke CI today

Constraint on children (2) and (3): an unattended run cannot fetch the upstream bytes

New evidence — this now also evicts PRs from the merge queue

Why this raises the priority rather than just adding a data point

Note on the blocker

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

security(render): prod render fetches third-party manifests unpinned, one from a moving branch #3196

Description

Historical evidence — 2026-08-17

Problem

Expected

Precedent — the pattern already exists here

Acceptance criteria

Suggested decomposition

Activity

coderabbitai commented on Aug 17, 2026

devantler commented on Aug 17, 2026

The failure reaches CI too, and the mechanism is sharper than the filing states

devantler commented on Aug 17, 2026

New evidence — this now blocks merge-critical work, not just local renders

The "is trunk gone?" question is still open — and my probe cannot answer it

Why this raises the priority

Suggested first child, unchanged in shape but firmer in motivation

devantler commented on Aug 18, 2026

New evidence, 2026-08-18: this now evicts PRs from the merge queue

devantler commented on Aug 19, 2026

A measured instance of the availability half of this, today

2 remaining items

devantler commented on Aug 30, 2026

Status check — child (1) is DONE and wired; (2) and (3) are the remainder

New evidence — the cert-approver remote broke CI today

Constraint on children (2) and (3): an unattended run cannot fetch the upstream bytes

devantler commented on Sep 2, 2026

New evidence — this now also evicts PRs from the merge queue

Why this raises the priority rather than just adding a data point

Note on the blocker

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

The "is `trunk` gone?" question is still open — and my probe cannot answer it