Skip to content

security(render): vendor the kubelet-serving-cert-approver install bundle instead of fetching it at render time #3446

Description

@devantler

🤖 Generated by the Agentic Engineer

Part of #3196

Evidence

k8s/providers/hetzner/infrastructure/controllers/kubelet-serving-cert-approver/kustomization.yaml fetches its install bundle at render time:

- https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml

This entry is active in the prod render (controllers/kustomization.yaml:45). Unlike the trunk fetch in #3445 it is pinned to a release tag, so it is the milder of the two — but a tag is mutable and carries no digest, and the fetch still runs on every render.

Problem

  1. Unreviewed third-party content reaches the prod authorization surface. This bundle contributes ServiceAccount / ClusterRole / (Cluster)RoleBinding documents to the very surface validate-eks-ci-role-policy exists to pin — so a re-cut tag changes prod authorization with no PR and no review.
  2. A required security gate is hostage to a CDN. The same 429 that produced security(render): prod render fetches third-party manifests unpinned, one from a moving branch #3196 takes this render down too.
  3. Tag ≠ digest. Nothing verifies the bytes fetched today are the bytes reviewed when the version was bumped.

Expected

The bundle resolves from reviewed, digest-pinned bytes committed to this repository, with the render performing no network fetch.

The coupling that must not be missed

.github/renovate.json carries a regex custom manager keyed on this URL's path:

"matchStrings": ["kubelet-serving-cert-approver/v(?<currentValue>[0-9.]+)/deploy/"]

Vendoring removes that URL, so the manager silently stops matching and the dependency stops being tracked altogether — a worse outcome than the fetch, because it fails open and quietly. The manager must be re-pointed at whatever pins the vendored copy in the same change.

The precedent for how vendoring and Renovate coexist here is already in that file: the shared-installer manager notes that those installers also pin the asset's SHA-256 on the next line, which no datasource can supply, so a bump fails the install with the digest to write rather than installing unverified bytes. The same shape applies here.

Also carry forward, do not drop

The kustomization's existing comment records two facts a vendoring change must preserve: the HA variant is deliberate (Coroot flagged the single replica as not node-failure resilient, and upstream's ha-install.yaml is the supported HA path — replicas: 2, leader election, the leases Role/RoleBinding, and a required hostname podAntiAffinity), and on a version bump the standalone PDB selector must be re-checked against the upstream labels.

Acceptance criteria

Size

Medium. Larger than #3445 despite the milder defect, because of the Renovate coupling and the workload (rather than CRD-only) Checkov surface.

Completed 2026-09-06: #3635 merged at 45841c15cd44d1a4eda8497bbea0995faf9951b7 after current-head review and successful merge-group CI and deployment. The remaining vendoring work is delivered; no maintainer action is pending.

Activity

  1. moved this to 📥 Backlog in 🌊 Project Boardon Aug 29, 2026
  2. devantler commented on Aug 29, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Startability note — this needs a boundary clearance an unattended run cannot obtain

    Vendoring means fetching the upstream bytes, and those bytes are repository-hosted content (raw.githubusercontent.com/<owner>/<repo>/…). The engineering contract's Professional-work repository boundary requires the maintainer's current, explicit, in-session confirmation before any action against a repository outside devantler-tech, and states that scheduled runs are portfolio-only; External-blocker verification separately rules out "the upstream repo page, tree, issue, API, or repository-hosted releases feed" and permits only non-repository channels. The boundary resolves ambiguity in the closed direction, so a scheduled tick fails closed here rather than reasoning its way to an exception.

    This is not a claim that the upstream is out of scope on the merits — it is a well-known public project and our own CI already fetches it, which is precisely the defect. It is that the clearance is a maintainer act, and no unattended run can produce one.

    So this issue is startable in an interactive session, and skippable (not blocked) in a scheduled one. Deliberately not labelled blocked: there is no external dependency to wait on and nothing to re-verify each tick, so a blocked label plus a **Blocker:** line would misrepresent it and park it behind a re-verification that has nothing to check. Recording it here instead so a scheduled run recognises the shape immediately rather than re-deriving it, and so the boundary judgement is visible and challengeable rather than silent.

    Everything not requiring the upstream fetch is startable unattended — most usefully, confirming that the exceptions-row deletion this issue owns actually fails the guard while the remote is still referenced.

  3. devantler commented on Aug 29, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Same blocker as #3445, recorded here so this issue is not selected and abandoned in turn

    Vendoring this bundle requires downloading https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/<ref>/deploy/ha-install.yaml, a path into a third-party repository tree. AGENTS.md Professional-work repository boundary requires the maintainer's explicit confirmation for any repository outside devantler-tech before even a read-only action, and an unattended run cannot obtain it.

    The reasoning, the counter-argument, and what would unblock both children in one decision are written up on #3445 — this is deliberately not restated here so there is one place to answer.

    This issue is otherwise ready: unclaimed, specified, with its exceptions row already in place awaiting deletion.

    Blocker: upstream-artifact-fetch-authorization | last-verified 2026-08-29: not granted — see #3445.

  4. devantler commented on Sep 2, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    New occurrence — this fetch broke CI today on an unrelated PR.

    While validating platform#3540 (a security-context rollout change touching only a namespace
    label and a test pin), the 🔐 Validate Production Authorization job failed outright:

    kustomization validation failed: run validation group:
    providers/hetzner/infrastructure/controllers/kubelet-serving-cert-approver:
    build failed: accumulating resources:
    accumulation err='accumulating resources from
    'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml':
    read: connection reset by peer'
    

    Recorded 2026-09-02, run 33684685162.

    Why this is worth noting on the issue rather than just re-running. The fragility this issue
    describes is usually framed as a supply-chain concern — an unpinned third-party manifest can change
    under us. This occurrence shows the second cost: the fetch is also a liveness dependency of the
    production authorization gate
    . A transient network failure at raw.githubusercontent.com turns a
    required check red on a change that has nothing to do with this controller, and the only remedy
    available to an unattended run is to re-run and hope.

    That makes the vendoring change a CI-reliability fix as well as a security fix, which may be
    relevant to prioritising the artifact-fetch clearance this issue is parked on.

    No action requested here — the blocker is unchanged and, per the standing note, the maintainer is
    deliberately not being re-notified for the same decision.

  5. moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Sep 6, 2026
  6. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Security.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions