Workload egress to the instance metadata address is unconstrained, and the old fix targets a deliberately deleted file #3153

Activity

1. github-project-automation

   added this to 🌊 Project Boardon Aug 15, 2026
2. devantler

   added the

   Security

   issue type on Aug 15, 2026
3. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Aug 15, 2026
4. coderabbitai commented on Aug 15, 2026

   

   coderabbitaiBot

   on Aug 15, 2026 – with coderabbitai

   Contributor

   More actions

   🔗 Related PRs

   #2704 - fix(aws): constrain eks-ci KMS grant creation [merged]
   #2728 - fix(apps): restrict public Umami routes [merged]
   #3037 - fix(cilium): refuse a zero-count gate release and correct the runbook order [merged]
   #3103 - feat(security): assert the node image verifier can enforce at all [merged]
   #3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
5. devantler

   self-assigned this

   on Aug 27, 2026
6. github-project-automation

   moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Aug 27, 2026
7. devantler

   closed this as completedin #3400on Aug 27, 2026
8. github-project-automation

   moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Aug 27, 2026
9. devantler

   reopened this on Aug 27, 2026
10. github-project-automation

    moved this from ✅ Done to 🫴 Ready in 🌊 Project Boardon Aug 27, 2026
11. devantler commented on Aug 27, 2026

    

    devantler

    on Aug 27, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Reopening: the fix that closed this is being reverted in #3404, so workload egress to the instance metadata address is unconstrained again.

    Why the fix was reverted. The cluster-wide policy also stopped CoreDNS reaching the Talos host DNS resolver, which took out external name resolution for the whole prod cluster. The policy landed on-cluster at 14:06:51Z and the first CoreDNS 169.254.116.108:53: i/o timeout is 9.9 seconds later, after 27 hours of clean logs, with both CoreDNS pods failing within 0.75s of each other.

    What this changes about the acceptance criteria. The control itself is still wanted; what failed was the evidence behind it. The test that shipped with it asserted only how the policy rendered — selector shape, one deny rule, the exact CIDR — so it passed while the policy broke the cluster. A render assertion cannot observe an egress decision.

    Re-landing therefore needs, in addition to the original criteria:

    • validation that exercises the host-directed egress path CoreDNS depends on, not just the rendered document, so that a policy which blocks it fails before it reaches prod;
    • confirmation of what actually broke that path. The deny targeted 169.254.169.254/32 alone, and the resolver is a different address in the same link-local range, so the mechanism is either the policy enabling egress default-deny on every endpoint it selects, or a CIDR identity change affecting host-directed traffic. That was not determined during the incident — kubectl exec into the Cilium agent is outside this agent's read-only prod access, and the question needs answering before the same selector is used again;
    • a narrower blast radius. The selector covered every pod outside crossplane-system, including cluster infrastructure in kube-system, which is not the workload the control is aimed at.

    Kept in place: the equivalent deny in the Crossplane namespaced policy. That namespace was excluded from the cluster-wide selector and already carried its own egress allow rules, so it is not implicated and the metadata address remains blocked there.
12. devantler commented on Aug 27, 2026

    

    devantler

    on Aug 27, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Already delivered and verified live — closing. Picked this up as the oldest unblocked type:Security issue; the assignment was ~21h stale with no PR and no claim ref, so I took it over, and found the work complete. Verifying rather than reimplementing.

    The intent was landed after this issue was filed, across three PRs, and none of them closed it:

    PR	what it did
    #3400	fix(security): block workload access to instance metadata — the deny, against the current layout
    #3406	fix(networking): keep metadata policy off Gateway identity
    #3407	fix(cilium): stop the metadata deny revoking all workload egress

    It lives at k8s/providers/hetzner/infrastructure/cilium/cilium-clusterwide-network-policy-deny-instance-metadata-egress.yaml — a new file, not the deleted one this issue warned against.

    Acceptance criteria

    • Denied for workloads, against the current layout. egressDeny: [toCIDR: 169.254.169.254/32], selecting pod namespaces via endpointSelector (excluding crossplane-system, which keeps the same deny in its own egress policy). Reached through .../cilium/kustomization.yaml, referenced at line 16 of the hetzner infrastructure kustomization. The deleted infrastructure/controllers/cilium/ path was not restored — it holds only components/ and patches/.
    • Host/Talos bootstrap unaffected. nodeSelector is deliberately omitted, so the policy covers pods and not hosts. Live: the policy has been in force since 2026-08-27T14:06:51Z, and ~8.5 h later the cluster reads 7/7 nodes Ready with 0 running pods carrying a non-ready container.
    • Mutual-auth regressions still pass. scripts/tests/test-cilium-mutual-auth-policy-regressions.sh → exit 0, "PASS: Cilium authentication guard covers every production policy shape and consumer state".
    • Rendered from a build that covers the changed path. kubectl kustomize k8s/providers/hetzner/infrastructure renders 11,125 lines and contains the policy with the correct spec. This also reproduces Documented no-KSail validation fallback validates zero base manifests #3151: kubectl kustomize k8s/clusters/prod renders only 126 lines and does not contain it — the documented false green, confirmed rather than assumed.
    • In effect, not merely schema-valid. Read live from the cluster: status.conditions reports Valid=True — Policy validation succeeded, and the realized spec is enableDefaultDeny={"egress":false} with egressDeny=[{"toCIDR":["169.254.169.254/32"]}].

    Stated limit: the evidence above is that Cilium accepted and programmed the policy on a healthy cluster, not an observed dropped packet. Capturing an individual verdict needs Hubble flow access, which is beyond the read-only kubectl get this check used. If a per-flow drop record is wanted as a standing signal, that is worth its own issue rather than holding this one open.

    Worth keeping visible

    #3407 is why the enableDefaultDeny block in that file exists, and it is the trap anyone touching this policy will meet again: Cilium turns default-deny on for any direction a policy has rules in — deny rules included — so a deny-only CCNP revokes all egress from every endpoint it selects. Here that took out cluster-autoscaler, snapshot-controller, tetragon-operator and hubble-relay, while CoreDNS survived only because it had an egress allow policy of its own. The explicit enableDefaultDeny: {egress: false} is what makes this policy subtract one destination instead of becoming every workload's egress allow-list. The comment in the file records this; please keep it there.
13. devantler

    closed this as completedon Aug 27, 2026
14. github-project-automation

    moved this from 🫴 Ready to ✅ Done in 🌊 Project Boardon Aug 27, 2026