Skip to content

Normalize Cilium devices before bandwidth-manager activation #2610

New issue
New issue
Closed
Performance
Closed
Normalize Cilium devices before bandwidth-manager activation#2610
Performance
Assignees
devantler

Description

@devantler
devantler
opened on Jul 12, 2026

🤖 Generated by the Daily AI Engineer

Summary

Make Cilium's production device selection homogeneous across every managed node before the default-off bandwidth manager is activated.

Motivation

The current Hetzner overlay intentionally pins private-network devices as enp7s0 eth1: workers use enp7s0, while control planes use eth1. That protects the existing WireGuard and mutual-auth path, but Cilium's bandwidth-manager guidance requires each manually listed device name to exist consistently on every managed node. Enabling #2608 before resolving this mismatch could leave one node class without the expected FQ qdisc or fail the agent roll.

Proposed solution

Reconfirm the live six-node inventory, then choose the smallest safe homogeneous strategy: normalize the private NIC name in Talos, or adopt a supported auto-detection/selection mechanism that still resolves only the private-network NIC on both node classes. Keep the bandwidth-manager component off until the strategy is proven.

Acceptance criteria

  • Live read-only inventory records the private-network interface name and address on every control-plane and worker node.
  • The chosen selection is homogeneous for all Cilium-managed nodes and preserves private-IP primary-device behavior.
  • Local and production renders validate in both default and staged states.
  • WireGuard, strict mode, SPIRE mutual-auth, Cilium health, and external egress assumptions are explicitly preserved.
  • Rollout is one node at a time in a low-traffic window with a documented rollback.
  • Post-roll verification proves Cilium Ready and the expected bandwidth-manager device on every node before feat(cilium): stage bandwidth manager with BBR #2608 activation proceeds.

Alternatives considered

  • Keep the heterogeneous manual list: rejected for bandwidth-manager activation because upstream requires consistent names.
  • Remove devices immediately: rejected because that pin currently prevents the public-NIC mutual-auth failure documented in the overlay.

Part of #2029. Blocks activation of #2608. Rough size: medium.

Activity

  1. devantler
    added
    enhancement
    performance
    on Jul 12, 2026
  2. github-project-automation
    added this to 🌊 Project Boardon Jul 12, 2026

  3. coderabbitai commented on Jul 12, 2026

    @coderabbitai
    coderabbitaiBot
    on Jul 12, 2026 – with coderabbitai
    Contributor
    🔗 Related PRs

    #2370 - fix(autoscaling): break the DaemonSet-VPA node ratchet; keep Longhorn storage off autoscale nodes [merged]
    #2462 - feat(talos): WireGuard server listener on control planes (WS3 phase 1) [merged]
    #2465 - docs: design for requiring VPN in front of critical services [merged]
    #2474 - refactor: name patch fragments by intent and codify patch naming rules [merged]
    #2524 - feat: enable the Hetzner floating-IP stable API endpoint on prod [merged]

    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan
    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  4. devantler
    added a commit that references this issue on Jul 17, 2026

    docs(cilium): encode the public-NIC masquerade attachment as a hard a…

    9c237f4

  5. devantler commented on Jul 17, 2026

    @devantler
    devantler
    on Jul 17, 2026
    ContributorAuthor

    🤖 Generated by the Daily AI Engineer

    Additional acceptance requirement surfaced by #2680's review (codex P1): homogeneous naming alone is not sufficient. BPF masquerading (staged default-off in #2680 for #2609) only processes interfaces carrying Cilium's eBPF programs, and pod->internet egress leaves via the PUBLIC default-route NICs (workers enp1s0 / control planes eth0) — which today's private-only devices: "enp7s0 eth1" pin excludes. This slice must therefore deliver a device selection that (a) attaches Cilium's programs — including the masquerade program — to the public default-route NICs, while (b) preserving the private-NIC primary-address behaviour the pin exists for (SPIRE mutual-auth dials primary addresses; see enforce-wireguard-strict-mode.yaml). Until that lands, the ebpf-host-routing component's reference carries a HARD do-not-enable blocker naming this issue.

  6. devantler
    added a parent issue on Jul 18, 2026
  7. devantler
    moved this to 📥 Backlog in 🌊 Project Boardon Jul 18, 2026
  8. devantler
    added the
    Performance
    issue type on Jul 18, 2026
  9. devantler
    self-assigned this
    on Jul 18, 2026
  10. devantler
    added a commit that references this issue on Jul 18, 2026

    feat(cilium): stage homogeneous wildcard device selection for datapat…

    bb45a3a

  11. devantler commented on Jul 18, 2026

    @devantler
    devantler
    on Jul 18, 2026
    ContributorAuthor

    🤖 Generated by the Daily AI Engineer

    AC1 delivered — live nine-node inventory (2026-07-18, read-only talosctl get addresses + kubectl get nodes/ciliumnodes, all nodes Ready):

    Node Class Public NIC (default route) Private NIC (10.0.1.x)
    prod-control-plane-2 static CP enp1s0 enp7s0
    prod-control-plane-4 static CP enp1s0 enp7s0
    prod-control-plane-5 static CP enp1s0 enp7s0
    prod-worker-1 static worker enp1s0 enp7s0
    prod-worker-2 static worker enp1s0 enp7s0
    prod-worker-3 static worker enp1s0 enp7s0
    autoscale-cx33-12ac9ae5b0b9effe autoscale eth0 (+IPv6) eth1
    autoscale-cx33-27e0aeaa0c6be36 autoscale eth0 (+IPv6) eth1
    autoscale-cx43-d63a60203d535b1 autoscale eth0 (+IPv6) eth1

    This INVERTS the issue premise: the split is no longer workers-vs-control-planes but static-vs-autoscale — the control planes were rebuilt onto enp-named machines (all three CP agents log devices=[enp7s0]). Today's pin "enp7s0 eth1" covers every private NIC only by union, each node matching exactly one name.

    Chosen strategy (staged in #2697, default-off): wildcard patterns devices: "en+ eth+" — identical on every node, per-node resolving to only the devices that exist, covering the public egress NICs the masquerade AC requires, and absorbing future autoscaler naming drift (ens*/eno*). Upstream documents wildcards for exactly this cross-node-inconsistent-naming case; an explicit four-name list would violate the bandwidth-manager consistency guidance on every node.

    The delivery PR also had to restructure the pin into an always-on component: render-testing proved a parent-patch devices value can NEVER be overridden by a component (parent patches apply last), so the old in-patch pin would have made the staged component a silent no-op.

    Remaining to close this issue: the activation flip (one-line component reference) executed per the component runbook — first-node primary-address verification before the roll continues, then all-nine post-roll verification. Only after that may #2609/#2608 activation proceed.

    Edited 2026-09-28 by the Agentic Engineer: per-node IP addresses removed from the table (sensitive-information hygiene); interface names and classes unchanged.

  12. devantler
    assigned
    devantler
    and unassigned
    devantler
    on Jul 18, 2026
  13. devantler
    added a commit that references this issue on Sep 28, 2026

    chore: agent-claim #2610 nonce=9ededebdd37ceab7c312d5aafed62b98

    4096868

  14. devantler commented on Sep 28, 2026

    @devantler
    devantler
    on Sep 28, 2026
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Delivered — closing with a fresh live check

    This issue's work shipped in #2697 (staged the wildcard device selection) and #2793 (activated it, 2026-07-25). The rollout gate was released afterwards on main (97721800, 533f6f3d). Nothing linked it back here, so it stayed open. I re-checked every criterion against production today (2026-09-28), read-only.

    Criterion Evidence today
    Live inventory of each node's private interface talosctl get addresses/get links on all 7 current nodes: every static node (3 control planes, 3 workers) uses enp1s0 public / enp7s0 private; the one autoscaler node uses eth0 public / eth1 private. Same static-vs-autoscale split recorded on 2026-07-18.
    Selection is the same on every node and keeps the private address primary The live cilium-config has devices: en+ eth+, identical everywhere. On all 7 nodes CiliumNode reports InternalIP on the private network. That includes the autoscaler node, which booted straight into the widened set.
    Renders validate main CI is green with homogeneous-devices/ referenced.
    Staged, reversible rollout The component header documents the stepped runbook and the rollback (re-comment the reference).
    Post-roll health before #2608 proceeds The Cilium DaemonSet shows 7/7 ready and 7/7 updated, with 0 restarts. Agent logs over the last 6h hold 0 error-level lines and 0 auth timeouts on every node.

    What stays open is tracked elsewhere. The bandwidth manager (#2608 component) and eBPF host routing (#2609 component) are still commented out in k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml. Activating them is the remaining work under #2029.

    Also removed the per-node IP addresses from my 2026-07-18 comment above. Interface names are enough for this record.

  15. devantler
    closed this as completedon Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Performance

Fields

No fields configured for Performance.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions