Repository navigation
Private-only nodes: fix cross-node Cilium mutual-auth (public-IP dial) + harden #2272
Description
🤖 Generated with Claude Code (interactive session with the maintainer)
Problem
Cross-node Cilium mutual authentication is failing intermittently, and as a result mTLS is failing open on the dropped flows (auth-required traffic isn't enforced when the handshake can't complete). Symptoms surfaced in Coroot: ClickHouse interserver (:9009) connect timeouts, Longhorn replica faults, SPIRE entry churn, and short-lived CronJob flakes (coroot-custom-cloud-pricing, umami-provision-tenants) — all the same root cause.
Root cause (verified)
The cilium-agent mutual-auth handshake (TCP :4250) is dialed to a node's public ExternalIP instead of its private 10.0.1.x, where the private-only Talos ingress firewall drops it (dial <public-ip>:4250: i/o timeout, fresh in agent logs).
It is non-deterministic (≈1 in 3 dials), which is why everything flakes rather than fails hard: Cilium's GetNodeIP(nodeID) (pkg/datapath/linux/node_ids.go) returns an arbitrary IP from an unordered set of all the node's addresses (private + pod + public) — no address-type preference, no sorting. Confirmed in the deployed v1.19.5 and upstream main; there is no config flag (cilium/cilium#31902 is an open feature request for address preference). The public IP is in that set because hcloud-ccm stamps every Node with ExternalIP=<public>, and hcloud-ccm v1.32 has no toggle to suppress it.
The existing
devices: "enp7s0 eth1"pin (.../cilium/patches/helm-release-patch.yaml) only steers the datapath node IP — it does not influence the auth dial, so it does not fix this. Its header comment currently claims it does; that comment must be corrected (see tasks). Upstream bug filed separately.
Proposed direction — private-only nodes
Stop nodes carrying a public ExternalIP at all. This removes the unreachable address from Cilium's node-address set entirely (so GetNodeIP can only ever return a reachable private IP), and hardens the cluster (no worker/CP exposed on the public internet). The cluster already runs effectively private: the Gateway LB targets node private IPs (load-balancer.hetzner.cloud/use-private-ip: true), and pod egress is device-agnostic masquerade.
App access is unaffected — end-user traffic is LB-public-IP → node-private-IP and never touches node public IPs.
Incremental rollout (each step independently shippable + reversible; nodes keep public IPs until proven)
- Egress path — stand up internet egress for the private network (NAT gateway / Hetzner) so private-only nodes can still pull images, do ACME, and reach the Hetzner API. Verify egress on a canary before removing any public IP.
- API admin path — the kube-apiserver is currently reached at a control-plane public IP (
https://49.13.53.183:6443). Provide a public→private API path that doesn't depend on a node public IP (control-plane API LoadBalancer → CP private IPs:6443, or VPN/bastion) and re-point kubeconfig. Do this before removing CP public IPs or you losekubectl. - Canary one worker private-only; validate egress, pod-to-pod, and that cilium-agent auth handshakes now use
10.0.1.x(no more:4250timeouts). Roll the remaining workers, then control-planes. - Autoscaler pools — configure the cx33/cx43/cx53 pools to create private-only nodes (disable
HCLOUD_PUBLIC_IPV4/IPV6) using the same egress. - Verify & clean up —
cilium connectivity testmutual-auth (cross-node) passes; Coroot ClickHouse/Longhorn/SPIRE alerts clear; correct/remove the misleadingdevices-pin comment and re-assess whether the pin is still needed.
Acceptance criteria
- No
:4250 i/o timeoutin any cilium-agent log; cross-node mutual-authcilium connectivity testpasses. - Coroot ClickHouse
:9009, LonghornFaulted, and SPIRE entry-churn alerts clear and stay clear. - No node carries a public
ExternalIP; app ingress +kubectladmin still work.
Risk / rollback
Each step is reversible and nodes retain public IPs until egress + API paths are proven. The only hard ordering constraint: egress and the API LoadBalancer must exist before the corresponding public IPs are removed.
Activity
devantler commentedon Jul 11, 2026
Fresh read-only prod evidence on 2026-07-11 ties two more user-visible flakes to this same public-
ExternalIPmutual-auth path; both recovered automatically, so no separate incident issue was opened.keda-hpa-homepage-primaryandkeda-hpa-umami-umami-primaryfailed external-metric fetches at 05:10:18Z. KEDA logged Coroot Prometheus HTTP timeouts while the Cilium agent on the Prometheus node logged failed SPIRE auth to the KEDA node's public IP on:4250. Both HPAs recoveredValidMetricFoundat 05:10:29Z; ScaledObjects are Ready/Active with zero failures and both apps remain 2/2.prod-worker-2. The same node's Cilium agent logged failed SPIRE auth to the peer node's publicExternalIP:4250throughout the window. All replicas rebuilt by 05:13:59Z; all 14 volumes are healthy and all 42 replicas are running. An earlier Coroot DB replica fault at 04:40Z shows the same cross-node/public-IP auth correlation and recovered within 38 seconds.Control-plane health stayed green: 9/9 nodes Ready, 10/10 Flux Kustomizations Ready, and 44/44 HelmReleases Ready. This strengthens the case that #2272 is the shared root cause rather than independent KEDA or Longhorn defects, and adds two concrete verification signals for the private-only-node rollout: external-metric stability and snapshot-purge RPC health.
devantler commentedon Jul 13, 2026
🤖 Generated by Codex Daily AI Engineer (scheduled automation).
Live recurrence evidence from 2026-07-13 (UTC), sanitized and read-only:
observability/coroot-db-6Longhorn volumepvc-13f08497…rebuilt a Faulted replica onprod-worker-3from 12:53:23–12:53:42. The volume is now healthy with 3/3 replicas, the node/disk are Ready, and CNPG is 3/3 Ready with backup/archiving healthy. The volume's Degraded event series has recurred 15 times across July 10–13.keda-operatorservice timeout at approximately 13:15, then recovered by 13:18. Both ScaledObjects are currently Ready/Active, failure count is zero, live external-metric reads succeed, KEDA pods/endpoints are Ready, and both workloads are 2/2. The shared timeout series has recurred roughly 12–13 times across July 12–13.This matches the paired transient KEDA/Coroot + Longhorn signatures already tracked here. No emergency mutation was made because both paths recovered; acceptance should still require eliminating the recurrent event series, not only observing a green point-in-time state.
devantler commentedon Jul 14, 2026
Live recurrence — 2026-07-14 16:48 UTC
The latest merge-group attempt for
platform#2611exposed a persistent Coroot CNPG replica failure that matches this issue's cross-node network signature; it is not evidenced as a regression in the Renovate KSail bump.Cluster/observability/coroot-dbremainingInProgress;appswas dependency-blocked behindinfrastructure.2/3ready with healthy primarycoroot-db-8and replicacoroot-db-6. Instancecoroot-db-7is running but PostgreSQL readiness returns HTTP 500.pg_rewindafter switchover. Its first approximately 582 MB rewind reached 99% and then failed withConnection timed out; later retries cannot restore/read WAL segment000000080000001C0000009E.ContinuousArchiving=True,LastBackupSucceeded=True, andcoroot-db-daily-20260714033000completed successfully at 03:30 UTC. No Secret data was read.Safe operational posture: keep
platform#2611dequeued, preserve the healthy primary and replica, and recover/reseed only the failed CNPG instance through the operator-supported path after re-verifying backup/archiving state. No manual production mutation or requeue was performed in this run.devantler commentedon Jul 14, 2026
🤖 Codex Daily AI Engineer instance
2026-07-14 17:07 UTC follow-up
The automatic main-heal path did not recover the incident:
workload reconcileexited non-zero2/3ready with healthy primarycoroot-db-8, healthy replicacoroot-db-6, and unready replicacoroot-db-7could not restore file "000000080000001C0000009E" from archivefollowed bycould not read WAL record at 1C/9E000028ContinuousArchiving=TrueandLastBackupSucceeded=True; the two healthy instances remain available, so this is degraded HA rather than total database lossinfrastructureis still progressing andappsremains blocked on that dependencyPR #2611 remains dequeued. No manual pod/PVC deletion, replica reseed, or other production mutation was performed; recovery remains constrained to the repository runbook/operator path.
devantler commentedon Oct 17, 2026
Upstream release review, 2026-10-17 — this epic's direction needs a decision. The Cilium 1.20 upgrade notes list Mutual Authentication (Beta) under Action Required: it is deprecated and will be removed in a future version, and the suggested alternative is Ztunnel Transparent Encryption (Beta).
Production runs Cilium 1.20.2 with Cilium authentication and its SPIRE integration disabled, and the platform instructions plan to re-enable them alongside a narrow consumer. Fixing cross-node mutual authentication for private-only nodes therefore builds on a feature upstream is removing. Before more work lands here, decide between Ztunnel-based encryption, the existing WireGuard encryption with network policy, or dropping the mutual-authentication goal, and record the choice on this issue.
devantler commentedon Oct 17, 2026
Decision record: the maintainer's direction of 2026-10-17 (canonical record: #5001).
The VPN is not for protecting internal web pages; zero trust, the sign-on gate in front of them, already does that. The VPN's purpose is to bridge into the home network so integrations can reach the maintainer's hardware and appliances. Each appliance is either exposed publicly through zero trust (preferred where judged safe) or kept on an internal route that requires the VPN, and anything found compromisable moves to VPN-only quickly until it is safe again.
For this epic: the home VPN has that one purpose, so step 2 should not plan on it as the Kubernetes API admin path (its "or VPN/bastion" alternative); the control-plane API load-balancer path fits the direction. This direction does not decide the open choice in the 2026-10-17 comment: "WireGuard encryption" there is Cilium's node-to-node encryption, not the home VPN.