Skip to content

feat(whoami): pilot user namespaces (hostUsers: false) in prod - #1808

Merged
devantler merged 6 commits into
mainfrom
claude/heuristic-shannon-dd4ee6
Jun 8, 2026
Merged

devantler merged 6 commits into
mainfrom
claude/heuristic-shannon-dd4ee6

Conversation

@devantler

Copy link
Copy Markdown
Contributor

Part of #1807 (k8s 1.36 readiness).

What

Run the stateless whoami pod inside a private user namespace by setting hostUsers: false on its pod spec — the first, canary step of adopting user namespaces across the platform.

User Namespaces (KEP-127) graduated to GA in Kubernetes 1.36 and has been beta/on-by-default since 1.30, so this already takes effect on the cluster's current v1.35.5 — no version bump required. The container's UID/GID (65534) is mapped to an unprivileged subordinate range on the host, so a container-runtime escape lands as a non-root, capability-less host user — defence in depth on top of the restricted securityContext already set in the base HelmRelease.

Why whoami, why prod-only

  • whoami is the ideal canary: stateless, no volumes, and automountServiceAccountToken is already false — so there are none of the idmap-volume constraints that make stateful workloads (Longhorn/CNPG) the last thing to migrate.
  • Scoped to the hetzner (prod) overlay because prod nodes run Talos on real Hetzner kernels where user namespaces + idmap mounts are supported, whereas the local Docker provider runs Talos nested in Docker where userns support is not guaranteed. This matches the repo's convention for environment-capability-dependent settings (cf. the prod-only require-mutual-auth policy).

How

An RFC 6902 add that appends (/-) to whoami's existing postRenderers[0].kustomize.patches list, so the base's strategy / ServiceAccount post-render patches are preserved rather than replaced (a strategic-merge patch on the list would replace it). The chart (cowboysysop/whoami 6.0.0) doesn't expose hostUsers in values, so a post-render patch is the mechanism.

Validation (static only — no cluster)

  • kubectl kustomize builds clean for providers/hetzner/apps, clusters/prod, and clusters/local.
  • The rendered whoami HelmRelease carries all three post-render patches, with hostUsers: false (boolean) on the Deployment pod template.
  • providers/docker/apps renders 0 resources → the pilot is genuinely prod-only.
  • ksail --config ksail.prod.yaml workload validate → providers/hetzner/apps validated ✔. The only failure is the pre-existing stale Coroot CRD-catalog schema (notificationIntegrations) in the infrastructure layer, unrelated to this change.

How to verify once deployed to prod

kubectl -n whoami get pods                 # whoami pods Running, not CrashLoop
kubectl -n whoami exec deploy/whoami -- id # UID 65534 inside; remapped on host

Watch for: pod fails to start / unable to set up user namespace events. Rollback = revert this PR (removes the patch; whoami returns to host userns).

Next (tracked in #1807)

homepage (apex; has read-only configMap mounts to verify under userns) → then stateful workloads, last.

Run the stateless whoami pod inside a private user namespace (KEP-127),
which graduated to GA in Kubernetes 1.36 and has been beta/on-by-default
since 1.30 (so it already works on the cluster's current v1.35.5). The
container's UID/GID (65534) is mapped to an unprivileged subordinate range
on the host, so a runtime escape lands as a non-root, capability-less host
user — defence in depth on top of the restricted securityContext already in
the base HelmRelease.

Scoped to the hetzner (prod) overlay: prod nodes run Talos on real Hetzner
kernels where user namespaces + idmap are supported, whereas the local
Docker provider runs Talos nested in Docker where userns support is not
guaranteed. whoami is the canary — stateless, no volumes,
automountServiceAccountToken already false — so there are no idmap-volume
constraints. Once validated in prod, extend to homepage and beyond.

Implemented as an RFC 6902 append (/-) to whoami's existing
postRenderers[0].kustomize.patches list so the base strategy / ServiceAccount
patches are preserved rather than replaced.

Validation: kubectl kustomize of providers/hetzner/apps, clusters/prod and
clusters/local all build; the rendered whoami HelmRelease carries all three
post-render patches with hostUsers: false; providers/docker/apps renders 0
resources (pilot is prod-only). `ksail --config ksail.prod.yaml workload
validate` reports providers/hetzner/apps validated — the only failure is the
pre-existing stale Coroot CRD-catalog schema in the infrastructure layer,
unrelated to this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kustomization comment restated the prod-only rationale already documented
in full at the top of whoami/patches/helm-release-patch.yaml. Reduce it to a
one-line pointer, matching the repo convention of keeping the rationale in the
patch file (surfaced by a /simplify pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devantler
devantler marked this pull request as ready for review June 6, 2026 08:16
@devantler
devantler marked this pull request as draft June 6, 2026 08:16
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Analysis by Claude Code (at the maintainer's request)

The System Test failure is the known infra-only reconcile flake — not caused by this PR. No code change is warranted; a re-run should clear it.

The cluster came up healthy (etcd, CNI, cert-manager, CSI, metrics-server, policy-engine, Flux all installed), then infrastructure, infrastructure-controllers, and apps failed to reconcile within the timeout with transient infrastructure signatures:

  • client rate limiter Wait returned an error: context deadline exceeded — API/etcd overload on the single-control-plane Docker runner
  • minio-create-bucket … dial tcp …:9000: connect: no route to host — Cilium networking-convergence race
  • helper-pod-create-pvc-… not found in namespace "local-path-storage" — local-path-storage race

Why it isn't this PR

  1. The identical code passed System Test yesterday — run on this branch at 2026-06-05T22:08 (commit a89cf446) was green. The only delta since is dbd74219, a one-line YAML comment trim — causally incapable of breaking a cluster reconcile.
  2. The failing Kustomizations are infrastructure / infrastructure-controllers, which this PR does not touch. apps fails only as a cascade (it dependsOn infrastructure).
  3. This change is prod-only (hetzner overlay); the local Docker apps overlay renders 0 resources, so whoami / hostUsers is never deployed in the System Test at all.
  4. The same flake is hitting unrelated runs right now — the main merge queue failed repeatedly today (02:10, 02:11, 06:31, 07:49) while other PRs intermittently pass.

Recommendation

Re-run the 🧪 System Test job (or let the merge queue retry). This matches the documented infra-only-flake handling. The flake recurrence across the merge queue suggests CI-runner capacity / the single-CP etcd-overload issue is acting up today and may warrant separate attention.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Update

Full root-cause of the stuck merge queue is now tracked in #1816 — a prod wedge (spire-server's Longhorn volume stranded on a deleted autoscaler node → SPIRE down → ExternalSecrets cascade), not this PR. This PR stays BLOCKED until the queue drains (see #1816 §1 for the un-wedge steps); no change needed here.

@devantler
devantler marked this pull request as ready for review June 6, 2026 22:58
@botantler-1
botantler-1 Bot enabled auto-merge June 7, 2026 17:41
@botantler-1
botantler-1 Bot added this pull request to the merge queue Jun 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 7, 2026
@devantler
devantler added this pull request to the merge queue Jun 8, 2026
Merged via the queue into main with commit d758266 Jun 8, 2026
10 checks passed
@devantler
devantler deleted the claude/heuristic-shannon-dd4ee6 branch June 8, 2026 16:43
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Jun 8, 2026
@botantler-1

botantler-1 Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 1.40.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

This branch was previously deployed

1 inactive deployment
ci — b1050815 Deployed Jun 7, 2026 by devantler via 🧪 System Test #2225
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant