Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932

Activity

1. devantler

   added

   enhancement

   refactor

   on Jun 9, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Jun 9, 2026
3. devantler commented on Jun 9, 2026

   

   devantler

   on Jun 9, 2026

   ContributorAuthor

   More actions

   Sibling archetype (composes with this one): #1933
4. devantler

   changed the title [-]Generalize the Tenant archetype — evaluate Capsule vs KRO vs Helm library chart[/-] [+]Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart[/+] on Jun 9, 2026
5. devantler commented on Jun 17, 2026

   

   devantler

   on Jun 17, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Assistant

   ADR (Proposed) — Tenant abstraction: Capsule vs KRO vs Helm chart

   Delivering acceptance-criterion 1 ("Decision recorded on Capsule vs KRO vs Helm library chart, with the build-vs-buy rationale") as a Proposed decision record for your ratification — the build-vs-buy call is yours; this is the reasoned recommendation + pilot plan to act on. Status flips to Accepted when you promote it (or redirect).

   Decision drivers (weighted by this platform's reality)

   1. No new controller/CRD in infrastructure-controllers. That layer is this platform's recurring wedge point (CRD-before-CR deadlocks; the merge-queue prod-deploy gates on apps health; nftables/Longhorn boot incidents). Adding an operator there is the highest-cost move available and the issue itself flags it.
   2. De-duplicate the whole skeleton, not a slice of it. The pain is a ~7-file copy. A solution that only abstracts 2–3 of those files leaves the copy-paste (and its drift) mostly intact.
   3. Preserve the audited isolation posture verbatim — cosign-verified OCIRepository, Flux-impersonated namespaced edit SA, Kyverno restrict-tenant-secret-stores + namespaced Vault SecretStore, ghcr-auth ExternalSecret, netpol.
   4. Match the actual scale — ~3–4 single-namespace tenants (ascoachingogvaner, wedding-app, + the github-config / unifi tenants), not a multi-team namespace fleet.
   5. Stay behavior-preserving + render-diffable — ksail workload validate parity, no added overlay/component stacking.

   Evaluation

   KRO (build, typed RGD) — REJECTED. Pre-1.0 v1alpha1; lands an alpha controller + dynamic CRDs squarely in the wedge-prone infrastructure-controllers layer (driver 1 ✗). The typing/kubectl-native upside doesn't pay for that risk at this scale. Revisit only post-1.0 with a concrete typed-API need.

   Capsule (buy, CNCF multi-tenancy operator) — DEFER (evaluated, not now). Mature and purpose-built, but a poor fit here:

   • Model mismatch (answers the issue's open question): Capsule's value is owner-based namespace grouping with auto-inherited ResourceQuota / LimitRange / NetworkPolicy / RBAC across many namespaces per owner. Our tenants are single-namespace + Flux-impersonation — they'd use a sliver of Capsule's model. It's overkill until tenants become multi-namespace, multi-team.
   • Abstracts the minority of the skeleton (driver 2 ✗): Capsule owns namespace/RBAC/netpol/quota. But the bulk of the duplicated files are Flux sources (OCIRepository+cosign, impersonating Kustomization), the ghcr-auth ExternalSecret, the SecretStore, and the SOPS block — none of which Capsule manages (it explicitly doesn't own Flux sources). So sync.yaml / ghcr-auth / secretstore stay copy-pasted; the 7-file copy becomes a ~4-file copy.
   • Layer + posture cost (drivers 1, 3): still a new controller + Tenant CRD in infrastructure-controllers (CRD-before-CR deadlock; needs ksail workload validate --skip-kinds, like Coroot). And it overlaps the existing Kyverno/Vault/Flux-impersonation posture — folding those into Capsule re-architects a working, audited isolation model; running both = two overlapping policy engines.
   • Reconsider when the fleet grows to multi-namespace teams where quota/RBAC inheritance + self-service namespace provisioning actually pay for the operator.

   Helm chart (build, simplest) — RECOMMENDED. A per-tenant chart (a library chart with thin wrappers, or a standard app chart instantiated via HelmRelease values) templating all ~7 resources; onboarding becomes a ~5-line values block (name, namespace, OCI URL, optional sops:).

   • No new controller/CRD (driver 1 ✓): Flux's helm-controller is already in the platform — HelmReleases are everywhere. Nothing new enters infrastructure-controllers.
   • Not overlay stacking (driver 5 ✓): one indirection (chart → values), far flatter than overlays-on-overlays, and Helm is already in the platform's vocabulary.
   • Owns the entire skeleton (driver 2 ✓): Flux sources, ghcr-auth ExternalSecret, netpol, and the conditional SOPS block — the full duplication, drift included.
   • Posture preserved verbatim (driver 3 ✓): it templates the exact cosign-verified OCIRepository, impersonated SA, Kyverno-compatible namespaced SecretStore, netpol; secrets stay referenced in ESO/SOPS, never inlined.
   • Honest caveats: (a) it adds a templating layer to read — but a single, Helm-native one; (b) the per-tenant HelmRelease placement + chart-source (local OCI/GitRepository chart path) needs a concrete choice in the pilot; (c) Helm release-ownership prune semantics differ slightly from a raw Kustomization — verify prune-on-removal in the pilot; (d) enforce-flux-best-practices requires templating interval/timeout/retries into the generated HelmRelease/Kustomization (bounded, known).

   Recommendation

   Option 3 (Helm chart) for iteration 1 — the only option that adds no controller to the fragile layer, de-duplicates the whole skeleton, and preserves the isolation posture unchanged, while matching the current single-namespace scale. Capsule deferred (revisit at fleet scale), KRO rejected.

   Consequences / pilot (remaining acceptance criteria)

   1. Build a tenant chart: namespace, SA, rolebinding, OCIRepository+cosign, impersonating Kustomization (with best-practice interval/timeout/retries), ghcr-auth ExternalSecret, networkpolicy; optional secretstore + SOPS via a values.sops conditional.
   2. Pilot-migrate ascoachingogvaner (the no-SOPS tenant) local-first; render-diff vs the current bases; confirm ksail workload validate parity.
   3. Keep wedding-app's SOPS block behind the chart conditional for the second migration.
   4. Leave the Kyverno / Vault-SecretStore / cosign posture exactly as-is (the chart re-emits the same resources).

   Composes cleanly with the WebApp archetype (#1933): a Tenant hosts WebApp workloads, so WebApp becomes another values section / sub-chart under the same Flux→HelmRelease model — no second mechanism.

   If you'd rather record this as a committed design doc under docs/ (matching the topic-doc style — there's no ADR dir today, docs/dr is Disaster Recovery), say the word and I'll open a draft PR adding docs/tenant-abstraction.md.
6. devantler commented on Jun 18, 2026

   

   devantler

   on Jun 18, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Assistant

   ADR (Revised) — Tenant/WebApp archetype: Helm ruled out, going Custom-Resource-based

   Maintainer decision (direct): "Custom Resource approach — more cloud-native, so Helm charts are ruled out." This supersedes the Helm-chart recommendation in my prior comment for both archetypes (this issue + the WebApp sibling #1933 — they share one engine). Re-evaluating the CR-composition engine, grounded in what the platform actually runs today.

   What's actually deployed (verified against main, not the stale submodule pin)

   • Crossplane v2 IS installed — but prod/Hetzner-only (k8s/providers/hetzner/infrastructure/controllers/crossplane/), not in the shared infrastructure-controllers layer (the docker/local overlay has no Crossplane). Its only provider is provider-upjet-github — i.e. it manages external APIs (the GitHub org), which is Crossplane's actual strength. There is no provider-kubernetes.
   • KRO is not installed. Latest upstream is v0.9.2 (2026-05-08), still pre-1.0 v1alpha1 — actively moving toward 1.0.
   • ksail-operator (your own, ksail.io/v1alpha1) is in the shared controllers layer, HA — but it reconciles Cluster lifecycle + the web UI/REST API, not workloads.

   CR-engine options

   A. Crossplane Composition (XRD + Composition + provider-kubernetes) — REJECTED (wrong tool here).
   Reusing Crossplane sounds free but isn't, for in-cluster workload composition:

   • It's prod-only today; a Tenant/WebApp XR wouldn't reconcile on local/docker clusters → breaks ksail local dev/validation and the "local-first pilot" unless we also move Crossplane into the shared layer.
   • It needs provider-kubernetes added (not present), and every child (Deployment/Service/HTTPRoute/HTTPScaledObject/ExternalSecret) becomes a wrapped Crossplane Object — off-label: Crossplane is built to project external APIs into the cluster, not to template in-cluster manifests. That's literally why KRO exists.
   • It mixes concerns: Crossplane's clean role here is the GitHub/Hetzner external-API control plane. Don't overload it with app composition. Keep Crossplane for what it's good at.

   B. KRO ResourceGraphDefinition — RECOMMENDED (iteration 1).
   KRO is purpose-built for grouping in-cluster Kubernetes resources behind a typed CRD:

   • Native fit: an RGD defines a typed, namespaced Tenant/WebApp CRD whose instance renders the child resources directly (no Object wrappers, no provider).
   • includeWhen conditionals map exactly onto the optional children — KEDA scale-to-zero set, ExternalSecret/SOPS, auth — which is the readability win the issue is asking for.
   • Works on every cluster (deploy KRO once in the shared infrastructure-controllers layer) → local-first pilot + ksail workload validate parity both work; not prod-gated like Crossplane.
   • Single lightweight controller, kubectl-native, cloud-native — matches the decision.
   • Honest cost (the real trade): pre-1.0 v1alpha1 controller in the wedge-prone infrastructure-controllers layer. You've accepted the CR+controller path; this is the cost of admission. Mitigate by pinning a known-good version, treating it like the other operators (HA, PDB, netpol, replica-floor exception), and keeping the RGD render-diffable.

   C. Extend ksail-operator with a Tenant/WebApp CRD — the maximal-ownership alternative.
   Add the archetype as a first-party CRD reconciled by your own operator (already shared-layer + HA, already serving ksail.io CRDs). Most "owned"/dogfooding + fully cloud-native, no third-party pre-1.0 dependency. Cost: real Go controller work in the ksail product + couples platform archetypes to the ksail-operator release cadence. Best if you want to own the controller rather than adopt KRO — but a larger build than B.

   Recommendation

   Option B (KRO) for iteration 1: the lowest-effort CR path that's purpose-built for in-cluster composition, works on all clusters, and whose includeWhen cleanly expresses the optional children — while leaving Crossplane to its external-API role. Revisit Option C (own it in ksail-operator) if KRO's pre-1.0 status or limits bite, or if dogfooding into ksail is the strategic goal. Crossplane rejected for this purpose.

   One thing to confirm before I build: KRO vs ksail-operator (B vs C) is genuinely your call — B is fastest, C is most-owned. I'll proceed on B unless you redirect.

   Pilot (once you confirm the engine)

   1. Deploy KRO in k8s/bases/infrastructure/controllers/kro/ (HelmRelease + namespace + netpol + PDB + replica-floor exception, matching the operator-tier conventions); add its CRDs to ksail.yaml skipKinds and any Kyverno exception needed.
   2. Author the Tenant RGD (namespace, SA, rolebinding, cosign-verified OCIRepository, impersonating Kustomization with best-practice interval/timeout/retries, ghcr-auth ExternalSecret, networkpolicy; includeWhen SOPS) and the WebApp RGD (Deployment/Service, HTTPRoute → central platform/kube-system Gateway, includeWhen KEDA 5-point set + the cross-ns ReferenceGrant note, includeWhen ExternalSecret).
   3. Pilot ascoachingogvaner (no SOPS, no scale-to-zero) local-first; render-diff the RGD output vs the current manifests; confirm ksail workload validate parity. Then wedding-app exercises the SOPS + scale-to-zero conditionals.
   4. A new app becomes a small typed WebApp CR — no Kustomize overlay/component stacking.

   Status flips to Accepted when you confirm B vs C; I'll then open the draft PR (KRO controller + the RGDs + the local-first pilot).
7. devantler commented on Jun 18, 2026

   

   devantler

   on Jun 18, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Assistant

   Decision (maintainer, 2026-06-19): KRO.

   The Tenant archetype is settled on KRO (kro.run ResourceGraphDefinition) — the purpose-built tool for composing internal Kubernetes resources, matching the "Custom Resource, more cloud-native" steer (Helm ruled out; Crossplane kept for external-API/prod-only).

   Incremental implementation (each behind a draft PR):

   1. Install KRO into the platform infra layer (health-gated like other controllers), pinned to a specific v0.x and kept additive (KRO is pre-1.0 / v1alpha1).
   2. Define the Tenant RGD — the shared namespace/RBAC/quota/SecretStore skeleton, with optional children via includeWhen.
   3. Pilot on ascoaching (local-first), then promote.

   The WebApp archetype (#1933) builds on the same KRO foundation.
8. devantler

   added 2 commits that reference this issue on Jun 22, 2026

   fix(kro): grant the Tenant controller watch on the tenants instance kind

   ...

   229baff

   fix(kro): grant the Tenant controller watch on the tenants instance k…

   ...

   a6dd705
9. devantler commented on Jul 2, 2026

   

   devantler

   on Jul 2, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Assistant

   Step 3 (local-first pilot) started: #2388 (child issue) → draft PR #2389 ships the pilot scaffolding — an opt-in Tenant CR for ascoachingogvaner on the Docker provider (renders unchanged by default), Tenant in both configs' validate skipKinds (the runtime-generated CRD has no catalog schema), and the TENANTS.md pointer. The live local pilot is now a one-line opt-in; the prod ownership transition (Flux-prune-vs-namespace hazard) is deliberately gated on the pilot and will be designed as the next child.
10. devantler commented on Jul 5, 2026

    

    devantler

    on Jul 5, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Daily AI Assistant

    ADR (Proposed) — Step 4: prod ownership transition, skeleton → Tenant CR (zero-prune migration)

    Step 3's scaffolding merged (#2388/#2389); before the live pilot and any prod swap, this records how the live, Flux-owned skeleton objects change hands to KRO without ever being deleted — the tenant namespace holds the CNPG database, and the tenant's own Flux Kustomization carries a kustomize-controller finalizer that garbage-collects the entire app if that Kustomization object is ever deleted with prune: true. Deletion anywhere in the handover is data loss, so the design goal is a migration in which no delete is ever issued.

    Mechanism facts (verified in kro v0.9.2 source — the version our chart 0.9.2 ships)

    1. kro adopts pre-existing objects silently. The instance applier is ApplySet-based server-side apply with FieldManager: kro.run/applyset and unconditional Force: true (pkg/controller/instance/applyset/applyset.go:313–316). The only refusal is a live applyset.kubernetes.io/part-of label pointing at a different ApplySet (applyset.go:345–367); Flux-created objects carry no such label, so kro takes field ownership from kustomize-controller on first reconcile — no "already exists" error, no downtime. Identity is the adoption key, and all 10 skeleton objects already match the RGD templates by kind/name/namespace and field-for-field spec (verified side-by-side today).
    2. No ownerReferences, by design. Children are tracked via ApplySet labels/annotations (kro.run/owned, applyset.kubernetes.io/part-of, instance-id labels); cross-namespace children (instance in flux-system, children in the tenant ns/kube-system) and the cluster-scoped Namespace are covered via additional-namespaces/contains-group-kinds on the parent. Our topology is fully supported.
    3. Flux prunes by inventory and ignores kro's labels. If the 10 objects leave the apps Kustomization's source while still in its inventory, kustomize-controller deletes them unless they carry kustomize.toolkit.fluxcd.io/prune: disabled. kro would recreate them — but for the Namespace and the tenant Kustomization, "delete then recreate" is the catastrophic path (cascade / finalizer GC). Prune-protection must be live before the swap.
    4. Instance deletion always cascades — no orphan option in v0.9.2. kro.run/finalizer walks children in reverse-topological order and deletes them all; DeletionPolicy is an unimplemented TODO, and kro.run/reconcile: suspended does NOT gate deletion (it runs earlier in the reconcile). Orphan/adopt lifecycle policies are open upstream work (KREP-014, KREP-014 Resource lifecycles kubernetes-sigs/kro#1091 — worth watching). So after the swap, deleting the Tenant CR deletes the namespace and the tenant's Flux Kustomization → full tenant GC.

    Decision (proposed): three phases, each a separate release

    • Phase A — prune-protect (safe now, behavior-preserving). Annotate all 10 skeleton objects with kustomize.toolkit.fluxcd.io/prune: disabled. No runtime effect today; it makes the later inventory removal a no-op instead of a delete, and doubles as standing protection against accidental skeleton-file deletion. Ships immediately as a draft PR (child issue below). Must be merged + reconciled on prod before Phase B starts.
    • Phase B — the swap (gated on the local pilot). One PR: delete k8s/bases/apps/ascoachingogvaner/ and add the (prune-disabled-annotated) Tenant CR to the prod apps overlay. Flux drops the 10 objects from inventory without deleting them; kro adopts them via forced SSA within one reconcile interval. Verification: kubectl get … --show-managed-fields shows kro.run/applyset ownership + ApplySet labels present, tenant app untouched (no pod restarts, CNPG untouched).
    • Phase C — cleanup (cosmetic). Stale kustomize.toolkit.fluxcd.io/* labels/annotations linger under the old field manager (kro keeps foreign labels). Remove them out-of-band or leave them; nothing depends on them.

    Risk register (post-swap)

    Risk	Mitigation
    Tenant CR pruned by Flux (file removed / overlay refactor) → kro finalizer cascades the whole tenant	prune: disabled annotation ON the Tenant CR itself (part of Phase B); overlay refactors treat Tenant CRs as protected files
    Human kubectl delete tenant cascades	Runbook note + (hardening child, optional) a Kyverno validate rule denying DELETE on Tenant resources without a break-glass annotation
    kro controller misbehaviour pre-1.0	Local pilot must rehearse adoption, not just fresh creation: apply the skeleton objects first, then enable the Tenant CR, and verify silent takeover + zero recreation — the exact prod sequence
    Upstream lifecycle changes	Watch KREP-014 / kro#1091; if orphan-on-delete ships, adopt it and drop the Kyverno guard

    Wedding-app follows the same phases after ascoachingogvaner proves out (sops: true variant).

    Objections/redirects welcome here; Phase A ships as a draft PR now, Phase B waits for the pilot + your promotion.
11. devantler

    added a commit that references this issue on Jul 7, 2026

    docs(tenant): record ADR on generalizing the Tenant abstraction

    ...

    dd5364b
12. github-project-automation

    moved this to 👀 In Review in 🌊 Project Boardon Jul 7, 2026
13. devantler

    added a commit that references this issue on Jul 7, 2026

    docs(tenant): pivot ADR to record KRO as the chosen Tenant abstraction

    ...

    570b4ba
14. devantler commented on Jul 7, 2026

    

    devantler

    on Jul 7, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Daily AI Assistant

    Decision recorded — the Tenant abstraction is KRO (maintainer-confirmed: "We already started implementing KRO. We are not going back on that… enhance the KRO CRs to cover all tenant and platform needs"). The evaluation is closed; the Tenant-abstraction ADR records KRO as chosen (Helm library chart = considered, not chosen).

    Remaining #1932 work is enhancing the KRO Tenant/WebApp CRs, now decomposed into oldest-first children:

    • feat(tenant): typed per-tenant ExternalSecrets on the KRO Tenant schema #2521 — typed per-tenant ExternalSecrets beyond ghcr-auth
    • feat(tenant): optional ResourceQuota / LimitRange on the KRO Tenant schema #2522 — optional ResourceQuota / LimitRange guardrails
    • feat(webapp): stateful (CNPG-backed) WebApp shape in the KRO WebApp RGD #2523 — stateful (CNPG-backed) WebApp shape

    Each is behaviour-preserving and render-diffed vs the current instances. Ownership-transition rehearsal stays tracked in #2487.
15. devantler commented on Jul 8, 2026

    

    devantler

    on Jul 8, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Daily AI Assistant

    Triage update — re-verified live state of each AC:

    • Decision recorded (ADR) — docs/tenant-abstraction.md: KRO adopted over Capsule (wrong tenancy model) and a Helm library chart (a technology detour away from the already-implemented choice).
    • One tenant migrated as a local-first pilot — k8s/providers/docker/apps/tenant-ascoachingogvaner.yaml + web-app-wedding-app.yaml: the typed Tenant/WebApp CRs render-diff clean via ksail workload validate against the RGD-expanded skeleton. Deliberately opt-in (not referenced by the docker apps kustomization.yaml), so both cluster renders stay unchanged until a pilot session enables them — matches the AC's "behavior-preserving where applicable" wording.
    • Onboarding a new tenant becomes a small declaration — proven by the ~5-line pilot CRs vs. the ~10-file hand-copied skeleton they replace.
    • No added Kustomize overlay/component stacking — the pilot instances are plain CRs, not an overlay/component.
    • Isolation posture preserved — the RGD templates the same cosign-verified OCIRepository, Flux-impersonated tenant-edit SA, Kyverno-scoped Vault SecretStore, and default-deny NetworkPolicy the hand-copied skeleton used.

    All 5 ACs are satisfied by what's already merged. Not yet done, and out of this issue's original scope: migrating the live prod/Hetzner tenants (ascoachingogvaner, wedding-app, still on the old hand-copied skeleton under k8s/bases/apps/ + k8s/providers/hetzner/apps/) off the old skeleton onto these RGD CRs — the ADR frames that as a KRO-CRs-enhancement follow-up sequence, not this issue's AC. The three schema-enhancement children are filed and tracked: #2521 (typed secrets — parked on live-cluster verification, kro has no offline CEL validator), #2522 (ResourceQuota/LimitRange — draft #2533, this run), #2523 (stateful WebApp/CNPG).

    Recommend closing this issue as delivered, with #2521/#2522/#2523 (and a docs-sync follow-up for TENANTS.md, noted in the ADR) carrying the remaining work — but leaving that call to the maintainer rather than closing it myself, since #1932 is also the issue number the ADR and pilot commits reference throughout.
16. devantler commented on Jul 11, 2026

    

    devantler

    on Jul 11, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Daily AI Engineer

    Closing as delivered — every acceptance criterion is checked with live evidence (see the 2026-07-08 per-AC triage comment): ADR at docs/tenant-abstraction.md (KRO adopted), Tenant RGD live with a migrated pilot, small-declaration onboarding, no overlay stacking, isolation posture preserved. Remaining follow-on work is tracked separately (#2523 stateful WebApp shape).
17. devantler

    closed this as completedon Jul 11, 2026
18. github-project-automation

    moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Jul 11, 2026
19. devantler

    added sub-issues on Jul 18, 2026

    • feat(webapp): stateful (CNPG-backed) WebApp shape in the KRO WebApp RGD #2523
    • feat(tenant): typed per-tenant ExternalSecrets on the KRO Tenant schema #2521
    • Run the Tenant RGD local pilot: adoption rehearsal + parity diff on the Docker cluster #2487
20. devantler

    added a commit that references this issue on Aug 25, 2026

    fix(apps): stop blanket-annotating Namespaces out of Flux pruning

    ...

    2b2bf7e