Repository navigation
Generalize the WebApp abstraction (ascoachingogvaner, wedding-app) #1933
Description
Summary
Generalize a WebApp archetype — a reusable definition of a containerized web application (workload + route + TLS + scaling + secrets) — so the tenant web apps (ascoachingogvaner, wedding-app) and future ones become a small per-app declaration instead of copied workload boilerplate.
Sibling to the Tenant archetype issue: a Tenant sets up the namespace/isolation/sync; a WebApp defines the application that runs inside it.
Context
ascoachingogvaner (coaching site) and wedding-app are simple web apps. Their workload manifests (Deployment/container or chart, Service, Gateway-API HTTPRoute, optional scaling, optional secrets) live in each tenant's own OCI artifact (synced via the platform-provisioned OCIRepository + Flux Kustomization), not in this repo — but they follow the same web-app shape and will multiply as more tenants onboard. The shape is near-identical: image/chart + host + TLS + a route to the central gateway + (optionally) auth and a secret.
Goal & constraint
One WebApp abstraction → a per-app instance specifying just {image|chart, host, tls, scaling, auth?, secretRefs?}. No more Kustomize overlay/component stacking (the readability reason we're choosing an archetype).
Options to evaluate
- Helm app/library chart — recommended start. A
webappchart (named templates for Namespace?/Deployment/Service/HTTPRoute/optionalHTTPScaledObject/optional ExternalSecret) consumed via a HelmRelease or referenced from the tenant artifact.values.schema.jsongives light typing; stays inside the Flux→HelmRelease model; zero new controllers; no pre-1.0 risk. - KRO
WebAppRGD. Typed,kubectl-native, conditional children viaincludeWhen(KEDA/OIDC). But KRO is pre-1.0 (v1alpha1) and adds an alpha controller at theinfrastructure-controllerschokepoint — see the KRO assessment. Better revisited once KRO hits beta/1.0.
Open design question — placement
Tenant workloads live in tenant-owned OCI artifacts. Decide whether the WebApp archetype is:
- a platform-published chart/RGD that tenants reference from their artifact (platform owns the template, tenant owns the values), or
- defined per-tenant.
The platform-published option keeps the archetype DRY across tenants and lets us evolve it centrally; it should be the default unless there's a reason to fork per tenant.
Platform constraints any solution must respect
HTTPRouteattaches to the central CiliumGatewayinkube-system(reference it, don't recreate it).- Optional KEDA HTTP scale-to-zero mirrors the existing 5-point recipe (interceptor repoint + netpol + initial replicas 0).
- Kyverno mutate policies apply to the rendered pods (security context, anti-affinity, spread, etc.) — the archetype shouldn't fight them.
- Secrets via ESO + SOPS, never inline.
enforce-flux-best-practices(Enforce) requiresinterval/timeout/retries on any templatedHelmRelease.ksail workload validatebuilds every kustomization standalone (skip-kinds for any new CRD-backed instance).
Acceptance criteria
- Decision recorded (ADR, or reuse the Tenant-archetype decision) on Helm chart vs KRO, including archetype placement (platform-published vs per-tenant).
- One app (
ascoachingogvanerorwedding-app) migrated as a pilot; behavior-preserving where applicable. - A new web app becomes a small declaration.
- No added Kustomize overlay/component stacking.
Related
- Composes with the Tenant archetype — see the sibling issue.
- Incremental DRY already shipped: refactor(infrastructure): hetzner tracks controller aggregate, not a hand list #1927, refactor(infrastructure): centralize HelmRelease Flux remediation defaults #1928, refactor(infrastructure): use path-only kustomize patches where self-identifying #1929, style(kustomize): add yaml document-start markers + normalize list indent #1931.
🤖 Filed from an architecture discussion with Claude Code. Source: kro.run.
Activity
devantler commentedon Jun 9, 2026
Sibling archetype (composes with this one): #1932
[-]Generalize the WebApp archetype (ascoachingogvaner, wedding-app)[/-][+]Generalize the WebApp abstraction (ascoachingogvaner, wedding-app)[/+]devantler commentedon Jun 18, 2026
ADR (Proposed) — WebApp archetype: Helm chart vs KRO, and where it lives
Delivering acceptance-criterion 1 ("Decision recorded on Helm chart vs KRO, including archetype placement (platform-published vs per-tenant)") as a Proposed record for your ratification. Status flips to Accepted when you promote it (or redirect). This composes with — and deliberately reuses the mechanism of — the Tenant archetype decision (#1932, Helm chart recommended).
The WebApp question has a twist the Tenant question doesn't: a Tenant registration skeleton lives in this repo (
k8s/bases/apps/<tenant>/), but a WebApp workload lives in the tenant's own OCI artifact (oci://ghcr.io/devantler-tech/<tenant>/manifests, cosign-verified, synced by the platform-provisionedOCIRepository+Kustomization— verified ink8s/bases/apps/{ascoachingogvaner,wedding-app}/sync.yaml). So "where the archetype lives" is the load-bearing decision here, not an afterthought.Decision drivers (weighted by this platform's reality)
infrastructure-controllers— the recurring wedge point (same driver 1 as Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932).helm-controlleris already cluster-wide; adding an operator here is the highest-cost move.{image|chart, host, tls, scaling?, auth?, secretRefs?}) is near-identical acrossascoachingogvaner,wedding-app, and every future webapp; the template must live in one place we can version, not be re-copied into each tenant repo.Gatewaynamedplatforminkube-system(HTTPRoutes attach viaparentRefs:[{name: platform, namespace: kube-system, sectionName: https}], e.g.k8s/bases/apps/actual-budget/httproute.yaml); the KEDA HTTP scale-to-zero recipe; Kyverno mutate policies on rendered pods; ESO+SOPS secrets (never inline);enforce-flux-best-practices(Kyverno Enforce);ksail workload validatestandalone-build parity.Evaluation
KRO
WebAppRGD (build, typed) — REJECTED (same as #1932). Pre-1.0v1alpha1; lands an alpha controller + dynamic CRDs in the wedge-proneinfrastructure-controllerslayer (driver 1 ✗). The typed/kubectl-native +includeWhenconditional-children upside is real but doesn't pay for the layer risk at 2-app scale. Revisit post-1.0 with a concrete typed-API need — the conditional KEDA/auth children are exactly KRO's sweet spot, so it's the right thing to re-evaluate later.Helm chart — RECOMMENDED, as a platform-published OCI chart. Resolving the open placement question explicitly:
Placement = platform-published, NOT per-tenant. Publish a
webappapplication chart as a cosign-signed OCI chart toghcr.io/devantler-tech/charts/webapp(mirroring the existing per-tenantpublish-appOIDC+cosign trust model). Each tenant's artifact then carries a thinHelmRelease(orOCIRepository→HelmReleasepair) pointingchart.spec.sourceRefat that chart with a per-app values block. Platform owns the template; tenant owns the values — the issue's own default, and the only option that gives DRY + central evolution (driver 2 ✓) without forking the template per tenant (driver 3 ✓). Per-tenant charts are rejected: they re-fork the very boilerplate we're removing.No new controller (driver 1 ✓):
helm-controlleralready reconciles HelmReleases platform-wide. The tenant'sKustomizationsimply emits aHelmReleaseinstead of raw manifests; nothing new entersinfrastructure-controllers.One indirection, not overlay stacking (acceptance ✓): chart → values, flatter than overlay-on-overlay, and Helm is already in the platform vocabulary.
Templates the whole webapp shape (driver 2 ✓): Deployment, Service,
HTTPRoute(with the central-GatewayparentRefbaked in), and — behind value conditionals — the KEDA scale-to-zero set and an ExternalSecret. A new webapp becomes a ~6-line values block.Constraints honored verbatim (driver 4 ✓):
HTTPRoute.parentRefshard-wired toplatform/kube-system/https; the app only supplieshost.values.scaleToZero, default off): templates the verified 5-point recipe — (1) initialreplicas, (2)HTTPScaledObject(min/max,scaledownPeriod, request-rate metric), (3)HTTPRoutebackend repointed tokeda-add-ons-http-interceptor-proxy.keda.svc:8080, (4)CiliumNetworkPolicyingress from thekedanamespace, (5) drain-safePodDisruptionBudget. Honest caveat: the interceptor lives in thekedanamespace and the cross-namespaceReferenceGrant(k8s/bases/infrastructure/controllers/keda-http-add-on/reference-grant.yaml) enumerates specific app namespaces — so enabling scale-to-zero for a new tenant namespace needs a one-line platform-sideReferenceGrantaddition (it can't be templated from the tenant artifact). The chart should emit a clear comment / values note pointing at that step.postRenderers+ mutate combo.values.secretRefsreference an ESOExternalSecret/SecretStore(and SOPS for thewedding-appcase) — never inlined.enforce-flux-best-practices(Enforce): the generatedHelmReleasemust templatespec.interval,spec.timeout,spec.install.remediation.retries,spec.upgrade.remediation.retries(≥1/-1), andspec.upgrade.remediation.remediateLastFailure: true(bounded, known — values defaults cover it).Honest caveats specific to WebApp (beyond Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932's):
Chart.yamlin-repo). So this adds a new chart-build-and-publish pipeline (apublish-chartworkflow + cosign signing), not just a chart. Bounded and reuses the existing OIDC/cosign publish pattern, but it's net-new CI surface — call it out as part of the cost.HelmRelease. Verify prune / release-ownership semantics on removal differ acceptably from a rawKustomization(test in the pilot).ksail workload validatebuilds each kustomization standalone → needs--skip-kindsforHTTPScaledObject(and any CRD-backed instance) when scale-to-zero is templated, same as existing CRD-backed apps.Recommendation
A platform-published, cosign-signed
webappOCI application chart, referenced by a thin per-tenantHelmReleasethat lives in the tenant artifact. Adds no controller to the fragile layer, gives DRY with central evolution, keeps the tenant-owned-artifact boundary intact, and reuses the Tenant-archetype Helm decision (#1932) so there's one archetype mechanism, not two. KRO deferred (post-1.0). This is consistent with #1932's closing note: a Tenant hosts WebApp workloads, so WebApp is the same Flux→HelmRelease model, one layer down.Consequences / pilot (remaining acceptance criteria)
webappapplication chart:Deployment+Service,HTTPRoute(central-Gateway parentRef), conditional KEDA 5-point set behindvalues.scaleToZero, conditionalExternalSecretbehindvalues.secretRefs;values.schema.jsonfor light typing; chart templates the Flux-best-practiceHelmReleasefields.publish-chartworkflow (OIDC + cosign) publishing toghcr.io/devantler-tech/charts/webapp.ascoachingogvaner(no SOPS, no scale-to-zero — the simplest case) local-first: tenant artifact emits a thinHelmRelease; render-diff vs current manifests; confirmksail workload validateparity.wedding-appexercises the SOPS-secret + (optional) scale-to-zero conditionals, including the platform-sideReferenceGrantaddition if scale-to-zero is wanted.docs/TENANTS.md.Sequencing with #1932
The Tenant chart (registration skeleton, in-repo) and the WebApp chart (workload, published OCI) are independent and complementary — either can pilot first. Suggested order: land the Tenant chart first (it's lower-risk, in-repo, no new publish pipeline), then WebApp, since WebApp's only net-new infrastructure is the chart-publish workflow.
If you'd rather record this as a committed design doc (e.g.
docs/webapp-archetype.md, matching the topic-doc style — there's still no ADR dir,docs/dris Disaster Recovery), say the word and I'll open a draft PR. Build-vs-buy and the publish-pipeline cost are your calls; this is the reasoned recommendation to act on.devantler commentedon Jun 18, 2026
ADR update — WebApp archetype: Helm ruled out → CR-based (KRO)
Maintainer decision (direct): "Custom Resource approach — more cloud-native, so Helm charts are ruled out." This supersedes the Helm-chart recommendation in my prior comment here.
The engine decision is shared with the Tenant archetype (#1932) — one mechanism, not two. Full evaluation (Crossplane vs KRO vs extending
ksail-operator, grounded in what's deployed onmain) is in the revised ADR on #1932. Summary as it applies to WebApp:provider-upjet-github(external-API role) + noprovider-kubernetes; composing in-cluster Deployment/Service/HTTPRoute via CrossplaneObjectwrappers is off-label. Keep Crossplane for the GitHub/Hetzner external APIs.ResourceGraphDefinition— recommended: a typed, namespacedWebAppCRD whose instance renders the children directly;includeWhenis a clean fit for this archetype's optional parts:HTTPRoute→ always,parentRefs:[{name: platform, namespace: kube-system, sectionName: https}](the central Cilium Gateway, verified); app supplies onlyhost.includeWhen scaleToZero→ the verified KEDA 5-point recipe: initial replicas,HTTPScaledObject, HTTPRoute backend repointed tokeda-add-ons-http-interceptor-proxy.keda.svc:8080,CiliumNetworkPolicyingress fromkeda, drain-safe PDB. Caveat unchanged: the cross-nsReferenceGrant(.../keda-http-add-on/reference-grant.yaml) enumerates app namespaces → a new tenant's scale-to-zero still needs a one-line platform-side ReferenceGrant addition (can't come from the tenant artifact).includeWhen secretRefs→ExternalSecret(ESO + SOPS forwedding-app); never inlined.ksail-operatorownership is the maximal-ownership alternative (own theWebAppCRD in your operator vs adopt KRO) — see Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932; your B-vs-C call.Placement (the WebApp-specific open question)
The
WebAppCRD/RGD is platform-defined (KRO RGD lives in the platform repo, in the shared layer); a tenant's OCI artifact carries only the smallWebAppinstance (the typed CR). Platform owns the schema/rendering, tenant owns the values — same split as before, now via a CR instead of a chart's values. Works because KRO is in the shared layer (unlike prod-only Crossplane).Pilot
Same as #1932's plan:
ascoachingogvanerfirst (no SOPS/scale-to-zero) — render-diff theWebAppRGD output vs current manifests, confirmksail workload validateparity (add the CRD toskipKinds); thenwedding-appexercises the SOPS + scale-to-zero conditionals + the ReferenceGrant step.I'll open the draft PR (KRO +
Tenant/WebAppRGDs + local-first pilot) once you confirm KRO vs ksail-operator on #1932.devantler commentedon Jun 18, 2026
Decision (maintainer, 2026-06-19): KRO.
The WebApp archetype is settled on KRO (
kro.runResourceGraphDefinition), consistent with the Tenant decision in #1932 (Helm ruled out; Crossplane reserved for external-API/prod-only).Sequencing: this builds on the KRO install + Tenant RGD groundwork in #1932. The WebApp RGD composes the per-app Deployment/Service/HTTPRoute/(optional KEDA scale-to-zero + ExternalSecret) skeleton, with optional children via
includeWhen. Pilot local-first after the Tenant RGD lands.fix(webapp): lead the pilot WebApp CR filename with its Kind + clarif…
devantler commentedon Jul 8, 2026
Same resolution as the sibling Tenant issue (#1932) — the Tenant-abstraction ADR already covers WebApp too: "The
TenantandWebAppRGDs are live" (KRO adopted, Helm rejected as a detour away from the already-implemented choice). The open design question — placement is answered by the same ADR: platform-published RGD (k8s/bases/infrastructure/resource-graph-definitions/webapp/), not per-tenant — everyWebAppCR expands from one central template, matching the "should be the default" preference this issue itself states.Live evidence:
k8s/providers/docker/apps/web-app-wedding-app.yamlis the pilot instance (opt-in, render-diff clean viaksail workload validate), covering the V1 skeleton scope (Deployment + Service + Gateway-APIHTTPRoute). App-specific extensions (secret-backed env, KEDA scale-to-zero, the www→apex redirect, OIDC) are explicitly out of the V1 skeleton per the instance's own comment — tracked as the "cover more app shapes" follow-up, alongside #2523 (stateful/CNPG-backedWebApp).wedding-app(docker provider, opt-in).Same disposition as #1932: recommend closing as delivered, with #2523 (stateful shapes) and the still-open "migrate live prod tenants off the old hand-copied skeleton" work carrying the remainder — leaving the close to the maintainer.
devantler commentedon Jul 11, 2026
Closing as delivered — all acceptance criteria checked with live evidence (2026-07-08 triage comment): the Tenant-abstraction ADR covers WebApp (platform-published RGD at
k8s/bases/infrastructure/resource-graph-definitions/webapp/),web-app-wedding-app.yamlis the migrated pilot, a new web app is a small declaration, no overlay stacking. The 'cover more app shapes' follow-up is tracked as #2523 (stateful/CNPG-backed WebApp).