Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/tenant-abstraction.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,21 @@ a new production revocation mechanism.
and `kubectl`-native with CRD-schema validation; no Helm/overlay stacking; the isolation posture is
templated verbatim.

### Public business-site staging hold

The platform installs `business-site-staging-hold` before enabling the public
business application. Tenant writers cannot create, update or delete that named
Cilium policy in `business-site`, including through reconciliation or pruning.
The existing trusted platform-controller carve-outs remain; a controller name
in a tenant namespace does not grant the same authority. Other policy names and
the same name in other namespaces retain their ordinary admission rules.

Reserved policy names do not inherit an exception for an already noncompliant
old object: platform creation is not permission for a tenant's later update.
This ownership boundary is separate from the hold's actual traffic effect.
Routing, projection and live hold/serve/isolate/recover acceptance remain under
#5247 before exposure; the source-only baseline does not enable an application.

### Public business-site configuration identity

The maintained `vault-config` Job declares the `business-site` Kubernetes auth
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,12 @@
# Without this a tenant would simply PATCH the generated floor into a permissive
# body, leaving the object present and the namespace open.
#
# Scope and carve-outs: only ksail-generated tenant namespaces (labelled
# The business-site staging hold additionally reserves one exact name in its
# exact namespace, including DELETE, without depending on namespace labels.
# Reserved names refuse tenant UPDATE even when an old platform-authored body
# already fails that rule for the tenant identity.
#
# General scope and carve-outs: only generated tenant namespaces (labelled
# app.kubernetes.io/managed-by: ksail), so the 50+ platform CiliumNetworkPolicies
# are untouched. Two applying identities are excluded — flux-system's
# kustomize-controller, which applies what the PLATFORM ships into a tenant
Expand Down Expand Up @@ -510,6 +515,17 @@ spec:
- key: app.kubernetes.io/managed-by
operator: In
values: [ksail, kro]
# Install the hold before enabling the application; tenant SSA/prune
# must not remove that platform control. Scope its name to this one
# namespace, independently of mutable namespace labels.
- resources:
kinds:
- CiliumNetworkPolicy
names:
- business-site-staging-hold
namespaces:
- business-site
operations: [CREATE, UPDATE, DELETE]
exclude:
any:
- subjects:
Expand All @@ -525,11 +541,16 @@ spec:
# if a future Kyverno stops honouring that field, a rule with no action
# of its own defaults to Audit and these violations would be ADMITTED.
failureAction: Enforce
# A platform-created reserved object already violates this rule when
# evaluated as a tenant. Do not exempt later tenant updates merely
# because that old object would also have been refused for that actor.
allowExistingViolations: false
message: >-
`default-deny`, `allow-dns` and `allow-cnpg-operator` are generated and
re-synchronised by the
platform (cluster-policies/best-practices/add-default-deny.yaml) and
are reserved names. Add your allow-list as a separate policy — Cilium
are reserved names. `business-site-staging-hold` in `business-site`
is also a platform-owned rollout control. Add your allow-list as a separate policy — Cilium
unions allow rules, so a policy of your own is additive and does not
need to replace the floor.
deny: {}
Expand Down
63 changes: 60 additions & 3 deletions scripts/tests/test-restrict-tenant-network-policies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,22 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(cd "${script_dir}/../.." && pwd)"
policy="${repo_root}/k8s/bases/infrastructure/cluster-policies/best-practices/restrict-tenant-network-policies.yaml"
tests="${repo_root}/tests/restrict-tenant-network-policies"
output_file="$(mktemp)"
trap 'rm -f "${output_file}"' EXIT
work_dir="$(mktemp -d)"
output_file="${work_dir}/output"
trap 'rm -rf -- "${work_dir}"' EXIT

# `kyverno test` treats a rule that matches nothing as Excluded, and an Excluded
# row satisfies `result: fail` too — so the declarative test can pass vacuously
# if a rule silently stops matching. Exercise the policy directly as a second
# gate, asserting the exact allow/deny counts rather than an exit code.
apply() {
local resources="$1" values="$2" userinfo="$3"
apply_status=0
kyverno apply "${policy}" \
--resource "${resources}" \
--values-file "${values}" \
--userinfo "${userinfo}" \
--remove-color >"${output_file}" 2>&1 || true
--remove-color >"${output_file}" 2>&1 || apply_status=$?
}

expect_summary() {
Expand Down Expand Up @@ -114,5 +116,60 @@ if [[ "${top_level}" != "absent" ]]; then
exit 1
fi

# The public staging hold is controlled by the platform before application
# reconciliation is enabled. Check the real policy and committed hold, retaining
# all ordinary tenant/floor fixtures above. A nonmatching rule cannot satisfy an
# exact one-denial summary; permitted actors must also return a successful exit.
hold="${repo_root}/k8s/providers/hetzner/apps/business-site/cilium-network-policy-staging-hold.yaml"
yq '.namespaceSelector += [{"name":"business-site","labels":{"app.kubernetes.io/managed-by":"kro"}}]' \
"${tests}/values.yaml" >"${work_dir}/hold-values.yaml"
cat >"${work_dir}/hold-tenant.yaml" <<'USER'
apiVersion: cli.kyverno.io/v1alpha1
kind: UserInfo
metadata:
name: fictional-business-tenant
userInfo:
username: system:serviceaccount:business-site:business-site
groups:
- system:serviceaccounts
- system:serviceaccounts:business-site
- system:authenticated
USER
for operation in CREATE UPDATE DELETE; do
yq ".globalValues.\"request.operation\" = \"${operation}\"" \
"${work_dir}/hold-values.yaml" >"${work_dir}/operation.yaml"
apply "${hold}" "${work_dir}/operation.yaml" "${work_dir}/hold-tenant.yaml"
expect_summary "pass: 4, fail: 1, warn: 0, error: 0, skip: 0" "tenant staging hold ${operation}"
[[ "${apply_status}" -eq 1 ]] || { echo "staging denial did not fail admission" >&2; exit 1; }
for author in platform-author kyverno-author; do
apply "${hold}" "${work_dir}/operation.yaml" "${tests}/${author}/user-info.yaml"
expect_summary "pass: 0, fail: 0, warn: 0, error: 0, skip: 0" "platform staging hold ${operation}"
[[ "${apply_status}" -eq 0 ]] || { echo "trusted platform operation failed" >&2; exit 1; }
done
done
# A controller name in a tenant namespace does not inherit the platform carve-out.
yq '.userInfo.username = "system:serviceaccount:business-site:kustomize-controller"' \
"${work_dir}/hold-tenant.yaml" >"${work_dir}/spoofed-controller.yaml"
apply "${hold}" "${work_dir}/hold-values.yaml" "${work_dir}/spoofed-controller.yaml"
expect_summary "pass: 4, fail: 1, warn: 0, error: 0, skip: 0" "tenant cannot borrow a controller name"
[[ "${apply_status}" -eq 1 ]] || exit 1
# Reserve only this name in this namespace; other ordinary policies stay usable.
for change in '.metadata.namespace = "other-tenant"' '.metadata.name = "tenant-allow-gateway"'; do
yq "${change}" "${hold}" >"${work_dir}/ordinary-policy.yaml"
apply "${work_dir}/ordinary-policy.yaml" "${work_dir}/hold-values.yaml" "${work_dir}/hold-tenant.yaml"
expect_summary "pass: 4, fail: 0, warn: 0, error: 0, skip: 0" "ordinary tenant policy"
[[ "${apply_status}" -eq 0 ]] || exit 1
done
# The same reserved-owner rule protects the generated floor. An old body that
# is invalid for a tenant must not exempt that tenant's subsequent UPDATE.
yq '.globalValues."request.operation" = "UPDATE"' \
"${tests}/kyverno-author/values.yaml" >"${work_dir}/floor-update.yaml"
apply "${tests}/kyverno-author/resources.yaml" "${work_dir}/floor-update.yaml" "${tests}/user-info.yaml"
expect_summary "pass: 8, fail: 3, warn: 0, error: 0, skip: 4" "reserved floor UPDATE ownership"
[[ "${apply_status}" -eq 1 ]] || exit 1
apply "${tests}/kyverno-author/resources.yaml" "${work_dir}/floor-update.yaml" "${tests}/kyverno-author/user-info.yaml"
expect_summary "pass: 0, fail: 0, warn: 0, error: 0, skip: 0" "generated floor controller UPDATE"
[[ "${apply_status}" -eq 0 ]] || exit 1

echo "Tenant CiliumNetworkPolicy policy enforced the expected boundary, and both"
echo "platform carve-outs were proven to depend on the applying identity."
Loading