Skip to content

ci(policy): run complete policy validation in parallel #5069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
devantler merged 6 commits into from
Oct 18, 2026
Merged

ci(policy): run complete policy validation in parallel #5069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ff3ea22
fix(kro): declare namespaced schema scope on the tenant and webapp RGDs
devantler Oct 18, 2026
befe5d8
fix(kro): approve the scoped tenant RGD in the authorization surface
devantler Oct 18, 2026
81fb21e
ci(policy): run complete policy validation in parallel
devantler Oct 18, 2026
d410169
ci: integrate schema-validation prerequisite
devantler Oct 18, 2026
acf88e6
test(ci): identify required authorization aggregator by action
devantler Oct 18, 2026
9c1bdc7
test(ci): track policy result regression as executable
devantler Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
219 changes: 135 additions & 84 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,11 @@ jobs:
go test ./scripts/validate-feature-flags
go run ./scripts/validate-feature-flags

- name: ⚖️ Validate policy-job result boundary
run: |
shellcheck scripts/tests/test-policy-ci-job.sh
bash scripts/tests/test-policy-ci-job.sh

- name: 🛡️ Validate preserved policy failure actions
run: |
go test ./scripts/validate-policy-actions
Expand Down Expand Up @@ -669,6 +674,7 @@ jobs:
- 'scripts/tests/test-arc-controller-availability.sh'
- 'scripts/validate-kyverno-fixture-evaluation.sh'
- 'scripts/tests/test-validate-kyverno-fixture-evaluation.sh'
- 'scripts/tests/test-policy-ci-job.sh'
# Every kyverno fixture, so a change to one alone still runs kyverno test
# and the fixture guard.
- 'tests/**'
Expand Down Expand Up @@ -1511,6 +1517,118 @@ jobs:
timeout-minutes: 10
run: ksail --config ksail.prod.yaml workload validate

validate-cluster-policies:
name: ⚖️ Validate Cluster Policies
needs: [changes]
if: github.event_name == 'pull_request' && needs.changes.outputs.k8s == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: 📑 Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: ⚙️ Setup Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod
cache: false

- name: 📌 Install the reviewed YAML parser
env:
GOBIN: ${{ runner.temp }}/platform-validation-tools
run: |
go install github.com/mikefarah/yq/v4@v4.54.1
printf '%s\n' "$GOBIN" >> "$GITHUB_PATH"
PATH="$GOBIN:$PATH" yq --version

- name: ⚖️ Read Kyverno CLI version
id: kyverno-version
run: bash .github/scripts/kyverno-version.sh

- name: ⚖️ Install Kyverno CLI
uses: kyverno/action-install-cli@fcee92fca5c883169ef9927acf543e0b5fc58289 # v0.2.0
with:
release: ${{ steps.kyverno-version.outputs.release }}

- name: ⚖️ Validate cluster policies
# Evaluate the complete policy census and its negative controls offline.
# No cluster access or secrets are required.
run: |
kyverno test ./tests
shellcheck scripts/validate-kyverno-fixture-evaluation.sh scripts/tests/test-validate-kyverno-fixture-evaluation.sh
bash scripts/tests/test-validate-kyverno-fixture-evaluation.sh
bash scripts/validate-kyverno-fixture-evaluation.sh tests
shellcheck scripts/tests/test-hcloud-volume-priority.sh
bash scripts/tests/test-hcloud-volume-priority.sh
shellcheck scripts/tests/test-tetragon-agent-priority.sh
bash scripts/tests/test-tetragon-agent-priority.sh
shellcheck scripts/tests/test-audit-privileged-rbac.sh
bash scripts/tests/test-audit-privileged-rbac.sh
shellcheck scripts/tests/test-arc-controller-availability.sh
bash scripts/tests/test-arc-controller-availability.sh
shellcheck scripts/tests/test-restrict-tenant-secret-stores.sh
bash scripts/tests/test-restrict-tenant-secret-stores.sh
shellcheck scripts/tests/test-restrict-tenant-network-policies.sh
bash scripts/tests/test-restrict-tenant-network-policies.sh
shellcheck .github/scripts/fetch-cilium-deny-candidate.sh scripts/prove-cilium-default-deny.sh scripts/tests/test-fetch-cilium-deny-candidate.sh scripts/tests/test-prove-cilium-default-deny.sh
bash scripts/tests/test-fetch-cilium-deny-candidate.sh
bash scripts/tests/test-prove-cilium-default-deny.sh
shellcheck scripts/tests/test-add-default-deny-shape.sh
bash scripts/tests/test-add-default-deny-shape.sh
shellcheck scripts/tests/test-restrict-homepage-service-groups.sh
bash scripts/tests/test-restrict-homepage-service-groups.sh
shellcheck scripts/tests/test-kyverno-umami-mutation-rbac.sh
bash scripts/tests/test-kyverno-umami-mutation-rbac.sh
shellcheck scripts/tests/test-kyverno-github-team-report-rbac.sh
bash scripts/tests/test-kyverno-github-team-report-rbac.sh
shellcheck scripts/tests/test-tenant-route-hostname-boundary.sh
bash scripts/tests/test-tenant-route-hostname-boundary.sh
shellcheck scripts/tests/test-actual-budget-auth-route.sh
bash scripts/tests/test-actual-budget-auth-route.sh
shellcheck scripts/tests/test-umami-sso-dashboard-route.sh
bash scripts/tests/test-umami-sso-dashboard-route.sh
shellcheck scripts/tests/test-data-product-controller-trial.sh
bash scripts/tests/test-data-product-controller-trial.sh
shellcheck scripts/tests/test-data-product-ui-host.sh
bash scripts/tests/test-data-product-ui-host.sh
shellcheck scripts/verify-data-product-ui-rollout.sh scripts/tests/test-data-product-ui-rollout.sh
bash scripts/tests/test-data-product-ui-rollout.sh
shellcheck scripts/tests/test-data-product-runtime-digests.sh
bash scripts/tests/test-data-product-runtime-digests.sh
shellcheck scripts/tests/test-oauth2-proxy-sso-redirects.sh
bash scripts/tests/test-oauth2-proxy-sso-redirects.sh
shellcheck scripts/tests/test-umami-tracking-probe.sh
bash scripts/tests/test-umami-tracking-probe.sh
shellcheck scripts/verify-umami-tracking.sh
shellcheck scripts/tests/test-provider-drc-pod-seccomp.sh
bash scripts/tests/test-provider-drc-pod-seccomp.sh
shellcheck scripts/tests/test-add-baseline-context.sh
bash scripts/tests/test-add-baseline-context.sh
shellcheck scripts/tests/test-label-baseline-workers.sh
bash scripts/tests/test-label-baseline-workers.sh
shellcheck scripts/tests/test-longhorn-registrar-prestop-policy.sh
bash scripts/tests/test-longhorn-registrar-prestop-policy.sh
shellcheck scripts/tests/test-csi-sidecar-baseline-context.sh
bash scripts/tests/test-csi-sidecar-baseline-context.sh
shellcheck scripts/tests/test-longhorn-csi-plugin-baseline-context.sh
bash scripts/tests/test-longhorn-csi-plugin-baseline-context.sh
shellcheck scripts/tests/test-longhorn-engine-image-baseline-context.sh
bash scripts/tests/test-longhorn-engine-image-baseline-context.sh
shellcheck scripts/tests/test-hcloud-baseline-worker-affinity.sh
bash scripts/tests/test-hcloud-baseline-worker-affinity.sh
shellcheck scripts/guard-longhorn-ui-baseline-context.sh scripts/tests/test-guard-longhorn-ui-baseline-context.sh scripts/tests/test-longhorn-ui-baseline-context.sh
bash scripts/tests/test-guard-longhorn-ui-baseline-context.sh
bash scripts/tests/test-longhorn-ui-baseline-context.sh
shellcheck scripts/tests/test-cilium-hubble-baseline-context.sh
bash scripts/tests/test-cilium-hubble-baseline-context.sh
shellcheck scripts/guard-coroot-baseline-context.sh scripts/tests/test-guard-coroot-baseline-context.sh
bash scripts/tests/test-guard-coroot-baseline-context.sh
shellcheck scripts/admit-coroot-baseline-context.sh scripts/tests/test-admit-coroot-baseline-context.sh
bash scripts/tests/test-admit-coroot-baseline-context.sh

validate:
name: 🧪 Validate Manifests
needs: [changes]
Expand Down Expand Up @@ -1848,90 +1966,6 @@ jobs:
shellcheck scripts/tests/test-vault-snapshot-group-readable.sh
bash scripts/tests/test-vault-snapshot-group-readable.sh

- name: ⚖️ Validate cluster policies
if: needs.changes.outputs.k8s == 'true'
# Runs the kyverno test fixtures under tests/. Until now nothing
# executed them, so tests/validate-replica-floor had never actually run
# in CI — a policy regression could ship green. No cluster or secrets
# required; the CLI evaluates the committed ClusterPolicy files offline.
# kyverno test counts a row whose rule never evaluated the resource
# (REASON=Excluded) as passing and silently drops a declared resource
# it never loaded, so the fixture guard fails a pass/fail row it never
# evaluated, a rule name no loaded policy defines, and a declared
# resource with no row.
run: |
kyverno test ./tests
shellcheck scripts/validate-kyverno-fixture-evaluation.sh scripts/tests/test-validate-kyverno-fixture-evaluation.sh
bash scripts/tests/test-validate-kyverno-fixture-evaluation.sh
bash scripts/validate-kyverno-fixture-evaluation.sh tests
shellcheck scripts/tests/test-hcloud-volume-priority.sh
bash scripts/tests/test-hcloud-volume-priority.sh
shellcheck scripts/tests/test-tetragon-agent-priority.sh
bash scripts/tests/test-tetragon-agent-priority.sh
shellcheck scripts/tests/test-audit-privileged-rbac.sh
bash scripts/tests/test-audit-privileged-rbac.sh
shellcheck scripts/tests/test-arc-controller-availability.sh
bash scripts/tests/test-arc-controller-availability.sh
shellcheck scripts/tests/test-restrict-tenant-secret-stores.sh
bash scripts/tests/test-restrict-tenant-secret-stores.sh
shellcheck scripts/tests/test-restrict-tenant-network-policies.sh
bash scripts/tests/test-restrict-tenant-network-policies.sh
shellcheck .github/scripts/fetch-cilium-deny-candidate.sh scripts/prove-cilium-default-deny.sh scripts/tests/test-fetch-cilium-deny-candidate.sh scripts/tests/test-prove-cilium-default-deny.sh
bash scripts/tests/test-fetch-cilium-deny-candidate.sh
bash scripts/tests/test-prove-cilium-default-deny.sh
shellcheck scripts/tests/test-add-default-deny-shape.sh
bash scripts/tests/test-add-default-deny-shape.sh
shellcheck scripts/tests/test-restrict-homepage-service-groups.sh
bash scripts/tests/test-restrict-homepage-service-groups.sh
shellcheck scripts/tests/test-kyverno-umami-mutation-rbac.sh
bash scripts/tests/test-kyverno-umami-mutation-rbac.sh
shellcheck scripts/tests/test-kyverno-github-team-report-rbac.sh
bash scripts/tests/test-kyverno-github-team-report-rbac.sh
shellcheck scripts/tests/test-tenant-route-hostname-boundary.sh
bash scripts/tests/test-tenant-route-hostname-boundary.sh
shellcheck scripts/tests/test-actual-budget-auth-route.sh
bash scripts/tests/test-actual-budget-auth-route.sh
shellcheck scripts/tests/test-umami-sso-dashboard-route.sh
bash scripts/tests/test-umami-sso-dashboard-route.sh
shellcheck scripts/tests/test-data-product-controller-trial.sh
bash scripts/tests/test-data-product-controller-trial.sh
shellcheck scripts/tests/test-data-product-ui-host.sh
bash scripts/tests/test-data-product-ui-host.sh
shellcheck scripts/verify-data-product-ui-rollout.sh scripts/tests/test-data-product-ui-rollout.sh
bash scripts/tests/test-data-product-ui-rollout.sh
shellcheck scripts/tests/test-data-product-runtime-digests.sh
bash scripts/tests/test-data-product-runtime-digests.sh
shellcheck scripts/tests/test-oauth2-proxy-sso-redirects.sh
bash scripts/tests/test-oauth2-proxy-sso-redirects.sh
shellcheck scripts/tests/test-umami-tracking-probe.sh
bash scripts/tests/test-umami-tracking-probe.sh
shellcheck scripts/verify-umami-tracking.sh
shellcheck scripts/tests/test-provider-drc-pod-seccomp.sh
bash scripts/tests/test-provider-drc-pod-seccomp.sh
shellcheck scripts/tests/test-add-baseline-context.sh
bash scripts/tests/test-add-baseline-context.sh
shellcheck scripts/tests/test-label-baseline-workers.sh
bash scripts/tests/test-label-baseline-workers.sh
shellcheck scripts/tests/test-longhorn-registrar-prestop-policy.sh
bash scripts/tests/test-longhorn-registrar-prestop-policy.sh
shellcheck scripts/tests/test-csi-sidecar-baseline-context.sh
bash scripts/tests/test-csi-sidecar-baseline-context.sh
shellcheck scripts/tests/test-longhorn-csi-plugin-baseline-context.sh
bash scripts/tests/test-longhorn-csi-plugin-baseline-context.sh
shellcheck scripts/tests/test-longhorn-engine-image-baseline-context.sh
bash scripts/tests/test-longhorn-engine-image-baseline-context.sh
shellcheck scripts/tests/test-hcloud-baseline-worker-affinity.sh
bash scripts/tests/test-hcloud-baseline-worker-affinity.sh
shellcheck scripts/guard-longhorn-ui-baseline-context.sh scripts/tests/test-guard-longhorn-ui-baseline-context.sh scripts/tests/test-longhorn-ui-baseline-context.sh
bash scripts/tests/test-guard-longhorn-ui-baseline-context.sh
bash scripts/tests/test-longhorn-ui-baseline-context.sh
shellcheck scripts/tests/test-cilium-hubble-baseline-context.sh
bash scripts/tests/test-cilium-hubble-baseline-context.sh
shellcheck scripts/guard-coroot-baseline-context.sh scripts/tests/test-guard-coroot-baseline-context.sh
bash scripts/tests/test-guard-coroot-baseline-context.sh
shellcheck scripts/admit-coroot-baseline-context.sh scripts/tests/test-admit-coroot-baseline-context.sh
bash scripts/tests/test-admit-coroot-baseline-context.sh

- name: 🛡️ Validate chart-rendered RBAC
if: needs.changes.outputs.k8s == 'true'
# Pulls the charts whose roles the privileged-RBAC policy excludes and applies
Expand Down Expand Up @@ -3899,6 +3933,7 @@ jobs:
validate-matcher-efficacy,
validate-eks-authorization,
validate,
validate-cluster-policies,
validate-floating-image-tags,
validate-helm-post-renderers,
validate-pdb-selectors,
Expand All @@ -3922,6 +3957,7 @@ jobs:
${{ needs.validate-matcher-efficacy.result }}
${{ needs.validate-eks-authorization.result }}
${{ needs.validate.result }}
${{ needs.validate-cluster-policies.result }}
${{ needs.validate-floating-image-tags.result }}
${{ needs.validate-helm-post-renderers.result }}
${{ needs.validate-pdb-selectors.result }}
Expand All @@ -3931,3 +3967,18 @@ jobs:
${{ needs.validate-rgd-templates-merge-group.result }}
${{ needs.validate-ghcr-fanout-merge-group.result }}
${{ needs.deploy-prod.result }}

- name: ⚖️ Require selected policy validation
env:
EVENT_NAME: ${{ github.event_name }}
SELECTED: ${{ needs.changes.outputs.k8s }}
POLICY_RESULT: ${{ needs.validate-cluster-policies.result }}
run: |
if [ "$EVENT_NAME" = pull_request ] && [ "$SELECTED" != true ] && [ "$SELECTED" != false ]; then
printf 'Cannot determine cluster-policy validation selection: %s\n' "$SELECTED" >&2
exit 1
fi
if [ "$EVENT_NAME" = pull_request ] && [ "$SELECTED" = true ] && [ "$POLICY_RESULT" != success ]; then
printf 'Selected cluster-policy validation did not succeed: %s\n' "$POLICY_RESULT" >&2
exit 1
fi
2 changes: 2 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ Timing evidence alone does not establish Coroot request attribution or recovery.

CI runs **static manifest validation** on PRs that touch k8s-related paths (`k8s/**`, `ksail*.yaml`, `.sops.yaml`, `talos*/**`, the naming configuration `.github/manifest-naming.yaml`, the validation scripts `scripts/validate-embedded-json/` / `scripts/validate-embedded-json.sh` / `scripts/generate-kubescape-exceptions/`, or `ci.yaml` — the authoritative list is the `k8s` filter in `.github/workflows/ci.yaml`) — the `validate` job in `.github/workflows/ci.yaml` first json-parses every registered embedded-JSON ConfigMap key via [`scripts/validate-embedded-json.sh`](scripts/validate-embedded-json.sh) (keys listed in the Go command's `registeredKeys` or ending in `.json` — schema validation treats such blobs as opaque strings, so a stray comma would otherwise ship silently; run `bash scripts/validate-embedded-json.sh` locally when touching one, from any directory via the script's absolute path), then runs `ksail workload validate` for both the local and prod overlays plus a Kubescape scan (`scripts/generate-kubescape-exceptions` converts the `ClusterSecurityException` CRs into Kubescape's exceptions format, then `ksail workload scan --framework nsa,mitre --exceptions <generated> --compliance-threshold <floor>` gates on the combined score — the exact floor lives in `ci.yaml`). It is fast, needs no secrets (so it runs on fork PRs too), and starts no cluster. PRs touching `talos/**` or `talos-local/**` additionally run the `validate-talos` job: it renders the machine config with every patch applied (placeholder values stand in for env-expanded secrets like `${WG_SERVER_PRIVATE_KEY}`) and `talosctl validate`s the result, so a broken patch or an empty env expansion fails the PR event instead of the merge group's deploy (#2477). There is **no longer a full-cluster system test**: the local Docker cluster is a thin manual test-bed (see [Local Development Cluster](#local-development-cluster)), not a CI prod stand-in.

The complete cluster-policy fixtures and their offline regression controls run concurrently with manifest validation in the `validate-cluster-policies` pull-request job, under the same `k8s` path selection. The required aggregate waits for this job and rejects an unsuccessful or unexpectedly skipped selected family. Run `bash scripts/tests/test-policy-ci-job.sh` to exercise that result boundary; policy evaluation uses the shared Kyverno CLI version from `.github/scripts/kyverno-version.sh` and requires no cluster or secrets.

The Talos gate renders both roles, applies the reviewed KSail post-installer kernel-argument fold, then validates both. Production's extension path clears `extraKernelArgs` and pins the UKI command line; the local path without extensions is unchanged. The unconditional pin check and real offline positive/invalid-patch controls are documented in [`scripts/reconcile-talos-kernel-args/README.md`](scripts/reconcile-talos-kernel-args/README.md). The pin check compares the pinned KSail release's fold inputs by content with the audited sets, so a KSail bump needs a new source audit of this fold only when one of those inputs changed.

The scan is a **hard gate**: it fails the PR if the combined compliance score drops below the threshold, so new findings must be fixed or justified before merge. It evaluates **two** frameworks, NSA-CISA and MITRE ATT&CK, because NSA-CISA evaluates 20 controls in total but only 17 of the 76 named by the `ClusterSecurityException` CRs — the RBAC controls those CRs exist to govern were outside the gate entirely. Three non-obvious limits:
Expand Down
1 change: 1 addition & 0 deletions k8s/bases/infrastructure/resource-graph-definitions/tenant/resource-graph-definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ spec:
schema:
apiVersion: v1alpha1
kind: Tenant
scope: Namespaced
# Nested types for the optional multi-tenancy guardrails (#2522). Each
# carries its own `enabled` toggle rather than relying on CEL `has()`
# presence-detection on the parent object — this mirrors the RGD's own
Expand Down
1 change: 1 addition & 0 deletions k8s/bases/infrastructure/resource-graph-definitions/webapp/resource-graph-definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ spec:
schema:
apiVersion: v1alpha1
kind: WebApp
scope: Namespaced
spec:
# App name — used verbatim as the Deployment/Service/HTTPRoute name AND as
# the target namespace (the tenant namespace the Tenant archetype created).
Expand Down
Loading
Loading