perf(ci): partition complete offline regressions

AGENTS.md

@@ -128,7 +128,7 @@ Timing evidence alone does not establish Coroot request attribution or recovery.
 
 CI runs **static manifest validation** on PRs that touch k8s-related paths (`k8s/**`, `ksail*.yaml`, `.sops.yaml`, `talos*/**`, the naming configuration `.github/manifest-naming.yaml`, the validation scripts `scripts/validate-embedded-json/` / `scripts/validate-embedded-json.sh` / `scripts/generate-kubescape-exceptions/`, or `ci.yaml` — the authoritative list is the `k8s` filter in `.github/workflows/ci.yaml`) — the `validate` job in `.github/workflows/ci.yaml` first json-parses every registered embedded-JSON ConfigMap key via [`scripts/validate-embedded-json.sh`](scripts/validate-embedded-json.sh) (keys listed in the Go command's `registeredKeys` or ending in `.json` — schema validation treats such blobs as opaque strings, so a stray comma would otherwise ship silently; run `bash scripts/validate-embedded-json.sh` locally when touching one, from any directory via the script's absolute path), then runs `ksail workload validate` for both the local and prod overlays plus a Kubescape scan (`scripts/generate-kubescape-exceptions` converts the `ClusterSecurityException` CRs into Kubescape's exceptions format, then `ksail workload scan --framework nsa,mitre --exceptions <generated> --compliance-threshold <floor>` gates on the combined score — the exact floor lives in `ci.yaml`). It is fast, needs no secrets (so it runs on fork PRs too), and starts no cluster. PRs touching `talos/**` or `talos-local/**` additionally run the `validate-talos` job: it renders the machine config with every patch applied (placeholder values stand in for env-expanded secrets like `${WG_SERVER_PRIVATE_KEY}`) and `talosctl validate`s the result, so a broken patch or an empty env expansion fails the PR event instead of the merge group's deploy (#2477). There is **no longer a full-cluster system test**: the local Docker cluster is a thin manual test-bed (see [Local Development Cluster](#local-development-cluster)), not a CI prod stand-in.
 
-The complete cluster-policy fixtures and their offline regression controls run concurrently with manifest validation in the `validate-cluster-policies` pull-request job, under the same `k8s` path selection. The required aggregate waits for this job and rejects an unsuccessful or unexpectedly skipped selected family. Run `bash scripts/tests/test-policy-ci-job.sh` to exercise that result boundary; policy evaluation uses the shared Kyverno CLI version from `.github/scripts/kyverno-version.sh` and requires no cluster or secrets.
+The complete cluster-policy fixtures and offline regressions run in fixed suites of the `validate-cluster-policies` PR matrix, under the existing `k8s` path selection. At most three suites run concurrently in isolated checkouts; every regression step selects exactly one declared suite. The publication-generator regressions run independently of the unconditional repository guards; required CI and production deployment both wait for the generator. Seven independent manifest regressions run in the read-only `validate-manifest-regressions` PR job with the same pinned tools and inputs, alongside manifest validation. The rollout fixture's independent mutation families use two isolated workers, retaining all cases, assertions and deadlines; foreground cases and shared-input changes wait for them. Required CI refuses an unsuccessful or unexpectedly skipped selected family. Run `bash scripts/tests/test-policy-ci-job.sh` for routing and result-boundary controls and `bash scripts/tests/test-fixture-workers.sh` for worker failure and completion controls; see the [partition decision](docs/adr/ci-regression-partitions.md). Policy evaluation uses the shared Kyverno CLI version from `.github/scripts/kyverno-version.sh` and needs no cluster or secrets.
 
 The Talos gate renders both roles, applies the reviewed KSail post-installer kernel-argument fold, then validates both. Production's extension path clears `extraKernelArgs` and pins the UKI command line; the local path without extensions is unchanged. The unconditional pin check and real offline positive/invalid-patch controls are documented in [`scripts/reconcile-talos-kernel-args/README.md`](scripts/reconcile-talos-kernel-args/README.md). The pin check compares the pinned KSail release's fold inputs by content with the audited sets, so a KSail bump needs a new source audit of this fold only when one of those inputs changed.
 

docs/adr/ci-regression-partitions.md

@@ -0,0 +1,108 @@
+# Partition complete offline CI regressions
+
+## Decision
+
+Run the complete cluster-policy regressions as explicit matrix suites in
+`ci.yaml`. Each suite receives an isolated checkout and the same pinned tools and
+read-only token. At most three suites run concurrently. Fail-fast is disabled, so
+a failed suite does not cancel the other diagnostics.
+
+Cluster policies have fixtures, tenant, services, placement, storage and
+observability suites. Every validation step belongs to exactly one declared
+suite. The publication-generator suite runs in its own required job with the same
+checkout, compiler and native tool setup. The other repository guards retain their
+unconditional steps, including the checks that independently protect CI wiring.
+Regressions that share one shell environment remain in the same step.
+
+Move the independent OpenBao standby, Crossplane egress, native-mail, Kyverno
+review-admission, unscanned-namespace, publisher-pin and Coroot application-health
+regressions out of the serial manifest job into one required PR job. Preserve
+their complete steps, timeouts and Kubernetes-path selection. Copy the pinned
+compiler, YAML parser, Kyverno, KSail and schema-cache restore prerequisites into
+the isolated checkout. This job has only read access to contents; SARIF upload
+remains in the existing manifest job. A distinct candidate cache key restores
+from the existing schema prefix; cached data never substitutes for evaluation.
+
+The existing selection predicates remain authoritative. The required aggregate
+consumes the complete matrix result and refuses failed, cancelled, missing or
+unexpectedly skipped selected validation. It explicitly requires generator
+success. Production deployment waits for both the generator and its existing
+merge-group prerequisites.
+The independent manifest-regression family remains PR-only, like its original
+execution. Required CI accepts success when selected and an intentional skip
+otherwise, rejecting unknown selection and every selected non-success result.
+
+The rollout regression's four snapshot-mutation families use two test-only
+workers. Each case owns its directory and reads an immutable seed. Foreground
+cases drain the workers first; shared policy changes have an explicit barrier.
+Every assigned case must return successfully and leave its exact completion
+receipt. Interruption, failure, missing receipts and duplicate assignments fail
+the suite, and cleanup joins the workers before removing shared scratch. The
+production verifier and all fixture assertions and real deadlines stay intact.
+
+## Evidence and alternatives
+
+Successful run [40272630712](https://github.com/devantler-tech/platform/actions/runs/40272630712)
+took 17m25s for repository guards and 21m06s for cluster policies. The generator
+controls occupied 7m48s of the serial guard job. Manifest validation took 14m14s;
+its unchanged duration and shared runner queues still bound overall feedback.
+
+The first [scheduling trial](https://github.com/devantler-tech/platform/actions/runs/40289350541)
+reduced the guard job to 9m32s and completed its longest policy suite in 12m09s.
+That attempt failed because existing validators assumed the old job locations;
+it is not evidence of a successful complete feedback loop. Its manifest job took
+17m22s. OpenBao standby, Crossplane egress and native mail accounted for 134s,
+143s and 129s respectively, motivating the additional independent job. The
+remaining validators now check the new owning jobs and their required
+dependencies, retaining the existing lost-command and bypass controls.
+
+The next [complete green trial](https://github.com/devantler-tech/platform/actions/runs/40293296500)
+took 17m12s including queue and setup. Manifest validation took 11m52s and its
+parallel regression job 6m23s, leaving room to rebalance four unchanged checks.
+The services policy suite took 12m11s; the rollout regression alone consumed
+11m10s. These observations motivate the bounded fixture workers and rebalancing;
+the complete feedback target remains unproven until another native run finishes.
+
+The [bounded-worker trial](https://github.com/devantler-tech/platform/actions/runs/40300652085)
+finished in 9m57s, but failed the executable-bit guard because the new worker
+test was tracked as mode `100644`. Its services suite passed in 8m02s; both
+manifest jobs passed in 9m18s. The mode is corrected to `100755`, and both
+executable guards pass. This failed attempt is timing evidence, not successful
+feedback acceptance. The follow-up must pass every native gate at its own head.
+
+[GitHub's matrix documentation](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations)
+defines independent job combinations, fail-fast and concurrency limits. The
+inference is that overlapping independent suites can shorten the critical path;
+only a complete first-attempt workflow measurement establishes the actual result.
+The measurement includes setup and runner queue time.
+
+Running all scripts concurrently inside one checkout would require proving that
+their temporary files, environment changes and fixture mutations never interfere.
+Separate hosted jobs provide that isolation. The narrowly scoped rollout workers
+instead prove distinct case directories and immutable inputs, with barriers for
+shared changes and serial foreground cases. Caching prior verdicts, omitting
+tests and increasing timeouts do not meet the feedback or coverage requirement.
+
+## Verification and recovery
+
+`scripts/tests/test-policy-ci-job.sh` checks the authored partition and exercises
+missing, duplicate and unknown suites, unselected and repeated regressions,
+ignored failures and premature cancellation. The generator and existing policy
+result controls exercise success and every selected non-success result.
+The additional manifest-regression boundary exercises selection, success,
+failure, cancellation, missing results and unexpected skips.
+`scripts/tests/test-fixture-workers.sh` proves actual overlap, complete receipts,
+serial operation and rejection of failures, early errors, duplicate or unknown
+cases, missing, corrupt or unassigned receipts, interruption and invalid worker
+bounds. A failing peer must still be joined with its healthy sibling before
+cleanup returns; serial mode also checks the observed concurrency bound.
+
+Before accepting a scheduling edit, compare its complete command inventory and
+step inputs with the reviewed baseline. A passing partition check establishes
+complete routing of the authored steps; it does not prove that a deleted command
+was an authorized removal. Review and native CI retain that responsibility.
+
+Observe every matrix suite and the required aggregate at the candidate and merged
+source. Record actual durations on #5362; a green workflow does not by itself
+prove the 15-minute target. Recovery restores serial scheduling through a reviewed
+change while retaining the same validation commands and required-result gates.

scripts/tests/lib/fixture-workers.lib.sh

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# Test-only workers. Callbacks own distinct case directories and read immutable
+# seed data. Drain before a foreground case or any shared-input mutation.
+
+# fixture_pool_error reports a pool contract violation as an infrastructure error.
+fixture_pool_error() {
+  printf 'fixture-workers: %s\n' "$*" >&2
+  return 2
+}
+
+# fixture_pool_init creates private tracking files for one or two isolated workers.
+fixture_pool_init() {
+  case "$2" in 1 | 2) ;; *)
+    fixture_pool_error 'workers must be 1 or 2'
+    return 2
+    ;;
+  esac
+  FIXTURE_POOL_ROOT="$1/fixture-workers"
+  mkdir "$FIXTURE_POOL_ROOT" "$FIXTURE_POOL_ROOT/logs" "$FIXTURE_POOL_ROOT/receipts" || return 2
+  FIXTURE_POOL_JOBS=$2 FIXTURE_POOL_PENDING=0 FIXTURE_POOL_FAILED=0 FIXTURE_POOL_INTERRUPTED=0
+  FIXTURE_POOL_FINISHED=0 FIXTURE_POOL_VERDICT=0
+  : >"$FIXTURE_POOL_ROOT/expected"
+  : >"$FIXTURE_POOL_ROOT/pending"
+  : >"$FIXTURE_POOL_ROOT/callbacks"
+}
+
+# fixture_pool_register admits each declared callback once before cases are submitted.
+fixture_pool_register() {
+  if ! [[ "$1" =~ ^[a-zA-Z_][a-zA-Z0-9_]*$ ]] || ! declare -F "$1" >/dev/null; then
+    fixture_pool_error 'callback is not a declared function'
+    return 2
+  fi
+  if grep -qxF "$1" "$FIXTURE_POOL_ROOT/callbacks"; then
+    fixture_pool_error 'duplicate callback'
+    return 2
+  fi
+  printf '%s\n' "$1" >>"$FIXTURE_POOL_ROOT/callbacks"
+}
+
+# fixture_pool_interrupt records cancellation so joining workers cannot report success.
+fixture_pool_interrupt() { FIXTURE_POOL_INTERRUPTED=1; }
+
+# fixture_pool_drain joins the current batch and emits every case log before returning failure.
+fixture_pool_drain() {
+  [ -n "${FIXTURE_POOL_ROOT:-}" ] || return 0
+  local pid name result
+  while read -r pid name; do
+    [ -n "$pid" ] || continue
+    while :; do
+      result=0
+      wait "$pid" || result=$?
+      if [ "$result" = 0 ]; then break; fi
+      if [ "$result" = 127 ] || ! kill -0 "$pid" 2>/dev/null; then
+        FIXTURE_POOL_FAILED=1
+        break
+      fi
+    done
+    cat "$FIXTURE_POOL_ROOT/logs/$name" || FIXTURE_POOL_FAILED=1
+  done <"$FIXTURE_POOL_ROOT/pending"
+  : >"$FIXTURE_POOL_ROOT/pending"
+  FIXTURE_POOL_PENDING=0
+  [ "$FIXTURE_POOL_INTERRUPTED" = 0 ] || return 2
+  [ "$FIXTURE_POOL_FAILED" = 0 ]
+}
+
+# fixture_pool_submit assigns a unique case to a registered callback and enforces the worker bound.
+fixture_pool_submit() {
+  local callback=$1 name=$2 pid
+  shift
+  [ "$FIXTURE_POOL_FINISHED" = 0 ] && [ "$FIXTURE_POOL_INTERRUPTED" = 0 ] || return 2
+  grep -qxF "$callback" "$FIXTURE_POOL_ROOT/callbacks" || {
+    fixture_pool_error 'unregistered callback'
+    return 2
+  }
+  [[ "$name" =~ ^[a-zA-Z0-9_-]+$ ]] || {
+    fixture_pool_error 'invalid case identifier'
+    return 2
+  }
+  if grep -qxF "$name" "$FIXTURE_POOL_ROOT/expected"; then
+    fixture_pool_error 'duplicate case assignment'
+    return 2
+  fi
+  printf '%s\n' "$name" >>"$FIXTURE_POOL_ROOT/expected"
+  (
+    # The parent owns shared scratch; a worker must never run its EXIT cleanup.
+    trap - EXIT HUP INT TERM
+    set -euo pipefail
+    export FIXTURE_POOL_WORKER=1
+    "$callback" "$@"
+    printf '%s\n' "$name" >"$FIXTURE_POOL_ROOT/receipts/$name"
+  ) >"$FIXTURE_POOL_ROOT/logs/$name" 2>&1 &
+  pid=$!
+  printf '%s %s\n' "$pid" "$name" >>"$FIXTURE_POOL_ROOT/pending"
+  FIXTURE_POOL_PENDING=$((FIXTURE_POOL_PENDING + 1))
+  [ "$FIXTURE_POOL_INTERRUPTED" = 0 ] || return 2
+  if [ "$FIXTURE_POOL_PENDING" -ge "$FIXTURE_POOL_JOBS" ]; then fixture_pool_drain; fi
+}
+
+# fixture_pool_finish joins workers and verifies an exact completion receipt for every assigned case.
+# Repeated calls preserve the first complete verdict, including interruption and missing receipts.
+fixture_pool_finish() {
+  [ -n "${FIXTURE_POOL_ROOT:-}" ] || return 0
+  if [ "$FIXTURE_POOL_FINISHED" = 1 ]; then return "$FIXTURE_POOL_VERDICT"; fi
+  local result=0 name count receipts=0 file
+  fixture_pool_drain || result=$?
+  while IFS= read -r name; do
+    if ! printf '%s\n' "$name" | cmp -s - "$FIXTURE_POOL_ROOT/receipts/$name"; then result=1; fi
+  done <"$FIXTURE_POOL_ROOT/expected"
+  count="$(wc -l <"$FIXTURE_POOL_ROOT/expected" | tr -d ' ')"
+  for file in "$FIXTURE_POOL_ROOT/receipts/"*; do
+    [ -f "$file" ] || continue
+    receipts=$((receipts + 1))
+  done
+  [ "$receipts" = "$count" ] || result=1
+  if [ "$FIXTURE_POOL_INTERRUPTED" != 0 ]; then result=2; fi
+  FIXTURE_POOL_FINISHED=1 FIXTURE_POOL_VERDICT=$result
+  if [ "$result" != 0 ]; then
+    printf 'fixture-workers: incomplete or failed cases\n' >&2
+    return "$result"
+  fi
+  printf 'FIXTURE-WORKERS COMPLETE cases=%s completed=%s workers=%s\n' "$count" "$receipts" "$FIXTURE_POOL_JOBS"
+}