Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 1 addition & 19 deletions .github/renovate.json
Original file line number Diff line number Diff line change
Expand Up @@ -131,18 +131,7 @@
},
{
"customType": "regex",
"description": "Track the bundled Postgres image tag pinned for the Crossview app's chart-bundled database (k8s/bases/apps/crossview/helm-release.yaml). The crossview chart defaults database.image.tag to `latest`; pinning it lives in the Flux HelmRelease's inline spec.values, which the built-in flux manager (chart version only) and the kubernetes/helm-values managers (values.yaml files only) never see — so a custom manager is the only way to keep the pin current. Matches the `# renovate:` comment and the tag line that follows it, resolving from the docker datasource (Docker Hub library/postgres).",
"managerFilePatterns": [
"/^k8s/bases/apps/crossview/helm-release\\.ya?ml$/"
],
"matchStrings": [
"# renovate: datasource=(?<datasource>\\S+) depName=(?<depName>\\S+)\\s+tag:\\s*\"(?<currentValue>[^\"]+)\""
],
"versioningTemplate": "docker"
},
{
"customType": "regex",
"description": "Track the PostgreSQL operand image pinned on the platform-owned CloudNativePG Clusters (coroot-db, umami-db — each a cluster.yaml). These pin spec.imageName explicitly instead of riding the CNPG operator default because the latest GA operator (v1.29.1) still defaults to postgresql:18.3-system-trixie — a now-frozen tag (last rebuilt 2026-05-11) carrying fixable base-OS CVEs (openssl/libssl3, libgnutls30, libc6, python3.13) — and the default only advances to 18.4 in the not-yet-GA v1.30, so the built-in flux manager bumping the cloudnative-pg operator chart does NOT clear them. spec.imageName lives in the Cluster CR (not a chart version or a values.yaml), so the flux, kubernetes, and helm-values managers never see it; a custom manager is the only way to keep it current. Matches the `# renovate:` comment and the imageName line that follows it (mirroring the crossview manager above), resolving from the docker datasource (ghcr.io/cloudnative-pg/postgresql); docker versioning tracks the 18.4-system-trixie -> 18.5-system-trixie minor line while preserving the -system-trixie flavor suffix. The tenant-owned wedding-db pins its own imageName in the external wedding-app artifact and is intentionally not covered here. NOTE (platform#2149): this manager's file pattern originally targeted a `postgres-cluster.yaml` filename that has never existed in this repo — the actual files are `cluster.yaml` — so the manager silently matched nothing since it was authored; fixed to the real filename.",
"description": "Track the PostgreSQL operand image pinned on the platform-owned CloudNativePG Clusters (coroot-db, umami-db — each a cluster.yaml). These pin spec.imageName explicitly instead of riding the CNPG operator default because the latest GA operator (v1.29.1) still defaults to postgresql:18.3-system-trixie — a now-frozen tag (last rebuilt 2026-05-11) carrying fixable base-OS CVEs (openssl/libssl3, libgnutls30, libc6, python3.13) — and the default only advances to 18.4 in the not-yet-GA v1.30, so the built-in flux manager bumping the cloudnative-pg operator chart does NOT clear them. spec.imageName lives in the Cluster CR (not a chart version or a values.yaml), so the flux, kubernetes, and helm-values managers never see it; a custom manager is the only way to keep it current. Matches the `# renovate:` comment and the imageName line that follows it, resolving from the docker datasource (ghcr.io/cloudnative-pg/postgresql); docker versioning tracks the 18.4-system-trixie -> 18.5-system-trixie minor line while preserving the -system-trixie flavor suffix. The tenant-owned wedding-db pins its own imageName in the external wedding-app artifact and is intentionally not covered here. NOTE (platform#2149): this manager's file pattern originally targeted a `postgres-cluster.yaml` filename that has never existed in this repo — the actual files are `cluster.yaml` — so the manager silently matched nothing since it was authored; fixed to the real filename.",
"managerFilePatterns": ["/(^|/)cluster\\.ya?ml$/"],
"matchStrings": [
"# renovate: datasource=(?<datasource>\\S+) depName=(?<depName>\\S+)\\s+imageName:\\s*\\S+:(?<currentValue>\\S+)"
Expand Down Expand Up @@ -322,13 +311,6 @@
"allowedVersions": "/^v?1\\.14\\.0$/",
"automerge": false
},
{
"description": "headlamp is HARD-PINNED at 0.42.0 (k8s/bases/apps/headlamp/helm-release.yaml): 0.43.0 introduced an in-cluster OIDC login regression in Safari, and the same broken popup handoff was reproduced after upgrading to 0.44.0 — the session cookie succeeds, but Safari ITP suppresses the cross-window localStorage storage event after the popup's cross-site Dex/GitHub hops, so the popup never redirects its opener (upstream kubernetes-sigs/headlamp#6353; fix PR #6355 unmerged; tracked in #2438). Keep the last Safari-verified release until a released upstream fix is manually verified end-to-end; automerge:false placed last so an unverified Headlamp update cannot inherit the minor/patch automerge defaults above.",
"matchDatasources": ["helm"],
"matchPackageNames": ["headlamp"],
"allowedVersions": "/^v?0\\.42\\.0$/",
"automerge": false
},
{
"description": "KubeVirt and CDI ship as VENDORED upstream operator manifests (k8s/bases/infrastructure/controllers/kubevirt/kubevirt-operator.yaml, .../cdi/cdi-operator.yaml) — thousands of lines of CRDs, RBAC and the operator Deployment vendored as one unit from each release's published asset. The kubernetes manager only sees the `image:` lines inside them, so a Renovate bump rewrites the operator image alone and silently skews it against the CRDs/RBAC vendored next to it; the system-test would still pass bring-up (operators tolerate startup skew) so automerge would land the skew unseen. Keep the update PR as the new-release signal but require a maintainer to re-vendor the full manifest from the matching release before merging. automerge:false placed last so it overrides the major/minor/patch automerge defaults above.",
"matchPackageNames": ["quay.io/kubevirt/**"],
Expand Down
95 changes: 18 additions & 77 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,6 @@ jobs:
k8s: ${{ steps.filter.outputs.k8s }}
bridge_validation: ${{ steps.filter.outputs.bridge_validation }}
talos: ${{ steps.filter.outputs.talos }}
crossview: ${{ steps.filter.outputs.crossview }}
queue_receipt: ${{ steps.queue-receipt.outputs.receipt }}
steps:
- name: 📑 Checkout
Expand Down Expand Up @@ -228,7 +227,6 @@ jobs:
- 'tests/validate-pod-security-provider-deployment/**'
- 'scripts/tests/test-pvc-prune-safety.sh'
- 'scripts/tests/test-flux-kustomization-prune.sh'
- 'scripts/tests/test-headlamp-plugin-removal.sh'
- 'scripts/tests/test-github-config-role-activation-parity.sh'
- 'scripts/tests/test-restrict-tenant-secret-stores.sh'
- 'scripts/tests/test-tenant-kro-pull-secret.sh'
Expand Down Expand Up @@ -456,10 +454,6 @@ jobs:
- 'scripts/tests/test-cilium-mutual-auth-policy.sh'
- 'scripts/tests/test-cilium-mutual-auth-policy-regressions.sh'
- 'scripts/tests/test-coroot-postgres-scrape-policy.sh'
- 'scripts/tests/test-crossview-coroot-postgres-integration.sh'
- 'scripts/tests/test-crossview-postgres-durable-storage.sh'
- 'scripts/tests/test-crossview-login-alerter.sh'
- 'scripts/tests/test-crossview-login-alerter-runtime.sh'
- 'scripts/tests/test-coroot-api-key-contract.sh'
- 'scripts/tests/test-coroot-availability-remediation.sh'
- 'scripts/tests/test-descheduler-availability.sh'
Expand Down Expand Up @@ -743,21 +737,6 @@ jobs:
- 'go.sum'
- '.github/workflows/ci.yaml'
- '.github/workflows/cd.yaml'
# What the Docker reproduction of the Crossview login outage reads:
# the chart pin and values, the prod apps overlay and the shared
# components it applies to every HelmRelease, the sensor and its test.
crossview:
- 'k8s/bases/apps/crossview/**'
- 'k8s/bases/apps/kustomization.yaml'
- 'k8s/bases/components/annotations-transformers/**'
- 'k8s/bases/components/helmrelease-drift-detection/**'
- 'k8s/bases/components/helmrelease-flux-defaults/**'
- 'k8s/providers/hetzner/apps/crossview/**'
- 'k8s/providers/hetzner/apps/kustomization.yaml'
- 'k8s/providers/hetzner/infrastructure/coroot/cron-job-crossview-login-alerter.yaml'
- 'k8s/clusters/prod/bootstrap/config-map.yaml'
- 'scripts/tests/test-crossview-login-alerter-runtime.sh'
- '.github/workflows/ci.yaml'

# The full offline fixture suite runs alongside changes, which retains the real production
# render and ShellCheck. Both deployment and the required gate wait for these regressions.
Expand Down Expand Up @@ -964,11 +943,6 @@ jobs:
- name: 💾 Validate protected Longhorn client maintenance
run: go test -race ./scripts/maintain-longhorn-clients

- name: 💾 Validate retained portal storage protection
env:
PORTAL_STORAGE_REQUIRE_NATIVE_CONTROL: '1'
run: go test -race -count=1 ./internal/portalstorage ./scripts/protect-backstage-storage

- name: ⎈ Install checksum-verified native Helm
shell: bash
env:
Expand Down Expand Up @@ -1008,7 +982,7 @@ jobs:
run: |
shellcheck scripts/verify-wedding-backup-denial.sh scripts/verify-wedding-backup-denial-pod.sh scripts/tests/test-verify-wedding-backup-denial.sh
bash scripts/tests/test-verify-wedding-backup-denial.sh
shellcheck scripts/tests/test-wedding-backup-denial-runtime.sh
shellcheck scripts/tests/test-wedding-backup-denial-runtime.sh scripts/tests/lib/configure-hosted-registry-cache.sh
bash scripts/tests/test-wedding-backup-denial-runtime.sh

# The stale shared copy of the Wedding catalogue is removed on this proof's
Expand All @@ -1020,6 +994,16 @@ jobs:
shellcheck scripts/verify-wedding-shared-backup-coverage.sh scripts/verify-wedding-shared-backup-coverage-pod.sh scripts/remove-wedding-shared-backup-pod.sh scripts/verify-shared-backup-consumers.sh scripts/tests/test-verify-wedding-shared-backup-coverage.sh
bash scripts/tests/test-verify-wedding-shared-backup-coverage.sh

# The Crossview, Headlamp and Backstage leftovers, including retained
# database volumes, are deleted on this script's word, and deleted data
# cannot be restored. These cases keep an undeployed removal, a foreign
# object, attached or unrelated storage, a failed read or a survivor from
# passing.
- name: 🧹 Validate the unused-app retirement
run: |
shellcheck scripts/retire-unused-apps.sh scripts/tests/test-retire-unused-apps.sh
bash scripts/tests/test-retire-unused-apps.sh

- name: 💾 Validate two-stage PVC retirement
# A merge-group artifact is speculative, but a PVC deletion is not: its
# deletionTimestamp survives queue eviction and main cannot undo it.
Expand Down Expand Up @@ -1272,16 +1256,10 @@ jobs:
go test ./scripts/validate-concurrency-queue
go run ./scripts/validate-concurrency-queue "${workflows[@]}"

# This guard spans the HelmRelease and Renovate policy, so it must run
# without routing policy-only changes into the production k8s deploy path.
- name: 🔐 Validate Headlamp OIDC version gate
run: |
shellcheck scripts/tests/test-headlamp-oidc-version-gate.sh
bash scripts/tests/test-headlamp-oidc-version-gate.sh

# The renderer's Helm SDK follows the deployed controller, not Helm's own
# releases. This spans the build guard and the Renovate policy, so it runs
# here for the same reason as the Headlamp gate above (#4738).
# releases. This spans the build guard and the Renovate policy, so it must
# run without routing policy-only changes into the production k8s deploy
# path (#4738).
- name: 📌 Validate the controller Helm SDK Renovate pin
run: |
shellcheck scripts/tests/test-controller-helm-sdk-renovate-pin.sh
Expand Down Expand Up @@ -2018,10 +1996,9 @@ jobs:

- name: 🧩 Validate embedded JSON blobs
# ConfigMaps that embed whole JSON documents as block scalars (e.g. the
# Headlamp Kubescape exceptions ConfigMap's exceptionPolicies) are opaque
# strings to schema validation, so a stray comma ships silently and only
# surfaces at consumption time — where an empty exceptions view reads
# identical to a clean posture. Local Go tests and validation need no
# Coroot dashboards) are opaque strings to schema validation, so a stray
# comma ships silently and only surfaces at consumption time. Local Go
# tests and validation need no
# cluster or credentials. See scripts/validate-embedded-json/ (#2480).
run: |
shellcheck scripts/validate-embedded-json.sh
Expand Down Expand Up @@ -2162,12 +2139,6 @@ jobs:
shellcheck scripts/tests/test-world-zone-trial.sh
bash scripts/tests/test-world-zone-trial.sh

- name: 🧩 Validate Headlamp plugin removal
if: needs.changes.outputs.k8s == 'true'
# Removing runtime-downloaded plugins must also prune upgrade state and
# preserve Crossview's authenticated fresh-install entry point.
run: bash scripts/tests/test-headlamp-plugin-removal.sh

- name: 🔑 Validate github-config Role covers the activated MR kinds
if: needs.changes.outputs.k8s == 'true'
# The tenant Role enumerates managed-resource kinds instead of `*`, so
Expand Down Expand Up @@ -2375,35 +2346,6 @@ jobs:
run: |
shellcheck scripts/tests/test-coroot-postgres-scrape-policy.sh
bash scripts/tests/test-coroot-postgres-scrape-policy.sh
shellcheck scripts/tests/test-crossview-coroot-postgres-integration.sh
bash scripts/tests/test-crossview-coroot-postgres-integration.sh

- name: 💾 Validate Crossview PostgreSQL durable storage
if: needs.changes.outputs.k8s == 'true'
# Render the pinned chart through Flux's production post-renderers so
# persistence cannot regress to an emptyDir or an overlapping rollout.
run: |
shellcheck scripts/tests/test-crossview-postgres-durable-storage.sh
bash scripts/tests/test-crossview-postgres-durable-storage.sh

- name: 🔓 Validate the Crossview login alert
if: needs.changes.outputs.k8s == 'true'
# Crossview's only probe never touches its database, so an empty schema
# broke every sign-in for six days with the pod Ready (#3315). This pins
# the sensor's verdicts and wiring against stubbed answers.
run: |
shellcheck scripts/tests/test-crossview-login-alerter.sh scripts/tests/test-crossview-login-alerter-runtime.sh
bash scripts/tests/test-crossview-login-alerter.sh

- name: 🔓 Reproduce the Crossview login outage
if: needs.changes.outputs.crossview == 'true'
# Measured at about 30s; the bound only stops a hung pull holding the job.
timeout-minutes: 15
# Runs the pinned app and database in Docker, empties the database
# under the running app and runs the deployed sensor against it, so a
# chart bump that changes what /api/auth/check reports fails here. Only
# Crossview changes pay for the image pulls.
run: bash scripts/tests/test-crossview-login-alerter-runtime.sh

- name: 🔑 Validate Coroot API key and registry egress
if: needs.changes.outputs.k8s == 'true'
Expand Down Expand Up @@ -3093,8 +3035,7 @@ jobs:
# enforced and except-only findings suppressed, the threshold gates the
# residual REAL posture instead of sitting ~15 points under it.
#
# Two surfaces apply these exceptions: this scan, and the Headlamp view
# via the generated headlamp-exceptions ConfigMap. The kubescape-operator's
# Only this scan applies these exceptions. The kubescape-operator's
# stored scan resources (workloadconfigurationscans and the two summary
# kinds) do NOT — their appliedIgnoreRules is empty, so they carry
# pre-exception verdicts. Never read a posture number off those to judge
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/dr-rebuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# without it the step prints the manual Scenario 9 instructions)
#
# Known limits (documented in the runbook):
# * Per-app PVC data (headlamp, actual-budget) is NOT rehydrated into
# * Per-app PVC data (actual-budget) is NOT rehydrated into
# already-running pods — Velero skips existing resources. Recover an app's
# data with the per-app reset dance (runbook Scenario 5).
# * CNPG databases (umami-db) recover from their own barman backups —
Expand Down Expand Up @@ -636,6 +636,6 @@ jobs:
echo "- Load balancer IP: ${IP:-<pending>}"
echo "- DNS: external-dns (policy: sync) repoints the Cloudflare records automatically once HTTPRoutes are Ready — verify with: kubectl -n external-dns logs deploy/external-dns"
echo "- CNPG databases (umami-db): recover from barman/R2 per docs/dr/runbook.md Scenario 5 (kubectl cnpg restore)."
echo "- Per-app PVC data (headlamp, actual-budget): already-running pods keep their fresh volumes; use the per-app reset dance from runbook Scenario 5 if old data is needed."
echo "- Per-app PVC data (actual-budget): already-running pods keep their fresh volumes; use the per-app reset dance from runbook Scenario 5 if old data is needed."
kubectl get kustomizations.kustomize.toolkit.fluxcd.io -A -o wide
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded 2>/dev/null | head -30
Loading
Loading