Repository navigation
feat(observability): alert on an unhealthy Crossplane provider fleet #5026
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
devantler
merged 3 commits into
main
from
claude/observability-crossplane-provider-alert-3457
Oct 17, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
8a4ee97
feat(observability): alert on an unhealthy Crossplane provider fleet
devantler 1b2beaa
chore(kubescape): regenerate the exception mirror for the provider alert
devantler 75198ac
test(observability): exercise the provider alert against the full Pro…
devantler File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 changes: 16 additions & 0 deletions
16
...tructure/controllers/coroot/cluster-role-binding-crossplane-provider-unhealthy-alert.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Grants the crossplane-provider-unhealthy-alert ServiceAccount its read-only view | ||
| # of Crossplane Providers. See | ||
| # cluster-role-crossplane-provider-unhealthy-alert.yaml for the scope rationale. | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRoleBinding | ||
| metadata: | ||
| name: crossplane-provider-unhealthy-alert | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: crossplane-provider-unhealthy-alert | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: crossplane-provider-unhealthy-alert | ||
| namespace: observability |
18 changes: 18 additions & 0 deletions
18
...s/infrastructure/controllers/coroot/cluster-role-crossplane-provider-unhealthy-alert.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # What the crossplane-provider-unhealthy-alert CronJob may read. | ||
| # | ||
| # Read-only, and ONLY the Crossplane Provider package. Providers are | ||
| # cluster-scoped, so a ClusterRole is required; it grants get/list on that single | ||
| # resource type and nothing else. | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: crossplane-provider-unhealthy-alert | ||
| rules: | ||
| - apiGroups: | ||
| - pkg.crossplane.io | ||
| resources: | ||
| - providers | ||
| verbs: | ||
| - get | ||
| - list |
229 changes: 229 additions & 0 deletions
229
...bases/infrastructure/controllers/coroot/cron-job-crossplane-provider-unhealthy-alert.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,229 @@ | ||
| # Alerts on a Crossplane `Provider` whose `Healthy` condition has not been | ||
| # `True` (False, Unknown or missing) for longer than a grace period. | ||
| # | ||
| # WHY THIS EXISTS AS A SEPARATE CHECK | ||
| # | ||
| # kstatus has no readiness rule for a Crossplane `Provider`, so the | ||
| # `wait: true` health gate described in docs/dr/alerting.md does not gate on | ||
| # one: an unhealthy Provider fleet leaves `infrastructure` Ready=True. During | ||
| # #3454 every Provider was Healthy=False, no provider pod ran, and Flux and every | ||
| # GitHub surface stayed green; Coroot inspections see no crashloop because there | ||
| # is no pod at all. This check reads Provider health directly through the API | ||
| # server and is independent of every Flux health gate (#3457). | ||
| # | ||
| # It follows cron-job-cnpg-degraded-alert.yaml exactly — same hardening, webhook | ||
| # handling, silent-zero guards, delivery and repeat-every-tick behaviour. Read | ||
| # that file for the rationale behind each guard; it is not repeated here. | ||
| # | ||
| # GRACE PERIOD — `Healthy`'s `lastTransitionTime` IS A REAL TRANSITION TIME | ||
| # | ||
| # The grace period keys on the Healthy condition's `lastTransitionTime`. | ||
| # crossplane-runtime keeps a condition's stored timestamp whenever type, status, | ||
| # reason and message are unchanged. Measured on this cluster, with the | ||
| # 2026-10-17 samples taken on Crossplane v2.4.2 (re-measure after a Crossplane | ||
| # minor upgrade, since this rule is what makes the grace reachable): | ||
| # * during #3454 (2026-08-29T15:09:53Z) all five Providers were Healthy=False | ||
| # with `lastTransitionTime` stable at 14:51:40–14:51:42Z for ~18 minutes | ||
| # across an active reconcile loop; | ||
| # * a healthy fleet of four Providers, sampled on prod at | ||
| # 2026-10-17T17:31:40Z and 50s later, had byte-identical Healthy | ||
| # `lastTransitionTime` values (and an unchanged `resourceVersion`, so the | ||
| # healthy fleet is not rewriting its status); | ||
| # * the provider-upjet-unifi v1.1.2 upgrade on 2026-10-17 went Installed=True | ||
| # at 16:54:05Z and Healthy=True at 16:55:06Z — about one minute not healthy | ||
| # while the new revision started. | ||
| # So the grace is reachable, and an ordinary upgrade stays well inside the 900s | ||
| # grace and does not page. (This is the property the CNPG header warns must be | ||
| # re-measured before keying a Crossplane condition on it.) | ||
| # | ||
| # A Provider with no Healthy condition at all uses its creationTimestamp as the | ||
| # conservative clock, so a package that never reports health still alerts once | ||
| # the grace has passed — the same rule as the CNPG archiver branch. | ||
| # | ||
| # On a cluster without Crossplane the Provider API returns 404 and the check | ||
| # exits 0. Lives in the observability namespace to reuse the allow-coroot | ||
| # CiliumNetworkPolicy's kube-apiserver, hooks.slack.com:443 and DNS egress. | ||
| # Unset (local/CI) the webhook defaults to the reserved example.invalid host, | ||
| # which the script skips; every other delivery failure fails the Job. | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: CronJob | ||
| metadata: | ||
| name: crossplane-provider-unhealthy-alert | ||
| namespace: observability | ||
| annotations: | ||
| checkov.io/skip1: "CKV_K8S_38=the check container reads its SA token to list Crossplane Providers through the API server" | ||
| # `${GRACE_SECONDS}` below is the container's own env var, not a Flux | ||
| # post-build variable; substitution would blank it (see the CNPG CronJob). | ||
| kustomize.toolkit.fluxcd.io/substitute: disabled | ||
| spec: | ||
| schedule: "*/30 * * * *" | ||
| concurrencyPolicy: Forbid | ||
| startingDeadlineSeconds: 60 | ||
| successfulJobsHistoryLimit: 1 | ||
| failedJobsHistoryLimit: 1 | ||
| jobTemplate: | ||
| spec: | ||
| backoffLimit: 0 | ||
| activeDeadlineSeconds: 180 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: crossplane-provider-unhealthy-alert | ||
| spec: | ||
| restartPolicy: Never | ||
| serviceAccountName: crossplane-provider-unhealthy-alert | ||
| # This check calls the Kubernetes API, so it needs its token. C-0034 | ||
| # is already excepted for the observability namespace. | ||
| automountServiceAccountToken: true | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| runAsUser: 65532 | ||
| # fsGroup makes the mode-0440 webhook file readable by this | ||
| # non-root container (see the CNPG CronJob). | ||
| fsGroup: 65532 | ||
| fsGroupChangePolicy: OnRootMismatch | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| containers: | ||
| - name: check | ||
| image: docker.io/badouralix/curl-jq:latest@sha256:1e7c0284e24572ace7170df9fc91f15fd3b79ebf056d4dde17244d5d74bbfabc | ||
| imagePullPolicy: IfNotPresent | ||
| env: | ||
| - name: GRACE_SECONDS | ||
| value: "900" | ||
| command: | ||
| - /bin/sh | ||
| - -c | ||
| - | | ||
| set -eu | ||
| API=https://kubernetes.default.svc | ||
| SA=/var/run/secrets/kubernetes.io/serviceaccount | ||
|
|
||
| # Read once from the mounted Secret, never from env | ||
| # (checkov CKV_K8S_35). | ||
| WEBHOOK_URL=$(cat /etc/crossplane-provider-unhealthy-alert/url) | ||
|
|
||
| # The value becomes a curl config directive, where a newline | ||
| # would start a second directive: reject control characters. | ||
| if [ "$(printf '%s' "$WEBHOOK_URL" | tr -d '[:cntrl:]')" != "$WEBHOOK_URL" ]; then | ||
| echo "ERROR: webhook URL contains control characters" >&2 | ||
| exit 1 | ||
| fi | ||
| # The token travels in the URL path and egress is allowed by | ||
| # port, not scheme: deliver only to https://. | ||
| case "$WEBHOOK_URL" in | ||
| https://*) ;; | ||
| *) | ||
| echo "ERROR: webhook URL must use https://" >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| # A failed or malformed API read must fail the Job, never | ||
| # read as "every provider healthy". | ||
| code=$(curl -sS -o /tmp/body -w '%{http_code}' \ | ||
| --cacert "$SA/ca.crt" \ | ||
| -H "Authorization: Bearer $(cat "$SA/token")" \ | ||
| "$API/apis/pkg.crossplane.io/v1/providers") | ||
| if [ "$code" = "404" ]; then | ||
| echo "Crossplane Provider CRD not installed on this cluster; nothing to check." | ||
| exit 0 | ||
| fi | ||
| if [ "$code" != "200" ]; then | ||
| echo "ERROR: kube API returned HTTP $code listing Crossplane providers" >&2 | ||
| exit 1 | ||
| fi | ||
| # Require the items ARRAY, not merely the key: {"items":{}} | ||
| # would otherwise count zero providers as healthy. | ||
| if ! jq -e '(.items | type) == "array"' /tmp/body >/dev/null; then | ||
| echo "ERROR: unexpected response shape from kube API" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| total=$(jq -r '.items | length' /tmp/body) | ||
| unhealthy=$(jq -r --argjson grace "${GRACE_SECONDS}" \ | ||
| --argjson now "$(date -u +%s)" ' | ||
| .items[] | ||
| | . as $p | ||
| | (($p.status.conditions // []) | map(select(.type == "Healthy")) | first) as $healthy | ||
| | ($healthy.status // "missing") as $status | ||
| | ($healthy.lastTransitionTime // $p.metadata.creationTimestamp) as $since | ||
| | (if $since == null | ||
| then false | ||
| else (($now - ($since | fromdate)) >= $grace) | ||
| end) as $pastGrace | ||
| | if ($status != "True" and $pastGrace) | ||
| then "• `\($p.metadata.name)` — Healthy=\($status) since \($since)" | ||
| + (if (($healthy.message // "") | length) > 0 | ||
| then ": \($healthy.message | .[0:300])" | ||
| else "" | ||
| end) | ||
| else empty | ||
| end | ||
| ' /tmp/body) | ||
|
|
||
| if [ -z "$unhealthy" ]; then | ||
| echo "Checked $total Crossplane provider(s); none unhealthy beyond ${GRACE_SECONDS}s." | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Unhealthy Crossplane provider(s) detected:" | ||
| echo "$unhealthy" | ||
|
|
||
| text=$(printf '*Unhealthy Crossplane provider(s)*\n\n%s\n\n_A Provider has not been Healthy for longer than %ss. Its managed resources are not being reconciled; Flux does not gate on Provider health, so nothing else reports this. Inspect `kubectl get providers.pkg.crossplane.io` and the crossplane-system pods._' \ | ||
| "$unhealthy" "${GRACE_SECONDS}") | ||
| jq -n --arg t "$text" '{text: $t}' > /tmp/payload.json | ||
|
|
||
| # Only the RFC 2606 placeholder host may skip delivery. | ||
| case "$WEBHOOK_URL" in | ||
| https://example.invalid/*) | ||
| echo "No Slack webhook configured; alert not delivered." | ||
| exit 0 | ||
| ;; | ||
| esac | ||
|
|
||
| # No `|| true`: a failed POST must fail the Job. The URL | ||
| # reaches curl on stdin as an unquoted config value, never in | ||
| # argv (see the CNPG CronJob for both rationales). | ||
| printf 'url = %s\n' "$WEBHOOK_URL" | | ||
| curl -sf --max-time 15 \ | ||
| --retry 3 --retry-delay 2 --retry-all-errors --retry-connrefused \ | ||
| -H 'Content-Type: application/json' \ | ||
| --data @/tmp/payload.json \ | ||
| --config - | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 65532 | ||
| runAsGroup: 65532 | ||
| # CIS-5.7.3 (C-0211); a documented no-op on this AppArmor cluster. | ||
| seLinuxOptions: {} | ||
| capabilities: | ||
| drop: | ||
| - ALL | ||
| volumeMounts: | ||
| # Writable scratch for the API response and the payload. | ||
| - name: tmp | ||
| mountPath: /tmp | ||
| - name: webhook | ||
| mountPath: /etc/crossplane-provider-unhealthy-alert | ||
| readOnly: true | ||
| resources: | ||
| requests: | ||
| cpu: 5m | ||
| memory: 32Mi | ||
| limits: | ||
| # CPU limit set for kubescape C-0270. | ||
| cpu: 100m | ||
| memory: 64Mi | ||
| volumes: | ||
| - name: tmp | ||
| emptyDir: {} | ||
| # 288 is 0440, written in decimal so YAML 1.1 and 1.2 readers agree | ||
| # (see the CNPG CronJob). | ||
| - name: webhook | ||
| secret: | ||
| secretName: crossplane-provider-unhealthy-alert-webhook | ||
| defaultMode: 288 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 changes: 16 additions & 0 deletions
16
k8s/bases/infrastructure/controllers/coroot/secret-crossplane-provider-unhealthy-alert.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Slack incoming-webhook for the crossplane-provider-unhealthy-alert CronJob. It | ||
| # is the same ${alertmanager_webhook_url} the cnpg-degraded-alert Secret carries, | ||
| # held in a Secret rather than inline env for the same reasons (see | ||
| # secret-cnpg-degraded-alert.yaml). | ||
| # | ||
| # The inline default keeps `ksail workload validate` (which has no SOPS access) | ||
| # building and leaves local/CI inert. | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: crossplane-provider-unhealthy-alert-webhook | ||
| namespace: observability | ||
| type: Opaque | ||
| stringData: | ||
| url: ${alertmanager_webhook_url:=https://example.invalid/no-slack-configured} |
10 changes: 10 additions & 0 deletions
10
...nfrastructure/controllers/coroot/service-account-crossplane-provider-unhealthy-alert.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| # Identity for the crossplane-provider-unhealthy-alert CronJob, which lists the | ||
| # cluster-scoped Crossplane Providers. See | ||
| # cron-job-crossplane-provider-unhealthy-alert.yaml for why the check exists and | ||
| # cluster-role-crossplane-provider-unhealthy-alert.yaml for what it may read. | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: crossplane-provider-unhealthy-alert | ||
| namespace: observability |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.