fix(observability): a degraded CNPG replica is invisible until it wedges the merge queue #2978

🤖 Generated by the Agentic Engineer

Evidence

A degraded CNPG replica is currently invisible to alerting until it wedges the platform merge queue —
and then it is only visible as a Flux failure, 20 minutes at a time.

Measured on prod today (2026-08-05 ~17:1xZ):

Nothing alerted on that 5-hour degradation. What eventually surfaced it was the apps Kustomization
health check timing out and evicting every PR from the merge queue (see the parent issue).

Why the current coverage misses it

docs/dr/alerting.md states the design explicitly: the four top-level Kustomizations wait: true on
their children, "so a failed controller / app / HelmRelease surfaces here as its parent going
NotReady". So today the only signal for a degraded database is the merge-queue gate.

That has two consequences:

1. The alert fires ~20 minutes late, as a Flux error naming a Kustomization rather than a database.
2. It cannot be fixed by relaxing the gate — parent issue direction (1) — because relaxing the gate
   removes the only coverage that exists. The alert must therefore land first.

Coroot's auto-inspections cover crashloops, so today's umami-db-13 (58 restarts) would be caught by
that path — but the 07-14 coroot-db case was a pod that stayed Running, never Ready, with zero
restarts, which no Coroot inspection covers. The parent issue recorded "nothing alerts on it".

What to build

A scan-safe synthetic check that alerts on a CNPG Cluster that has been degraded past a grace
period, independent of any Flux gate — same shape as the existing cluster-heartbeat and
coroot-alert-autosuppressor CronJobs, posting to the ${alertmanager_webhook_url} Slack webhook
already used by Coroot and flux-notifications.

Detection: status.readyInstances < spec.instances, or the Ready condition not True.

Grace period keyed on the Ready condition's lastTransitionTime. This was verified rather than
assumed: sampled 45s apart, every cluster's lastTransitionTime was byte-identical, so CNPG does
not rewrite it each reconcile and it is a genuine transition timestamp. (This is the opposite of
Crossplane's Synced condition, which does rewrite it every ~60s and would make a grace period
silently unreachable.)

Acceptance criteria

• A degraded cluster past the grace period produces exactly one Slack alert naming the namespace,
  cluster, and ready/desired counts.
• The three currently-healthy clusters produce no alert (negative control).
• A cluster inside its grace period produces no alert, so a routine rolling restart or switchover
  does not page.
• The check does not depend on any Flux Kustomization's health gate.
• No new always-on pod (CronJob, matching the existing scan-safe pattern).

Rough size

S–M. Prerequisite for the parent's direction (1); relaxing the gate is a separate follow-up and must
not land before this.