fix(observability): a single stuck coroot-db replica blocks every platform merge

🤖 Generated by the Daily AI Engineer

Problem

A single degraded replica in the coroot-db observability database silently blocked every
platform PR from merging for ~4 hours today, and degraded prod HA. The failure is systemic, not a
one-off.

The chain:

A CNPG switchover left the demoted old primary (coroot-db-7) unable to rejoin: pg_rewind found
the servers diverged on timeline 8 and could not restore the WAL segment it needed from the archive.
CNPG retried pg_rewind forever — 4+ hours, zero restarts, no self-healing and no re-clone. The
pod stayed Running but never Ready, so the Cluster sat in Waiting for the instances to become active (2 of 3 ready).
The infrastructure Flux Kustomization health-checks that Cluster, so it failed with
health check failed after 20m: timeout waiting for: [Cluster/observability/coroot-db].
🚀 Deploy to Prod therefore failed in every merge_group run, so every platform PR was
evicted from the merge queue (observed on fix(backstage): give the startup probe a real budget so it survives a restart #2636).

The important part: the database was serving fine the whole time — primary healthy, one healthy
replica, ContinuousArchiving=True, LastBackupSucceeded=True. A non-critical, still-serving
observability database with one lagging replica took down the entire merge lane.

I unwedged prod by hand (deleted the diverged replica's PVC + pod so CNPG re-cloned it from the
primary; the cluster is back to 3/3 healthy and the merge queue is moving). That is a manual remedy,
not a fix.

Proposed direction

Two independent defects, either of which alone would have prevented this:

A degraded-but-serving observability DB must not gate the merge queue. Deploy health gating
should not hang for 20 minutes on a replica that has no bearing on whether the deploy is good.
Either scope the infrastructure health checks away from non-critical stateful replicas, or make
this class alert instead of block.
A replica that cannot pg_rewind should be re-cloned, not retried forever. Retrying a rewind
that can never succeed (the WAL segment is not in the archive and never will be) is an infinite
loop. Options: ensure WAL needed for rewind is retained (HA replication slots / WAL retention), or
detect the unrecoverable-rewind condition and re-bootstrap the instance.

Also worth noting for the runbook: this switchover was very likely triggered by the rolling Talos node
reboots performed during the GHCR-credential remediation earlier today. After a rolling node reboot,
stateful workloads must be verified re-converged — a reboot can leave a demoted primary permanently
stuck, and nothing alerts on it.

Rough size

M — (1) is a manifest/health-check scoping change; (2) is a CNPG configuration or operator-behaviour
question that may need an upstream conversation.

🔗 Related PRs

#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2424 - docs(dr): add CNPG StorageClass migration runbook (longhorn → longhorn-wffc) [merged]
#2425 - fix(cnpg): move platform Postgres clusters to the longhorn-wffc StorageClass [merged]
#2578 - fix(policy-reporter): run all components at the HA replica floor [merged]
#2622 - fix(ci): heal cancelled merge-group deploys [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

This recurred today on a different cluster — same chain, same outcome

umami-db is now doing exactly what coroot-db did on 07-14, and the platform merge queue is wedged
again as I write this.

Live evidence (2026-08-05 ~17:0xZ, prod):

umami-db-13 is in CrashLoopBackOff, 58 restarts over ~5h. It cannot self-heal — every start
ends in PANIC: could not locate a valid checkpoint record at A/13000028, i.e. its WAL/pg_control is
unrecoverable. Unlike the pg_rewind case, this one does restart; it just PANICs each time.
Cluster/umami/umami-db therefore sits at 2 of 3 ready, phase: "Cluster cannot proceed to reconciliation due to an error while interacting with plugins".
The database is serving fine — primary umami-db-12 healthy, currentPrimary == targetPrimary,
ContinuousArchiving=True, LastBackupSucceeded=True. Exactly the "degraded but serving" shape
this issue describes.
The apps Kustomization (spec.wait: true, no explicit healthChecks, 20m timeout) consequently
fails, and Flux names the single culprit:
health check failed after 20m0.03s: timeout waiting for: [Cluster/umami/umami-db status: 'InProgress']
So 🚀 Deploy to Prod fails in every merge_group run and every platform PR is evicted.
Measured today on fix(k8s): declare CoreDNS's nonroot UID instead of inheriting it #2975: added_to_merge_queue 14:35 → removed 15:20, added 16:18 → removed 16:57.
fix(ci): retry the cordon claim when an unrelated Node write breaks its CAS test #2977 and fix(ci): fail the deploy when the Cilium rollout gate outruns its window #2973 are cycling through the same 20-minute failure right now.

Two distinct causes were behind #2975's two evictions, worth separating:

15:20 — a GHCR secondary rate limit while fetching the provenance attestation
(DENIED: You have exceeded a secondary rate limit). Transient, and already tracked as A transient GHCR push denial is treated as non-retryable and evicts PRs from the merge queue #2957 —
the same verify_evidence step succeeded 4 minutes later in the Heal job, which is a clean control.
16:57 — this issue: apps health-check timeout on Cluster/umami/umami-db.

Why this raises the priority

This is the second occurrence in three weeks, on a different database, in a different
Kustomization (apps, not infrastructure). That rules out the one-off reading: any CNPG cluster
anywhere in the tree can wedge the entire merge lane by losing one non-critical replica, and the
blast radius is every product's ability to ship.

Direction (1) in the issue body — a degraded-but-serving database must not gate the merge queue — is
the part that would have prevented both occurrences, and it is a manifest change rather than an
upstream conversation. I am treating it as the actionable half.

What I could not do

Unwedging needs the corrupt replica's PVC + pod deleted so CNPG re-clones it from the primary — the
same manual remedy used on 07-14. Cluster mutation is outside my runtime authority (and deleting a
production PVC is destructive-shaped even though CNPG makes it recoverable), so I have not
attempted it and have raised the unwedge to the maintainer instead.

@devantler

🤖 Generated with Claude Code in an interactive session with @devantler — verified analysis, not direction.

Second occurrence today on `umami-db` — and the mechanism is not what we recorded

umami-db-13 (prod, umami namespace) wedged the merge queue the same way this issue describes for
coroot-db: Cluster stuck at 2/3, apps Kustomization failing its 20m health check, 🚀 Deploy to Prod failing in every merge_group run, PRs evicted (#2975 twice, then #2977/#2973).

Unwedged by the documented remedy (delete the replica's PVC + pod → CNPG re-cloned from the primary).
Recovery took 47 seconds; cluster is Ready=True, 3/3, all instances on TL 61. Note CNPG assigned
the new instance serial 1, not 13 — it picks the lowest free serial, so the replica is now
umami-db-1.

The corrupt-WAL framing is wrong, and it matters for the fix

Nothing was corrupt. Evidence from the live cluster:

Storage was healthy. Longhorn volume pvc-9ce20317-…: state: attached, robustness: healthy.
Node prod-worker-3 Ready, no disk or memory pressure.
restore_command was working. The barman sidecar successfully restored 0000002E.history
through 0000003D.history from the archive during every crash loop.
The replica was stranded across timeline switches. Its pg_control sat on timeline 44
(0000002C0000000A00000013), last checkpoint Aug 2 19:27 — three days stale. The cluster was on
timeline 61. wal-restore even logged
Refusing to restore future timeline history file … fileTimeline:62, clusterTimeline:61.
The WAL it needed no longer existed anywhere. TL 44's own segment is not in the archive, so
PostgreSQL followed the history and was served the TL 45 copy 0000002D0000000A00000013. The record
at A/13000028 in that file is not a valid checkpoint → invalid xl_info in checkpoint record →
PANIC: could not locate a valid checkpoint record at A/13000028.

So the failure is: a replica that is down or behind across a timeline switch can never rejoin, because
the pre-switch timeline's WAL segment was not retained in the archive. It is unrecoverable by
construction — no amount of retrying, and no pg_rewind, can produce a WAL segment that does not exist.

This is a different mechanism from the coroot-db case in the issue body (which was pg_rewind
divergence on a demoted primary), but it has the same root: the WAL needed to reconcile across a
timeline switch is not retained. Proposed direction #2 above is therefore aimed correctly, but its
framing ("a replica that cannot pg_rewind") is too narrow — it does not describe today's failure, which
never reached pg_rewind at all. It failed in standby startup.

Why it recurs: timeline churn

The cluster was created 2026-06-03 and is on timeline 61 — 60 timeline switches in 63 days,
roughly one per day. Every switch is another chance for a replica that is briefly down to be stranded
permanently. Two contributing observations, both live:

umami-db-11 and umami-db-12 are 8-day-old pods carrying 20 and 18 in-place container restarts
(all graceful, exitCode 0; both last terminated at the 10:32Z promotion). Every primary restart is a
failover is a burned timeline. I have not diagnosed what restarts them that often — that is the
open question, and it is probably the highest-leverage one here.
The primary runs on an autoscaler node (autoscale-cx43-…). The cluster sets only
podAntiAffinityType: preferred — no nodeAffinity/nodeSelector keeping instances on stable
workers — so autoscaler scale-down forces a failover.

Suggested retarget

WAL retention across timeline switches is the load-bearing fix — HA replication slots, or archive
retention that outlives the staleness window a replica can plausibly accumulate. This is what makes
the failure survivable rather than terminal.
Cut the failover rate. Find the restart driver, and pin the DB off autoscaler nodes. ~1 timeline
switch/day on a 3-node cluster is the thing generating the exposure.
Detection is already tracked in fix(observability): a degraded CNPG replica is invisible until it wedges the merge queue #2978 / fix(observability): alert on a degraded CNPG cluster instead of only wedging the merge queue #2979 — a degraded replica was invisible for ~8h here.
The merge-gating half (direction Add ansible task to load rbd on boot #1) stands unchanged and is still worth doing independently.

Detail worth knowing for the runbook

The Cluster's phase read Cluster cannot proceed to reconciliation due to an error while interacting with plugins, naming dial tcp 10.99.190.72:9090: connect: connection refused. That string was stale —
the barman plugin endpoint was ready and completing pre-hooks every 3s throughout. Do not read that
phase as "the operator is stuck"; check the plugin endpoint and the plugin's own logs before acting on it.

Also: kubectl get cluster resolves to clusters.ksail.io on this cluster, not CNPG. The fully-qualified
clusters.postgresql.cnpg.io is required or you get a confusing NotFound.

Navigation Menu

Repository navigation

fix(observability): a single stuck coroot-db replica blocks every platform merge #2639

Problem

Proposed direction

Rough size

This recurred today on a different cluster — same chain, same outcome

Why this raises the priority

What I could not do

Second occurrence today on `umami-db` — and the mechanism is not what we recorded

The corrupt-WAL framing is wrong, and it matters for the fix

Why it recurs: timeline churn

Suggested retarget

Detail worth knowing for the runbook

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(observability): a single stuck coroot-db replica blocks every platform merge #2639

Description

Problem

Proposed direction

Rough size

Activity

coderabbitai commented on Jul 14, 2026

devantler commented on Aug 5, 2026

This recurred today on a different cluster — same chain, same outcome

Why this raises the priority

What I could not do

devantler commented on Aug 5, 2026

Second occurrence today on umami-db — and the mechanism is not what we recorded

The corrupt-WAL framing is wrong, and it matters for the fix

Why it recurs: timeline churn

Suggested retarget

Detail worth knowing for the runbook

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Second occurrence today on `umami-db` — and the mechanism is not what we recorded