Recover a Pending database replica blocked by missing volume clients #4745

🤖 Generated by the Agentic Engineer

Problem and evidence

A database replica can remain Pending after it is scheduled to a worker that does not yet have the volume client. The current CloudNativePG controller does not repair that already-scheduled case, so incomplete database redundancy prevents the reviewed client-placement maintenance from starting. A fresh scoped read on 11 October confirmed the storage-client attachment failure, a healthy remaining primary, retained claims, and an absent second replica.

This is different from the historical checkpoint failure in #4662 and the telemetry identity issue in #4720. Neither explains this fresh attachment failure.

Proposed bounded recovery

Add a default-read-only, protected recovery for the affected metadata database. After the active deployment finishes, freshly verify exact primary, replica, worker and retained-storage identities. Temporarily prevent scheduling on only the worker missing the client, and submit a normal eviction only for an exact still-Pending non-primary replica. Use UID and resource-version preconditions, preserve default graceful termination and the existing disruption budgets, retain every claim and volume, and restore only the recovery's owned scheduling change on every exit.

The original placement repair in #4718 remains the durable fix. Its full-health maintenance gate must not be weakened. No live recovery may run before regression tests, exact-head review, CI and merge.

Acceptance

• Failed, missing, stale or incomplete observations authorize no write; a running or primary Pod is never evicted.
• Concurrent identity, scheduling, storage or source changes reject the conditional write without a forced fallback.
• Interrupted execution leaves a recoverable ownership record; cleanup restores only its own cordon and reports incomplete cleanup as failure.
• Production runs serialize with other deployment writers and use the reviewed current main revision.
• The primary and all original claim/volume identities remain unchanged; full redundancy is observed twice before declaring recovery.
• After fix(storage): stage safe volume-client maintenance #4719 and fix: allow volume clients on compute workers #4744 deploy, verify eligible-worker client registration and continued database health before fix(talos): stop the audit flood by selecting AppArmor correctly #4197 proceeds.

Priority: High. Audience: platform services and their operators. This resolves confirmed live redundancy loss and unblocks existing delivery; it is an incident-recovery WIP exception, not new feature scope. Effort: Medium, including failure-path tests, review, deployment and separated live verification.

Part of #2043.