Wedding database replica fails PostgreSQL checkpoint recovery

Description

A Wedding application database replica repeatedly fails PostgreSQL checkpoint recovery. The database remains below its declared readiness, reducing its resilience. Other members report ready in the bounded readback; those checks do not establish application availability or data integrity.

This is a separate affected database from the Coroot failure in #4662. Whether the two incidents share an initiating cause remains unproven.

Steps to observe

Read the Wedding database's current readiness and instance restart state without changing production.
Read a finite previous-startup log from the failed replica.
Correlate the checkpoint failure with the database operator's primary and readiness state.

Actual behavior

Bounded observations on 2026-10-08 show repeated replica startup failures because PostgreSQL cannot locate a valid checkpoint record. The operator reports incomplete database readiness. Backup and archiving conditions report success, which does not prove a successful restore or complete data integrity.

The initiating cause and relationship to preceding Platform recovery activity remain unknown. Raw logs, storage identifiers, checkpoint positions and operator commands stay in private evidence.

Expected behavior

Every declared replica completes recovery and stays ready in its intended replication role.

Recovery and acceptance

Preserve healthy members and retain failed-replica evidence; never reset the primary's WAL to clear this symptom.
Establish the initiating cause with checkpoint, WAL and storage evidence before attributing it to an upgrade or recovery.
Validate and perform a supported, targeted replica recovery through the reviewed operational path.
Verify complete database readiness, an actual Wedding application user path, and sustained health after the next protected deployment.
If a source or configuration defect is confirmed, deliver a regression test and durable repair through the normal reviewed pipeline. Immediate recovery alone does not close this issue.

Rough size: medium; diagnosis and targeted recovery, with a source repair if the evidence requires one.

Part of #2043.

Read-only evidence from the scheduled Claude run, 2026-10-08 22:45–22:55Z. No production change was made.

What the bounded reads establish, and what they rule out:

Not a storage fault. The failed member's volume is attached and healthy with its full replica count; nothing on it reads degraded.
Not the old primary. At the node loss around 21:45Z the member that later fails was a standby ("shut down in recovery"). The member that was primary rejoined cleanly: rewind found nothing to rewind and it streams from the new primary.
The failing step is a WAL segment taken from the archive. On every start the member fetches, from the object-store archive, the previous timeline's segment that holds its last restart point, and PostgreSQL then rejects the record at that position as not a checkpoint record (invalid xl_info in checkpoint record, followed by could not locate a valid checkpoint record). The member never reaches the point where it would stream from the primary, so it cannot heal by waiting — 16 identical restarts in an hour.
The two healthy members did not need that archived segment — they continued by streaming — so their readiness says nothing about whether the archived copy of that segment is good.

So the open question is narrow: is the archived copy of that one segment different from what the timeline's primary actually wrote? Either the archive holds a divergent or incomplete copy under that name, or this member's control data points at a position its own timeline never checkpointed. Comparing the archived segment's record at that position with the copy still in the failed member's own WAL directory answers it, and it needs the failed volume kept — which is one more reason not to delete it yet.

Why this matters beyond one replica: if the archived segment is wrong, a point-in-time restore through that window would fail the same way. "Archiving succeeded" and "last backup succeeded" cannot see that.

Related observation, same read: the five database clusters report timeline numbers between 50 and 222 — i.e. that many promotions each. For the 29-day-old wedding database that is roughly two failovers a day. Every promotion is another chance for an archive/timeline mismatch like this one, so the failover rate is worth its own look (#4649/#4650 address one cause).

What I did not do, and why. The supported recovery is to discard this member and let the operator clone a fresh one from the primary. That permanently deletes the failed member's volume, which is also the evidence above, and it is a production data deletion — not something an unattended run does. It is prepared for whoever picks this up with a human in the loop: confirm the primary and one standby are healthy, keep a copy (or snapshot) of the failed volume, then remove the failed instance's claim and pod so the operator re-creates it; read back three ready members afterwards.

Private detail (positions, segment names, object names) is in the run's operator notes, not here.

This replica now fails production deploys. Read from the log of the #4669 merge-queue deploy (run 37842323835, failed 22:54Z):

The deploy had to reboot the node holding a healthy standby of this database. With one member already down, the database's disruption budget allowed no eviction; the drain waited its full 45 minutes and the deploy failed. fix: unblock strict production boot-image recovery #4669 was removed from the queue.
Any deploy that must reboot a node holding a member of this database will fail the same way until the broken member is replaced.
The same deploy had a second, unrelated failure earlier: the newer deployment tool tried to reinstall cert-manager and was refused because Flux owns those objects. The Codex lane has a fix in draft (ksail#7639). fix: unblock strict production boot-image recovery #4669 should not be re-queued before that lands.

Asked the maintainer to replace the broken member (a production volume deletion; the engineer's cluster access is read-only): https://devantlertech.slack.com/archives/D0B1JG2EWLW/p1791500222613189 — sent 2026-10-08 ~23:10Z. The answer will be in that thread.

Collected the reviewed Replace Failed Wedding Standby run at current-main source 4622b50e1e11bfc2f4a9c7133e9cec3f7632f01c. Its protected repair step failed with REPAIR=HOLD: archiving, backup or fencing precondition is not clear. The combined message does not identify which prerequisite failed or establish successful replacement.

The exact reviewed procedure retains its volume, identity, backup and fencing guards. This receipt provides no completed recovery, sustained database-readiness or application-path proof. The current read-only health check still finds the failed replica and blocked reconciliation. The existing operator request was re-read with its thread and has no human reply.

No competing recovery or production mutation was initiated by this run. Retain the existing recovery dependency and collect the missing prerequisite evidence before a further protected attempt.

Why the reviewed repair held (run 37869788743) — read-only evidence, 2026-10-09 02:35–02:50Z. No production change was made by this run.

The combined HOLD message was the fence clause, not archiving or backup:

Both database conditions the guard reads have been continuously true since before the run (archiving since 2026-10-08 21:47Z, last backup since 03:00Z), so neither can have tripped it.
The repair got as far as its first two writes: the failed member's old volume now reads Retain, and the fence was written.
At 01:44:09Z the Git synchronisation re-applied the database object (its event reads "configured" and it is now the only manager recorded on the object's metadata). The fence annotation is gone and no trace of the repair's write remains. The repair's next observation, at 01:44:31Z, found the fence missing and held — 22 seconds later.

Cause: the repair wrote with the default command-line field manager. Flux takes over fields owned by that manager family and removes whatever Git does not declare, so the fence could never survive a reconciliation.

State now: primary and the other standby healthy (2 of 3 ready), the failed member's claim still attached and bound, its volume retained, no fence present. A second dispatch therefore starts from a clean precondition.

Fix: #4680 makes every repair write carry its own field manager, with a test that fails if one is sent without it. After it merges, the repair needs one fresh dispatch from main (the guard only accepts a first attempt of a new run).

The operator request in the maintainer's thread still has no reply and is still valid as the direct alternative.

Status 05:20 UTC — the broken standby is now the only thing stopping production deploys, and the order of the merge queue decides how soon it clears.

fix(deploy): let the credential probe run on a worker with almost no free memory #4666 (the credential-check fix) is deploying at the head of the queue. It got past the step where every deploy since 00:43 UTC was refused: at 05:03 UTC a worker was set aside for a restart, which no deploy tonight reached.
That worker holds wedding-db-3. With wedding-db-1 still crash-looping (90 restarts), the database's disruption budget allows no eviction (read live: 0 allowed), so the deploy is waiting on that one pod. I expect it to time out the same way fix: unblock strict production boot-image recovery #4669 did and leave the queue. I am not cancelling a production deploy in flight.
fix(recovery): keep the standby repair's isolation mark from being reverted #4680 (the repair tool fix from my previous comment) is next in the queue. It changes only the repair tool, like fix: preserve data when replacing the failed database standby #4677, which merged in about ten minutes without a deploy. So it should merge shortly after fix(deploy): let the credential probe run on a worker with almost no free memory #4666 leaves the queue.
Next step, mine: once fix(recovery): keep the standby repair's isolation mark from being reverted #4680 is on main, start the standby repair from main, read back 3 of 3 ready, then put fix(deploy): let the credential probe run on a worker with almost no free memory #4666 and the other removed pull requests back in the queue.

Nothing is needed from the maintainer for this path. The earlier request to remove the broken copy by hand still works as a shortcut and is unchanged.

Correction to my 05:20 UTC status: the standby repair has not been started, and this path does need the maintainer after all.

fix(deploy): let the credential probe run on a worker with almost no free memory #4666's deploy failed at 05:48 UTC exactly as expected: the drain of the worker holding wedding-db-3 hit its 45-minute limit on the database's disruption budget (run 37885114549). fix(deploy): let the credential probe run on a worker with almost no free memory #4666 left the merge queue; it is still open and clean.
fix(recovery): keep the standby repair's isolation mark from being reverted #4680 (the repair tool fix) is on main.
At 06:10 UTC I re-read the preconditions (2 of 3 ready, primary wedding-db-2, no fence, failed member unchanged since 2026-10-08 21:47:31Z) and tried to start the reviewed repair from main. The scheduled run's permission layer refused the dispatch as a change to shared production resources. I did not retry or route around it. No production change was made.

What unblocks it — either one, both the maintainer's:

Start the reviewed repair: Actions → Replace Failed Wedding Standby → Run workflow on main, with the typed confirmation the form asks for and the two identifiers read fresh from the cluster (the database object's UID and the UID of the wedding-db-1 pod). It keeps the old volume.
Or the manual step already asked for in Slack on 2026-10-08 (still unanswered, still valid).

Until then every deploy that must restart a worker holding a member of this database fails the same way. Six dependency updates are in the merge queue now and will each be tried; I am not cancelling them.

After either step I read back 3 of 3 ready, put #4666 back in the queue first, then the rest.

The maintainer asked this interactive session to complete the approved, storage-preserving repair. Fresh checks confirmed a healthy primary, a healthy standby, and a completed daily backup before protected run 37894948001.

The offline race tests passed. Recovery then stopped safely: REPAIR=HOLD: instance did not acknowledge fencing; storage unchanged. Fresh OIDC readback still confirms both healthy peers are unchanged, the failed replica is fenced, and its original claim is Bound and attached to retained storage. No old storage was deleted or detached.

Root cause of this HOLD: the metrics confirmation used a Pod-proxy route that cannot reach the exporter under the current namespace networking. PR #4681 is now merged: the exact reviewed head 126446d4bdbac86943cd5d0560b7dfee69c1506b landed through the normal queue as ddb8016b2d00f99cc59b4033ed99635a2dcdaefd at 08:47:45Z on 9 October. It replaces that observation with an exact-target, timed loopback read and adds a separately protected, read-only fencing proof plus a narrowly guarded continuation for the original pre-detachment HOLD. CI passed, including the real database-image probe and fresh merge-group standby safety tests. The queue correctly skipped deployment for this tooling-only change. CodeRabbit's ancillary findings have specific resolution records; all three providers became quota-limited, and the permitted current-head local review is posted with no P0/P1 findings.

The continuation plan has been exercised again against production with OIDC reads and reports PLAN=PASS no writes. Fresh OIDC reads after merge confirm unchanged target, primary and healthy-standby identities; all original claims remain Bound, attached and non-terminating, with the failed replica's original volume retained. The first-attempt read-only fencing proof passed from merged main at 08:50:30Z: FENCE_PROOF=PASS instanceAcknowledged=1 retainedClaims=1 noMutations=1. The merged source guard also passed. This proves the repaired observation path on the live, exact failed replica; it neither replaces that replica nor reports three Ready instances. The separate replacement continuation still awaits the requested specific approval and a fresh protected dispatch; the failed run will not be rerun.

The merge queue also has a dependency loop. The #4666 queue run passed the corrected memory-admission probe, but could not drain the healthy Wedding standby while the other replica remained failed. Current production recovery still fails at the unchanged memory probe before publishing a new manifest (run 37901868011, #4664). Ordinary queue priority allowed #4681 to merge with required checks intact so the storage-preserving repair can unblock that rollout. Merging the tooling is not a successful database repair.

Recovery is not complete. This issue remains open for the replacement, checkpoint/WAL root cause, and sustained application and deployment verification.

chore: agent-claim #4673 nonce=b697509b959732641a702b9b29fe7358

The maintainer explicitly approved the retained-volume standby continuation in this session. A fresh OIDC plan from current main passed without writes, and the recovery implementation is unchanged from the reviewed, merged version.

The first-attempt protected continuation run was dispatched at 11:18:54Z on 9 October from ce0be3613fb2b3e9176f671308cc3f5b22c7f3ec. Its safety tests passed and it acquired the shared production lock after the preceding heal finished. The continuation detached the old claim, deleted the exact failed pod and removed its repair fence. It did not complete a successful replacement. I cancelled this run after observing unexpected reuse of the retained claim; GitHub confirmed the terminal cancelled result at 11:37:41Z. No failed dispatch was rerun.

The operator created a join job that mounted the retained claim. Its log reports renaming the existing data directory and then completing a base backup onto the same volume. The old claim and volume still have their original identities and binding, the claim is annotated detached with its Cluster owner removed, and the volume still has Retain policy. Their continued existence is not proof that the original filesystem is untouched: the operator has written to it. The original healthy primary and standby remain Ready with unchanged identities; the original failed pod is absent. The cluster is still only two of three Ready, repeatedly adopting the completed join job instead of creating a fresh-storage standby.

The operator source explains the observed naming collision. In CloudNativePG v1.30.1, generateNodeSerial chooses the first name absent from the managed instance list. The detached claim is no longer counted there. The replica path creates its join job before creating storage, while the PVC creation helper accepts an existing, non-terminating claim without verifying ownership or excluding detached storage. This contradicts the recovery tool's fresh-storage assumption. The public main branch still has that allocator and v1.30.1 is the latest published release. The earlier operator audit and passing offline fixtures did not cover this behavior; they were not adequate live proof of that assumption.

Today's scheduled database backup completed. The application's login returns HTTP 200 with valid TLS. A read-only invalid-session lookup through the guest route returns the expected login redirect rather than the database-error response; development auth bypass is not configured. That exercises database session lookup, but not an authenticated guest's complete journey. The previous failed instance's log confirms an invalid checkpoint record; its initiating cause remains unknown.

Blocker: A supported, separately reviewed recovery path must prevent retained-claim name reuse before another production mutation. The current approval was used for one continuation, not a destructive retry or a waiver of fresh-storage verification. The completed join job and retained storage are left in place for investigation; cancelling the workflow does not suspend the operator's reconciliation loop.

This issue remains open for the operator naming-collision repair, three-Ready-replica readback with independently proved preserved storage, the full application path, checkpoint/WAL root cause and sustained verification after the next protected deployment. No cleanup, deletion of retained storage, success-gate relaxation or second mutating dispatch was performed.

The maintainer approved a narrowly scoped upstream suspected-bug issue after an assumption and value review.

Upstream necessity: this recovery requires detached forensic storage to remain outside replacement bootstrap. The public CloudNativePG v1.30.1 source accepts an existing, non-terminating claim without excluding detached claims, after choosing an instance name from managed status. A platform-only recovery workaround can mitigate this incident, but cannot change the standard operator/plugin behavior or provide that protection for the supported destroy --keep-pvc path. Upstream triage is therefore needed; the platform recovery remains separately owned here.

The external report will contain only the sanitized symptom, immutable public source references and an explicitly unverified synthetic reproduction recipe. There is no completed official-command reproduction or confirmed original-data loss. Current upstream main no longer uses the join-job flow, so the completed-job adoption symptom must not be presented as a current-main defect. The separate retained-claim path remains a source-based hypothesis.

The target is the public CloudNativePG dependency, unrelated to employment repositories. Its contribution and AI policies have been checked. No patch or PR will be submitted before maintainer agreement on the defect and approach. No further production mutation is authorized or performed by this reporting step, and the original checkpoint failure remains unexplained.

Upstream report: cloudnative-pg/cloudnative-pg#11632. It was filed as a suspected bug with the official-command reproduction explicitly pending; no fix or PR has been submitted.

#4700 merged at a9c39fa7a7b41770f1bb0955e1ced3b9514c0a97 after exact-head review and successful merge-group CI. The separately approved first-attempt recovery 38039956878 passed its regression tests and source/dispatch guards, then stopped at the first conditional pause request with SERVER_INVALID. This attempt is consumed and will not be rerun.

Fresh OIDC readback proves the original claim and completed join identities unchanged, the original PV still retained, its backing volume detached, both protected peers Ready, and disruption budgets unchanged. No pause or fence remains. Database readiness is still two of three; neither the database recovery nor its held rollout dependencies are delivered.

An offline native-client application of the seven pause operations against the observed Cluster succeeds and changes only the intended annotation. This rules out an impossible predicate in that observation, not a production race. An independent source audit agrees. The shorter connection has not resolved the rejection, so frequent object-version changes remain only a lead, not causal attribution.

The next bounded diagnostic should distinguish a genuine generic Invalid response from Invalid with structured causes, without publishing raw response contents or weakening any recovery guard. The client currently drops the rejection body; Kubernetes also deliberately omits the underlying JSON-patch error in its generic envelope (server implementation). Such a diagnostic would preserve HOLD and make no storage changes. A separately approved non-persisting dispatch is needed; this record grants no recovery retry.

Measured read-only at 10:44–10:52Z: the database object's version changes about four times a second. That alone is enough to explain the rejected pause request.

What I read (no writes, no dry-runs, nothing dispatched):

Three watches of the Cluster object, 30 s, 30 s and 12 s long, saw 135, 130 and 53 changes: about 4.4 a second. In the timed 30 s watch the longest quiet stretch was 0.47 s, and four gaps in ten were under 0.15 s.
One status field is all that changes. The operator writes the phase reason Creating replica wedding-db-1-join and removes it again, over and over, alternately about 0.13 s and 0.3–0.45 s apart. Spec generation, primary, target primary, instance count, ready count, phase and annotations stayed the same throughout.
The operator's own log shows the loop behind it. In 20 s it logged Creating new Job 45 times and Job already exists, adopting it 45 times for this database: 2.25 reconciles a second with two status writes each, which matches the measured rate.
The pause request pins the exact version it observed (test /metadata/resourceVersion). The API server answers any failed JSON-patch operation, a failed test included, with the generic 422 and drops the reason: k8s.io/apiserver v0.36.5, pkg/endpoints/handlers/patch.go, the error branch after patchObj.Apply. That is the response the attempts reported as SERVER_INVALID.

What that does to a single attempt, computed from the 130 timed gaps:

Time from the server reading the object to the server applying the patch	Chance the version is still the same
0.1 s	57%
0.2 s	28%
0.3 s	8%
0.4 s or more	0%

Before #4700 that window held a process start, a new connection and an extra read, so the first two attempts had close to no chance. Since #4700 it is one round trip from the runner, so the 09:02Z attempt had odds somewhere around even and lost once. One loss does not show that #4700 failed.

This is inference from a measured rate and the server source. Nothing I can read shows which operation the server rejected.

Two things follow for the next step:

A dry-run diagnostic races the same version. It will most likely return the same generic 422 and spend an approval to say little. If one runs anyway, the telling detail is whether details.causes is empty (a failed patch operation) or filled (validation or admission).
While the version is pinned, every single-shot attempt is a gamble. Two ways out, both a change to a recovery guard and so the maintainer's call, for the session that owns this to weigh:
- Try again on 422 only, inside one approved run. A 422 means nothing was written. Read again, check every guard again, send once more, a bounded number of times (eight tries at 40% each is 98%). Never after a timeout, a 5xx or a conflict. The deploy's own node claim already works this way: a rejected version test is retried a bounded number of times against a fresh read, while every captured fact still holds (node_claim_preconditions_still_hold in scripts/refresh-flux-ghcr-auth-safety.sh).
- Pin what must not change instead of the version. Replace the version test with tests on metadata.generation (the whole spec), the ready count and the phase, beside the identity, primary, size and annotation tests already there. The request would then no longer notice unrelated status changes.

At this rate the operator writes the object about 380,000 times a day while the join is stuck.

I changed nothing and opened no pull request. The Claude lane is not working this issue.

The separately approved first-attempt diagnostic 38047663598 succeeded on merged main 0528f7d70bc956c795642020b677d6377a401f8e: PAUSE_DIAGNOSTIC=PASS requestAccepted=1 persistedMutations=0. It made one server-side dry-run request and no storage changes.

Fresh OIDC readback still shows two of three Ready, the same stable primary and two protected Ready peers, no pause, the completed join retained, the old PV Bound with Retain, unchanged disruption budgets and a recent successful backup. Diagnostic acceptance proves that observation only; it does not identify the predicate rejected by an earlier attempt or establish recovery.

#4703 preserves UID/version CAS and freezes the protected Cluster state while allowing up to five newly guarded requests within one minute per pause/resume phase, only after verified non-persisting native rejections. Every new request re-proves the complete recovery guards. Ambiguous responses, throttling, changed guards, diagnostics and storage deletions never enter that loop. The maintainer has approved one new recovery after this fix passes tests, exact-head review, CI and merge. Consumed attempts will not be rerun.

#4704 merged at 37381888ba9960dfcc8db137af205ef92298a80e after exact-head review and successful merge-group CI. The separately approved first-attempt recovery 38051182296 passed its regression suite, accepted the conditional pause, and then stopped with REPAIR=HOLD: required pods read failed. The attempt is consumed and will not be rerun. Storage cleanup was not reached.

There is a concrete acknowledgment-parser defect. Fresh operator logs contain the exact expected pause message, matching database identity fields and valid new reconciliation IDs, at info level. The recovery parser requires warning, so those records cannot satisfy its acknowledgment gate. The stop occurred at the five-minute acknowledgment observation bound; that timing is consistent with the final read exhausting its context, but the sanitized read error alone does not establish a transport cause. In the pinned source, the controller calls Warning, while the logging wrapper implements it through Info. Source method names were not sufficient evidence of the emitted log protocol.

Fresh OIDC readback proves the recovery-owned pause remains, both protected peers retain their identities and roles and are Ready, the primary is unchanged, and database readiness remains two of three. The completed join and detached claim identities are unchanged. The original PV is still Bound with Retain, its backing identity is unchanged and detached, all observed backups are completed, and disruption budgets are unchanged. All nine nodes are Ready and none are cordoned. Nothing material was deleted by this attempt.

The earlier production deployment 38049633538 held the shared lock while a disruption-budget-protected drain waited on this degraded database. It stopped at its bounded drain and released its node ownership and synchronization lease. No disruption budget was bypassed.

The next repair needs a regression-tested correction for the actual acknowledgment protocol and a separately reviewed, explicitly approved continuation of this existing owned pause. It must retain freshness, identity, leader, replay, source and storage guards; accepting arbitrary log levels or replaying this consumed dispatch is not a repair. The maintainer has been asked for that continuation approval. Database recovery and the held deployment dependencies remain undelivered.

The pause-acknowledgment repair has shipped in #4706 at 9f3783e2ad6e3c7f454fa9ca97c58e6784c7b211. Merge-queue CI passed. The merged recovery sources match the reviewed head; both complete race-enabled recovery suites and go vet pass on merged main. The parser delivery #4705 is closed and Done. No recovery was dispatched by this merge, and the merge-group deployment job was skipped.

Fresh OIDC-only reads still show the same two Ready protected database peers, the same primary, and the exclusively recovery-owned pause. The completed join Job and old claim identities are unchanged. The old PV remains Bound/Retain with the same reservation, and its backing volume remains present, detached and nonterminating. The observed backups are completed; the disruption budgets are unchanged. This is preserved HOLD, not restored redundancy or normal reconciliation.

The observed nodes are Ready. An active protected deployment currently owns a drain cordon. Its ownership marker identifies CD run 38053946044, whose production job is still running. I have not cancelled that deployment, removed its cordon, or bypassed a disruption budget.

The previous recovery attempt remains consumed. A new continuation must be separately approved and reviewed for the already-owned pause, obtain a fresh acknowledgment from the bound leader without resuming first, and preserve the same source, identity, storage, peer, backup and deletion limits. #4706 intentionally does not admit the paused state into the initial recovery path. The parent #4673 remains open until the actual recovery and its live/application checks pass.

PR #4708 merged after green current-head CI, a finding-free full CodeRabbit review and a successful merge-queue run. The merged source matches the reviewed source.

Fresh OIDC reads and the merged command's read-only plan establish the unchanged two-peer hold, exclusively recovery-owned pause, completed join, retained detached old storage, unchanged disruption budgets and complete backups. The read-only plan reports PLAN=PASS completedJoin=1 retainedVolumes=1 noWrites=1.

The separately approved single continuation is run 38063574034, first attempt on current main. Its recovery job is serialized behind the active deployment by the shared production lock; nothing was cancelled or bypassed. The earlier consumed run will not be replayed.

Recovery remains pending. This parent stays open until the protected run and separated live readbacks establish three healthy instances, fresh replacement storage, preserved old backing storage and unchanged protected peers, backups and budgets.

The database recovery has succeeded. PR #4708 is merged, and its separately approved first-attempt protected continuation completed successfully with REPAIR=PASS readyInstances=3 retainedVolumes=1 separatedSamples=2.

Independent OIDC-only readbacks at 16:11:29Z and 16:13:03Z both passed: all three instances are Ready, the replacement has fresh bound storage, and the original backing storage remains retained and detached. The primary and healthy replica identities and roles are unchanged. Every previously observed completed backup and its specification is preserved, archiving is healthy, disruption budgets are unchanged, and node-health checks pass. The recovery-owned pause and acknowledgment marker are gone.

The only removed resources were the stalled completed join identities and the old claim identity. The old backing volume was not deleted. A fresh read-only application lookup returned its expected invalid-code response without a database error or guest-data changes.

The next protected production deployment is already running on the merged main revision. This issue stays open for the remaining acceptance check: sustained database and application health after that deployment. The separate Flux dependency is being observed, not treated as clean or manually overridden.

Independent check of the upstream report cloudnative-pg/cloudnative-pg#11632: the behaviour it describes is real in v1.30.1, shown by running upstream's own code. It is not yet shown on a real cluster.

The maintainer asked whether that report was real or invented. It had been filed from source reading only, with its reproduction marked "not executed".

What was run

Upstream at tag v1.30.1 (2a35abb4628f209d149825ef3c38011e0701ff2f), with one added test that uses upstream's own fake-API test harness. It builds a healthy three-instance cluster, then calls the real destroy command code and the reconciler's real replica-creation step (reconcileMissingInstance).

Step	Observed
Before	3 instances: `pg-1 pg-2 pg-3`
`Destroy(…, "pg-2", keepPVC=true)`	claim `pg-2` is the same object, has no owner, `cnpg.io/pvcStatus=detached`
Reconcile, round 1	instance list `pg-1 pg-3`; next serial chosen is 2; Job `pg-2-join` is created and its pod template mounts claim `pg-2`
Join Job marked succeeded, rounds 2 and 3	instance list still `pg-1 pg-3`; serial 2 chosen again; the existing Job is adopted; the claim is still the original object, unowned and `detached`; no `pg-2` pod exists
Control: same cluster, `keepPVC=false`	the replacement gets a new claim owned by the cluster, and the instance list returns to `pg-1 pg-2 pg-3`

Both specs pass, and upstream's own tests in that package still pass beside them.

Why it happens

Since upstream PR 10548 (2026-06-18) a new instance takes the first free serial instead of the next number, so a destroyed instance's name is handed out again.
The instance list is built only from claims the cluster owns. A detached claim is not owned, so its name counts as free.
Claim creation treats "already exists and is not terminating" as success. It checks neither the owner nor the detached marker. An existing Job, by contrast, is refused unless the cluster owns it.
The reused claim is never given an owner, so it never counts as an instance. The reconciler therefore picks the same serial on every loop and adopts the same finished Job. That is the "Creating new Job" / "Job already exists, adopting it" pair measured here on 2026-10-10.
In a cluster with contiguous serials, destroying any one instance gives that same serial back, so this is the normal case, not a corner case.

What this does not show

No real cluster, kubelet, volume mount or pg_basebackup ran. That the Job mounts the retained claim is shown by its pod template naming it.
What the join does to the retained data was read, not executed: upstream's EnsureTargetDirectoriesDoNotExist moves an existing data directory aside when its control file is valid and removes it when it is not.
The full Reconcile function was not driven end to end; the gates before the replica-creation step were read.
Upstream has not replied (no comment, no label, as of 2026-10-10 18:50 UTC). The reproduction was posted there at 18:52 UTC, with the maintainer's go-ahead: [Bug]: Detached PVCs may be reused after destroy --keep-pvc cloudnative-pg/cloudnative-pg#11632 (comment)

Relevance here

Production runs the operator at 1.30.1. Any procedure that detaches a claim and expects it to be left alone, including kubectl cnpg destroy --keep-pvc, will have the replacement pick that claim up again on this version.

The added test and its forwarders (patch against v1.30.1)

Run with go test ./internal/cmd/plugin/destroy/ -v -ginkgo.focus REPRO -ginkgo.v.

diff --git a/internal/cmd/plugin/destroy/zz_repro_test.go b/internal/cmd/plugin/destroy/zz_repro_test.go
new file mode 100644
index 0000000..053be84
--- /dev/null
+++ b/internal/cmd/plugin/destroy/zz_repro_test.go
@@ -0,0 +1,244 @@
+package destroy
+
+import (
+	"context"
+	"fmt"
+
+	batchv1 "k8s.io/api/batch/v1"
+	corev1 "k8s.io/api/core/v1"
+	apierrs "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
+	apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
+	"github.com/cloudnative-pg/cloudnative-pg/internal/cmd/plugin"
+	"github.com/cloudnative-pg/cloudnative-pg/internal/controller"
+	"github.com/cloudnative-pg/cloudnative-pg/internal/scheme"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/postgres"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/reconciler/persistentvolumeclaim"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/utils"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// Local reproduction, not part of upstream. It runs the real destroy command
+// code and the reconciler's real replica-creation step against a fake API
+// server holding a healthy three-instance cluster.
+var _ = Describe("REPRO detached claim after destroy --keep-pvc", func() {
+	const (
+		ns   = "repro"
+		name = "pg"
+	)
+
+	type world struct {
+		cli     client.Client
+		r       *controller.ClusterReconciler
+		cluster *apiv1.Cluster
+	}
+
+	originalUID := func(serial int) types.UID { return types.UID(fmt.Sprintf("original-claim-%d", serial)) }
+
+	newWorld := func(ctx context.Context) *world {
+		sch := scheme.BuildWithAllKnownScheme()
+		cli := fake.NewClientBuilder().WithScheme(sch).
+			WithStatusSubresource(&apiv1.Cluster{}).
+			WithIndex(&corev1.PersistentVolumeClaim{}, controller.ReproOwnerKey, func(o client.Object) []string {
+				if owner, ok := controller.IsOwnedByCluster(o); ok {
+					return []string{owner}
+				}
+				return nil
+			}).
+			WithIndex(&batchv1.Job{}, controller.ReproOwnerKey, controller.ReproJobOwnerIndex).
+			WithIndex(&apiv1.Backup{}, ".spec.cluster.name", func(o client.Object) []string {
+				return []string{o.(*apiv1.Backup).Spec.Cluster.Name}
+			}).
+			Build()
+
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: ns},
+			Spec: apiv1.ClusterSpec{
+				Instances:            3,
+				StorageConfiguration: apiv1.StorageConfiguration{Size: "1Gi"},
+			},
+		}
+		cluster.SetDefaults()
+		Expect(cli.Create(ctx, cluster)).To(Succeed())
+		cluster.TypeMeta = metav1.TypeMeta{Kind: apiv1.ClusterKind, APIVersion: apiv1.SchemeGroupVersion.String()}
+
+		for serial := 1; serial <= 3; serial++ {
+			pvc, err := persistentvolumeclaim.Build(cluster, &persistentvolumeclaim.CreateConfiguration{
+				Status:     persistentvolumeclaim.StatusReady,
+				NodeSerial: serial,
+				Calculator: persistentvolumeclaim.NewPgDataCalculator(),
+				Storage:    cluster.Spec.StorageConfiguration,
+			})
+			Expect(err).ToNot(HaveOccurred())
+			cluster.SetInheritedDataAndOwnership(&pvc.ObjectMeta)
+			pvc.UID = originalUID(serial)
+			pvc.Status.Phase = corev1.ClaimBound
+			Expect(cli.Create(ctx, pvc)).To(Succeed())
+
+			pod := &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("%s-%d", name, serial), Namespace: ns},
+				Status: corev1.PodStatus{
+					Phase:      corev1.PodRunning,
+					Conditions: []corev1.PodCondition{{Type: corev1.PodReady, Status: corev1.ConditionTrue}},
+				},
+			}
+			cluster.SetInheritedDataAndOwnership(&pod.ObjectMeta)
+			Expect(cli.Create(ctx, pod)).To(Succeed())
+		}
+
+		plugin.Client = cli
+		plugin.Namespace = ns
+
+		return &world{
+			cli: cli,
+			r: &controller.ClusterReconciler{
+				Client:   cli,
+				Scheme:   sch,
+				Recorder: record.NewFakeRecorder(120),
+			},
+			cluster: cluster,
+		}
+	}
+
+	// observe does what the reconciler does at the start of a loop: list the
+	// objects it owns and derive the instance list from the owned claims.
+	observe := func(ctx context.Context, w *world) (
+		corev1.PodList, corev1.PersistentVolumeClaimList, batchv1.JobList, postgres.PostgresqlStatusList,
+	) {
+		Expect(w.cli.Get(ctx, client.ObjectKeyFromObject(w.cluster), w.cluster)).To(Succeed())
+		w.cluster.TypeMeta = metav1.TypeMeta{Kind: apiv1.ClusterKind, APIVersion: apiv1.SchemeGroupVersion.String()}
+
+		pvcs, err := w.r.ReproGetManagedPVCs(ctx, w.cluster)
+		Expect(err).ToNot(HaveOccurred())
+		jobs, err := w.r.ReproGetManagedJobs(ctx, w.cluster)
+		Expect(err).ToNot(HaveOccurred())
+		var pods corev1.PodList
+		Expect(w.cli.List(ctx, &pods, client.InNamespace(ns))).To(Succeed())
+
+		persistentvolumeclaim.EnrichStatus(ctx, w.cluster, pods.Items, jobs.Items, pvcs.Items)
+
+		var st postgres.PostgresqlStatusList
+		for i := range pods.Items {
+			st.Items = append(st.Items, postgres.PostgresqlStatus{Pod: &pods.Items[i]})
+		}
+		return pods, pvcs, jobs, st
+	}
+
+	claim := func(ctx context.Context, w *world, serial int) corev1.PersistentVolumeClaim {
+		var pvc corev1.PersistentVolumeClaim
+		Expect(w.cli.Get(ctx, client.ObjectKey{Namespace: ns, Name: fmt.Sprintf("%s-%d", name, serial)}, &pvc)).
+			To(Succeed())
+		return pvc
+	}
+
+	joinJob := func(ctx context.Context, w *world, serial int) batchv1.Job {
+		var job batchv1.Job
+		Expect(w.cli.Get(ctx, client.ObjectKey{Namespace: ns, Name: fmt.Sprintf("%s-%d-join", name, serial)}, &job)).
+			To(Succeed())
+		return job
+	}
+
+	claimsMountedBy := func(job batchv1.Job) []string {
+		var names []string
+		for _, v := range job.Spec.Template.Spec.Volumes {
+			if v.PersistentVolumeClaim != nil {
+				names = append(names, v.PersistentVolumeClaim.ClaimName)
+			}
+		}
+		return names
+	}
+
+	It("reuses the detached claim and never restores the instance", func(ctx SpecContext) {
+		redirectStdoutToGinkgoWriter()
+		w := newWorld(ctx)
+
+		observe(ctx, w)
+		GinkgoWriter.Printf("before destroy: instances=%d names=%v\n", w.cluster.Status.Instances, w.cluster.Status.InstanceNames)
+		Expect(w.cluster.Status.InstanceNames).To(Equal([]string{"pg-1", "pg-2", "pg-3"}))
+
+		// The official command's own code path: kubectl cnpg destroy pg 2 --keep-pvc
+		Expect(Destroy(ctx, name, "pg-2", true)).To(Succeed())
+
+		kept := claim(ctx, w, 2)
+		GinkgoWriter.Printf("after destroy --keep-pvc: claim pg-2 uid=%s owners=%d %s=%s\n",
+			kept.UID, len(kept.OwnerReferences), utils.PVCStatusAnnotationName, kept.Annotations[utils.PVCStatusAnnotationName])
+		Expect(kept.UID).To(Equal(originalUID(2)))
+		Expect(kept.OwnerReferences).To(BeEmpty())
+		Expect(kept.Annotations[utils.PVCStatusAnnotationName]).To(Equal(persistentvolumeclaim.StatusDetached))
+
+		for round := 1; round <= 3; round++ {
+			pods, pvcs, jobs, st := observe(ctx, w)
+			Expect(w.cluster.Status.InstanceNames).To(Equal([]string{"pg-1", "pg-3"}))
+			Expect(w.cluster.Status.Instances).To(Equal(2))
+
+			seenInstances, seenNames := w.cluster.Status.Instances, w.cluster.Status.InstanceNames
+			serial := w.r.ReproGenerateNodeSerial(w.cluster)
+			Expect(serial).To(Equal(2))
+
+			_, err := w.r.ReproReconcileMissingInstance(ctx, w.cluster, pods, pvcs, jobs, st)
+			Expect(err).To(MatchError(controller.ErrNextLoop))
+
+			job := joinJob(ctx, w, 2)
+			Expect(claimsMountedBy(job)).To(ContainElement("pg-2"))
+
+			// The claim the join Job mounts is the retained one: same object,
+			// still not owned by the cluster, still marked detached.
+			kept = claim(ctx, w, 2)
+			Expect(kept.UID).To(Equal(originalUID(2)))
+			Expect(kept.OwnerReferences).To(BeEmpty())
+			Expect(kept.Annotations[utils.PVCStatusAnnotationName]).To(Equal(persistentvolumeclaim.StatusDetached))
+
+			// No instance Pod is ever created for it.
+			var pod corev1.Pod
+			err = w.cli.Get(ctx, client.ObjectKey{Namespace: ns, Name: "pg-2"}, &pod)
+			Expect(apierrs.IsNotFound(err)).To(BeTrue())
+
+			GinkgoWriter.Printf(
+				"round %d: instances=%d names=%v next serial=%d job=%s mounts=%v succeeded=%d claim uid=%s owners=%d status=%s pod pg-2 exists=false\n",
+				round, seenInstances, seenNames, serial, job.Name,
+				claimsMountedBy(job), job.Status.Succeeded, kept.UID, len(kept.OwnerReferences),
+				kept.Annotations[utils.PVCStatusAnnotationName])
+
+			if round == 1 {
+				// What the kubelet reports once the join has finished.
+				job.Status.Succeeded = 1
+				Expect(w.cli.Status().Update(ctx, &job)).To(Succeed())
+			}
+		}
+	})
+
+	It("control: without --keep-pvc the replacement gets a new claim and is counted again", func(ctx SpecContext) {
+		redirectStdoutToGinkgoWriter()
+		w := newWorld(ctx)
+
+		Expect(Destroy(ctx, name, "pg-2", false)).To(Succeed())
+
+		pods, pvcs, jobs, st := observe(ctx, w)
+		Expect(w.cluster.Status.InstanceNames).To(Equal([]string{"pg-1", "pg-3"}))
+		Expect(w.r.ReproGenerateNodeSerial(w.cluster)).To(Equal(2))
+
+		_, err := w.r.ReproReconcileMissingInstance(ctx, w.cluster, pods, pvcs, jobs, st)
+		Expect(err).To(MatchError(controller.ErrNextLoop))
+
+		fresh := claim(ctx, w, 2)
+		Expect(fresh.UID).ToNot(Equal(originalUID(2)))
+		owner, owned := controller.IsOwnedByCluster(&fresh)
+		Expect(owned).To(BeTrue())
+		Expect(owner).To(Equal(name))
+
+		// Once the storage layer binds the new claim, it is an instance again.
+		fresh.Status.Phase = corev1.ClaimBound
+		Expect(w.cli.Status().Update(ctx, &fresh)).To(Succeed())
+		observe(ctx, w)
+		GinkgoWriter.Printf("control: new claim owner=%s uid=%q instances=%d names=%v\n",
+			owner, fresh.UID, w.cluster.Status.Instances, w.cluster.Status.InstanceNames)
+		Expect(w.cluster.Status.InstanceNames).To(Equal([]string{"pg-1", "pg-2", "pg-3"}))
+	})
+})
diff --git a/internal/controller/zz_repro_forwarders.go b/internal/controller/zz_repro_forwarders.go
new file mode 100644
index 0000000..78d2812
--- /dev/null
+++ b/internal/controller/zz_repro_forwarders.go
@@ -0,0 +1,57 @@
+package controller
+
+import (
+	"context"
+
+	batchv1 "k8s.io/api/batch/v1"
+	corev1 "k8s.io/api/core/v1"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/postgres"
+)
+
+// Forwarders for a local reproduction, not part of upstream. They add no logic
+// of their own: each one only calls the unexported function, or exposes the
+// unexported value, that it names.
+
+// ReproOwnerKey is the field index key the reconciler lists owned objects by.
+const ReproOwnerKey = pvcOwnerKey
+
+// ReproJobOwnerIndex is the reconciler's own Job owner index function.
+func ReproJobOwnerIndex(o client.Object) []string { return jobOwnerIndexFunc(o) }
+
+// ReproGetManagedPVCs calls getManagedPVCs.
+func (r *ClusterReconciler) ReproGetManagedPVCs(
+	ctx context.Context, cluster *apiv1.Cluster,
+) (corev1.PersistentVolumeClaimList, error) {
+	return r.getManagedPVCs(ctx, cluster)
+}
+
+// ReproGetManagedJobs calls getManagedJobs.
+func (r *ClusterReconciler) ReproGetManagedJobs(
+	ctx context.Context, cluster *apiv1.Cluster,
+) (batchv1.JobList, error) {
+	return r.getManagedJobs(ctx, cluster)
+}
+
+// ReproGenerateNodeSerial calls generateNodeSerial.
+func (r *ClusterReconciler) ReproGenerateNodeSerial(cluster *apiv1.Cluster) int {
+	return r.generateNodeSerial(cluster)
+}
+
+// ReproReconcileMissingInstance calls reconcileMissingInstance with the
+// resources the reconciler would have gathered.
+func (r *ClusterReconciler) ReproReconcileMissingInstance(
+	ctx context.Context,
+	cluster *apiv1.Cluster,
+	pods corev1.PodList,
+	pvcs corev1.PersistentVolumeClaimList,
+	jobs batchv1.JobList,
+	instancesStatus postgres.PostgresqlStatusList,
+) (ctrl.Result, error) {
+	return r.reconcileMissingInstance(
+		ctx, cluster, &managedResources{instances: pods, pvcs: pvcs, jobs: jobs}, instancesStatus,
+	)
+}

The remaining recovery acceptance check is complete: database and application health have stayed healthy through a subsequent protected production deployment.

PR #4710 is merged, and its protected deployment succeeded. Independent read-only checks at 21:14:56Z and 21:16:25Z on 10 October passed, with another preservation check at 21:26:34Z. All three database instances are healthy, archiving and backups are healthy, and the primary, bound storage, completed backups and disruption budgets are preserved. The failed member's original backing storage remains retained and detached; it has not been deleted or reset.

The application's real database-backed invalid-code lookup also passed at 21:29:59Z, returning the expected rejection without creating a session or changing guest data.

This closes the production recovery, following the successful protected continuation in run 38063574034. The separate upstream retained-claim report remains cloudnative-pg/cloudnative-pg#11632. That bookkeeping defect does not establish the cause of the original checkpoint failure; the retained evidence is preserved for that distinction.

Navigation Menu

Repository navigation

Wedding database replica fails PostgreSQL checkpoint recovery #4673

Description

Steps to observe

Actual behavior

Expected behavior

Recovery and acceptance

14 remaining items

What was run

Why it happens

What this does not show

Relevance here

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Wedding database replica fails PostgreSQL checkpoint recovery #4673

Description

Description

Steps to observe

Actual behavior

Expected behavior

Recovery and acceptance

Activity

devantler commented on Oct 8, 2026

devantler commented on Oct 8, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

devantler commented on Oct 9, 2026

14 remaining items

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

devantler commented on Oct 10, 2026

What was run

Why it happens

What this does not show

Relevance here

devantler commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions