fix: preserve data when replacing the failed database standby

.github/workflows/replace-wedding-standby.yaml

@@ -0,0 +1,107 @@
+name: Replace Failed Wedding Standby
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - .github/workflows/replace-wedding-standby.yaml
+      - scripts/replace-wedding-standby/**
+  merge_group:
+  workflow_dispatch:
+    inputs:
+      confirm:
+        description: "Type exactly: retain-volumes-replace-failed-standby"
+        required: true
+        type: string
+      cluster_uid:
+        description: Current Wedding Cluster UID from an OIDC read
+        required: true
+        type: string
+      pod_uid:
+        description: Current failed standby Pod UID from an OIDC read
+        required: true
+        type: string
+
+permissions: {}
+
+jobs:
+  tests:
+    name: 🛡️ Validate standby replacement safety
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    steps:
+      - name: 📑 Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - name: ⚙️ Setup Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: Test complete observations and conditional writes
+        run: go test -race -count=1 -timeout=5m ./scripts/replace-wedding-standby
+
+  repair:
+    name: 💒 Replace failed standby without deleting its storage
+    if: ${{ github.event_name == 'workflow_dispatch' }}
+    needs: tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    environment: prod
+    permissions:
+      contents: read
+    concurrency:
+      group: prod-deploy
+      cancel-in-progress: false
+      queue: max
+    env:
+      WEDDING_REPAIR_CONFIRM: ${{ inputs.confirm }}
+      GH_TOKEN: ${{ github.token }}
+    steps:
+      - name: Require main and explicit confirmation
+        shell: bash
+        env:
+          CONFIRM: ${{ inputs.confirm }}
+          DISPATCH_REF: ${{ github.ref }}
+        run: |
+          set -euo pipefail
+          [[ "${DISPATCH_REF}" == refs/heads/main && "${CONFIRM}" == retain-volumes-replace-failed-standby && "${GITHUB_RUN_ATTEMPT}" == 1 ]]
+      - name: 📑 Checkout reviewed main revision
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          ref: ${{ github.sha }}
+      - name: Require the current main recovery source
+        run: ./scripts/verify-prod-recovery-source.sh "${GITHUB_SHA}"
+      - name: ⚙️ Setup Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          cache: false
+      - name: 🔑 Restore production kubeconfig
+        shell: bash
+        env:
+          KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
+        run: |
+          set -euo pipefail
+          test -n "${KUBE_CONFIG}"
+          umask 077
+          mkdir -p ~/.kube
+          printf '%s' "${KUBE_CONFIG}" >~/.kube/config
+          chmod 600 ~/.kube/config
+      - name: 🎯 Select stable production API endpoint
+        shell: bash
+        env:
+          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
+        run: ./scripts/use-prod-stable-api-endpoint.sh
+      - name: Preserve old storage and prove the replacement healthy
+        shell: bash
+        env:
+          CLUSTER_UID: ${{ inputs.cluster_uid }}
+          POD_UID: ${{ inputs.pod_uid }}
+        run: |
+          set -euo pipefail
+          go run ./scripts/replace-wedding-standby --execute --cluster-uid "${CLUSTER_UID}" --pod-uid "${POD_UID}"

scripts/replace-wedding-standby/README.md

@@ -0,0 +1,64 @@
+# Failed Wedding standby replacement
+
+This command repairs only the failed `wedding-db-1` replica. Its default mode is
+an OIDC-only, read-only plan. Mutations require the first, explicitly confirmed
+main dispatch of `Replace Failed Wedding Standby`, behind the `prod` environment
+and the shared production deployment lock. Publishing or merging this procedure
+does not authorize running it; the maintainer must approve the particular repair.
+
+The procedure requires three observed database instances, a stable healthy primary,
+one other healthy replica, healthy archiving, and a completed cluster-bound backup
+no older than 24 hours. Current Cluster and failed Pod UIDs must be provided. The
+operator must be the audited, fully rolled-out CloudNativePG 1.30.1. An unknown
+read, different volume layout, active database Job, existing fence, changed
+identity or changed primary stops the repair.
+
+The failed replica's single existing volume is changed to `Retain`. The replica
+is fenced, and its own metrics must acknowledge that PostgreSQL is fenced before
+the claim is detached. Detachment follows the operator's
+[`destroy --keep-pvc` procedure](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.1/internal/cmd/plugin/destroy/destroy.go):
+remove only the verified Cluster owner reference and mark the claim `detached`
+before deleting the exact failed Pod with UID and resource-version preconditions.
+The operator's [fenced-candidate ordering](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.1/pkg/postgres/status.go)
+keeps the fenced replica out of primary election. No PVC, PV, backup, primary,
+healthy replica, disruption budget or namespace is deleted or relaxed.
+
+The fence is removed only after the old Pod is absent and the same protected
+peers remain healthy. Success requires two separated, complete samples showing
+three Ready instances, fresh replacement storage, healthy archiving, and the old
+claim and volume still bound to their original identities. Only then may ordinary
+maintenance proceed. This proves database repair, not overall platform health.
+
+## Read-only plan
+
+From current main, get the current Cluster and failed Pod UIDs with OIDC reads,
+then run:
+
+```sh
+go run ./scripts/replace-wedding-standby --cluster-uid "$CLUSTER_UID" --pod-uid "$POD_UID"
+```
+
+The command reports `PLAN=PASS no writes` only after complete observations pass.
+The protected workflow requires confirmation
+`retain-volumes-replace-failed-standby` and those same UIDs. It rechecks that its
+validated checkout is current main before every conditional write. Never rerun
+a failed dispatch: a new attempt requires fresh inspection and a new approval
+for any continuation beyond the original procedure.
+
+## Failure and retained data
+
+A failure returns `REPAIR=HOLD`, never health clearance. Earlier successful
+steps may remain: a Retain volume, a fenced failed replica, or a detached claim.
+There is no automatic destructive cleanup, reverse ownership change, WAL reset,
+or repeated mutation. Inspect the current state with OIDC reads and prepare a
+reviewed continuation. Keep the old claim and volume for diagnosis; retirement
+is a separately approved data-lifecycle operation.
+
+## Offline validation
+
+```sh
+go test -race -count=1 ./scripts/replace-wedding-standby
+```
+
+The workflow runs this package on its pull request, merge group and explicit
+dispatch before any production credentials are restored.