Write and drill the procedure for reclaiming a released volume #4442

🤖 Generated by the Agentic Engineer

Problem

Once volumes are retained, every deleted claim leaves a Released volume behind. It keeps costing until someone reclaims it. An hcloud volume is billed for as long as it exists. A Longhorn volume keeps the space of its replicas on the storage nodes.

The ADR for #3369 describes how to bring such a volume back and how to discard it, but the procedure has not been run on this cluster. One part of it is easy to get wrong: restoring a manifest does not restore its data. The recreated claim is given a new, empty volume unless the released one is reserved for it first.

Proposal

This is step 4 of the rollout in docs/adr/storage-retention.md. Add the procedure to the disaster-recovery runbook and drill it once for each driver:

• Bring a volume back: reserve the released volume for the claim that should own it, by leaving only that claim's namespace and name in the volume's claim reference and removing the old claim's uid. Then let the claim be created.
• Discard a volume: set that one volume to Delete, and confirm the storage behind it is gone.
• Include one claim that an operator owns (a CloudNativePG instance), and record what the operator needs on the claim to adopt it.

Reserve the retained volume before the replacement claim or its operator can provision an empty substitute. Record and verify the requested storage class, access modes, volume mode, selector, capacity and compatible workload placement under the PV's node affinity. A reservation alone does not prove that a workload can use the volume.

The drill changes volumes, which the agent's cluster credential cannot do. Deliver it as a job reviewed in Git, or have the maintainer run the commands.

How long a released volume is kept is a spend decision and stays with the maintainer. This issue only makes the options and their cost visible.

Acceptance criteria

• docs/dr/runbook.md contains the procedure, with the commands that were actually run.
• A released Longhorn volume and a released hcloud volume were each brought back with their data intact. The evidence is on this issue.
• One of each was discarded, and the storage behind it was confirmed gone.
• For the operator-owned claim: either the adoption worked and the required labels and annotations are recorded, or the issue records that it is not supported and that restoring from backup is the path.
• The procedure covers both binding modes (Immediate and WaitForFirstConsumer).
• The published evidence contains no host names, addresses or volume identifiers.
• Each recovery seeds a known data checksum before deletion, preserves the same PV UID and storage handle, binds the replacement PVC to that PV, mounts it in a scheduled workload, and reads the same checksum. A Pending claim, an empty replacement volume or source inspection alone fails the drill.
• The private evidence records the old/new claim UIDs, reciprocal binding and actual driver-side deletion readback. Public evidence contains only the sanitized outcome. No production workload is used as a disposable drill fixture.

Rough size: medium.