Replace OpenBao servers one at a time through a protected, health-gated procedure #4970

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Part of #4756. This is the third step of that issue's recommended order. It depends on the health judge (#4962) and on the native exercise from the second step.

Problem

An OpenBao chart or server update needs servers replaced one at a time, and today nothing proves the secrets service healthy between replacements. Kubernetes Ready does not prove it, because the readiness probe accepts sealed servers on purpose. The chart update in #4535 and the runtime check in #4912 both wait on this.

Outcome

A protected, main-only maintenance workflow replaces at most one server per step. It modifies nothing until a fresh capture of every server passes the health judge and an authenticated Raft membership and quorum read agrees. It verifies recovery and tenant secret delivery before the next server, and it has an explicit, documented path back.

Acceptance criteria

• Dispatch-only on main, behind the protected environment and the shared production-deploy concurrency group, following the existing OpenBao audit-capacity procedure.
• Before each replacement: current source and deployment binding, retained-storage evidence, a PROVEN health verdict for every server, and an authenticated membership and quorum read. Any missing or failed read holds the procedure; nothing is inferred from Ready.
• After each replacement: the replaced server's new pod identity, a fresh PROVEN verdict, quorum restored, and tenant secret delivery confirmed before the next step.
• No credential, address or raw response reaches the public log.
• A documented recovery path, and a runbook page under docs/operations/.

Rough effort: High.