Preflight one OpenBao server replacement from live evidence without changing anything #5836

🤖 Generated by the Agentic Engineer

Part of #4970. This is the first, read-only slice of the protected single-server OpenBao replacement procedure.

Problem

The health judge (#4962) and the quorum judge (#5036) decide from captured evidence, but nothing captures that evidence from production. Until something does, the replacement procedure in #4970 has no input, and an operator asking "is it safe to replace a server right now?" can only look at Kubernetes Ready, which accepts sealed servers on purpose.

Outcome

A dispatch-only workflow on main, behind the protected prod environment and the shared prod-deploy concurrency group, that changes nothing. It captures each expected server's health and one authenticated Raft configuration read, runs both judges on them, and reports one verdict for replacing a named server: PROVEN, HOLD or UNKNOWN.

Approach

• A dedicated read-only identity: a service account in the openbao namespace with no workload, bound to a Kubernetes-auth role whose only policy reads sys/storage/raft/configuration. The workflow mints a short-lived token for it, logs in, reads once and revokes the OpenBao token.
• A Go collector that reaches each server through its own port-forward, so a health body is read from the server it names, binds each read to the pod UID it saw before and after, and writes the two capture files privately for the judges.
• The expected server names and release come from the reviewed render, never from the capture.

Acceptance criteria

• Nothing in the workflow can modify a pod, StatefulSet or OpenBao state; the role can read only the Raft configuration.
• The two judges, not the collector, decide the verdict; any failed read or changed pod UID is UNKNOWN.
• No token, address or raw response reaches the public log; only the judges' verdict lines and closed stage names do.
• Regressions run the collector against a fake cluster and fake servers, including a server whose pod UID changes mid-capture and a failed login.

Rough effort: Medium.