Repository navigation
Gate OpenBao upgrades on unsealed recovery proofs #4756
Description
🤖 Generated by the Agentic Engineer
User Story
As a platform operator,
I want OpenBao upgrades to prove that every required server is initialized, unsealed and healthy without blocking bootstrap or recovery,
So that an apparently Ready pod cannot authorize the next replacement while the secrets service is unavailable.
Acceptance Criteria
- Regression tests reject sealed, uninitialized, missing, stale-version and incomplete member observations, including Kubernetes Ready responses that are not unsealed-health proof.
- Exercise cold start and recovery, preserving the initialization dependency: do not make the bootstrap readiness check require an unseal operation that itself waits for readiness.
- Render the proposed chart and actual production post-renderers; prove retained storage, security configuration, unseal behavior and replace-on-delete strategy are preserved. Do not update a pinned regression or checksum alone to permit an upgrade.
- A reviewed protected rollout replaces at most one member at a time and requires fresh positive peer, unseal and configured storage-health proof before the next step. Retain an explicit recovery path.
- Exact-head review and native CI pass; after merge, record two separated live readbacks for the approved release and tenant secret delivery. Missing API/quorum evidence remains unknown, never inferred from desired images or Ready alone.
Context
The released 2.6.3 repair tracked by #4391 is now present on all three running servers: separated OIDC-only reads on 11 October report version 2.6.3, initialized and unsealed, exactly one active member, and the latest StatefulSet revision. The original automatic partition plan was superseded by the reviewed replace-on-delete recovery in #4710. No additional replacement is needed to deliver that patch.
The separate next-chart upgrade in #4535 remains blocked on the bootstrap/recovery-safe health proof described in its existing hold. This issue preserves that unfinished obligation; it does not make the upgrade ready or claim fresh cold-start, negative-health or quorum API evidence.
Part of #2653. Reliability value: protect the secrets service and unblock controlled future upgrades. This is captured follow-up work, not a started implementation or a delivered safety gate.
Activity
devantler commentedon Oct 15, 2026
Source-backed refinement for the upgrade gate: keep bootstrap readiness separate from permission to replace a server. The current server configuration intentionally accepts sealed and uninitialized health responses so the downstream initialization job can run. The initialization job reaches individual members, initializes only the designated first member and then unseals peers. Making that Helm readiness probe strict would recreate the dependency cycle, not deliver this issue.
Recommended implementation order:
Keep the actual chart/post-renderer and pinned native regression in #4535 authoritative; changing their version or checksum alone remains insufficient. The same controlled rollout can later establish runtime adoption for #4912, whose config cleanup is already merged but whose running servers still use their startup copies.
This is refinement of the existing blocker, not a started rollout or a change to its unchecked acceptance criteria. No production credential, server, probe, storage or authorization was changed.
devantler commentedon Oct 16, 2026
Refinement, 16 October: moved to Ready, first step split out as #4962
First step, #4962: an offline judge of captured server health with negative controls. It takes no production action and adds no replacement path. It follows step 1 of the order recorded above. Steps 2 and 3 (exercising native servers, and the protected single-server procedure) will be filed once the judge's evidence format is settled.
The acceptance criteria above are unchanged.