Repository navigation
Conversation
Production OpenBao has run on two of three servers since openbao-0 was lost: the partitioned rolling update makes the StatefulSet controller build every ordinal below the partition from the stored current revision, which predates the baseline-worker pins, and the storage-placement policy refuses that Pod. Return production to the chart's OnDelete strategy. A missing Pod is then created from the newest revision, which the canary has run unsealed, and no running server is replaced. Part of #4391 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai review |
✅ Action performedReview finished.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (8)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details🧰 Additional context used🧠 Learnings (1)📚 Learning: 2026-08-11T12:41:28.242ZApplied to files:
🔇 Additional comments (8)
📝 WalkthroughWalkthroughOpenBao StatefulSet patches change from partitioned Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to A lost server should be recreated without replacing running servers, and the configured placement and startup paths align with that goal. End-to-end rejoining remains unproven in checked-in tests, so retain the tracked readback as follow-up; no concrete merge blocker is established. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. (6 skipped: 6 unsupported.)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Evaluation at
Review: CodeRabbit completed at this commit with no actionable comments. |
Cross-lane dependency readback: the failed #4197 deployment was followed by an unsuccessful Heal Prod run. Recovery stopped before publication because the Coroot baseline-context guard requires every operator-owned Coroot template to be fully ready. Fresh OIDC reads still find an incomplete Coroot server rollout; #4652 tracks its connection-exhaustion repair. The same before-publication guard applies to the standard deployment path for this OpenBao repair. This is a separate live blocker from OpenBao’s stale revision and must be resolved without bypassing the guard. I am leaving this actively owned PR unchanged and continuing the Coroot work. |
Why this left the merge queue (02:53Z): its merge-group run 37719268317 failed in Head Blocker: #4652 | authority | last-verified 2026-10-08: Coroot server unready, deploys refused | asked slack 2026-10-08 |
Why this left the merge queue at 02:53 UTC, and why it is not re-queued yet. Its merge-group deploy stopped before publishing at the Coroot readiness check, not at anything in this change. The same check has refused every deploy since 01:52 UTC (#4652, #4658). Re-queuing now would fail the same way, so this waits on the Coroot restart asked of the maintainer; it goes back in first once Coroot is fully ready. Head is unchanged at |
Reviewed commit: The Docstring Coverage advisory is not a missing public API contract. The Go change renames one
|
@coderabbitai full review |
✅ Action performedFull review finished. |
Reviewed commit: The deployment blocker has cleared. The authorized recovery restored the secret service and observability service to their desired replica counts. The database reports healthy, with both standbys streaming and participating in synchronous replication. Existing database storage and backups were preserved. The unchanged pre-publication Coroot guard passes. Its post-reconciliation guard also passes: all six templates have the required baseline fields, are ready, and remain stable across three separated samples. The platform live-health check returns The reviewed OnDelete behavior was exercised during recovery: the missing server was recreated while the two running servers were retained. CodeRabbit's fresh full review is green at this head. Its identical test-docstring advisory retains the specific same-head refutation in the resolution record; there are no new findings or unresolved threads. Required checks are present and green, and no other local session holds this PR. I am clearing the stale blocker record and returning this unchanged head to the protected merge queue. Queue admission is not completion; I will verify deployment and the merged head afterward. |
Blocker: #4673 | upstream | last-verified 2026-10-10: still open. Unchanged head 41d374f is clean with checks green, and the fault it repairs is still live: read at 10:46Z, the third secrets-store server is still refused at admission and the configuration job still waits for it, so infrastructure and apps do not reconcile. It cannot be delivered yet: a deploy from main still fails at the credential probe (run 38042707843, 09:48Z), and the deploy that carried the probe repair (#4666) stopped at the database drain. Order once the database is back and no recovery run is active: #4666 first, then this, then read back that the server is recreated and unsealed and record it on #4391. |
The follow-up protected deployment confirmed that the two existing fixes must #4710 carries the exact intended source changes from this PR and its companion, |
Closing as superseded by #4710, which merged at 21:17 UTC on 2026-10-10 and carries every file this pull request changes together with the two transition fixes its first deploy attempt showed were also needed. Read back at 21:20 UTC: the platform health check reports OK, all three secrets-store servers are running, and the configuration job completed. Nothing in this pull request remains to deliver. |
Why
The production secrets store has been running on two of its three servers since 7 October 23:52 UTC, and the third cannot come back on its own. The staged-rollout setting added last week makes Kubernetes rebuild a lost server from an outdated stored template, which the storage-placement guard correctly refuses. Until this is fixed, losing one more server stops the secrets store, production reports unhealthy, and every deploy in the merge queue is held up behind it.
What
Production goes back to the setting it used before the staged rollout: a server is replaced only when it is deleted or lost, and the replacement is built from the newest configuration. That is the same release the first upgraded server has run healthy and unsealed since 7 October. No running server is replaced by this change; the remaining older server stays as it is until the staged rollout issue decides how to replace it.
Part of #4391
👉 After merge/promotion: the deploy should bring the third server back by itself; the readback to confirm it is recorded on #4391.