[devantler]

🤖 Generated by the Agentic Engineer

Why

The wedding database now backs up with its own dedicated key, but the old shared backup access can only be switched off once we have actually watched the shared backup store turn that dedicated key away. Until now nothing could check that against the key that is really deployed, so the isolation rested on how the key was set up rather than on what the store enforces.

What

Adds a manually started production check that tries to list, read and write the shared backup store with the dedicated key and passes only when every attempt is explicitly refused for lack of permission. It first proves the key works on its own backups and that the shared target really exists, so a typo or a network fault can never pass as a refusal. The credential-rotation runbook now requires this check for every new wedding backup key.

Part of #3253

👉 After merge/promotion: dispatch "Verify Wedding Backup Denial" on main; its result decides whether shared access can be retired.

[devantler]

🤖 Generated by the Agentic Engineer

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 56 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 82f26c7e-06c7-456b-9333-aad0c2c993db

📥 Commits

Reviewing files that changed from the base of the PR and between 84b87e0 and 53e8ade.

📒 Files selected for processing (10)

• .github/workflows/ci.yaml
• .github/workflows/verify-wedding-backup-denial.yaml
• docs/dr/restore-drill.md
• docs/dr/runbook.md
• docs/dr/velero-cnpg.md
• scripts/tests/s3-access-denied-stub/main.go
• scripts/tests/test-mirror-wedding-backup-runtime.sh
• scripts/tests/test-verify-wedding-backup-denial.sh
• scripts/verify-wedding-backup-denial-pod.sh
• scripts/verify-wedding-backup-denial.sh

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devantler]

🤖 Generated by the Agentic Engineer

Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)

Reviewed commit: 09f6a45d72da3f8ece531a6923f884719b63e0b2

• CodeRabbit: rate limited on this PR at 2026-10-02T15:15:40Z ("Review rate limited"; its summary states "Next included review available in 56 minutes"). Lane health reads LIMITED until 2026-10-02T16:17:48Z.
• Codex: usage limit, "You have reached your Codex usage limits for code reviews" (platform#4327, 2026-10-01T18:54:43Z). No retry window; only the maintainer can lift it.
• Cursor Bugbot: usage limit, "Bugbot couldn't run - usage limit reached" (platform#4327, 2026-10-01T18:56:08Z). No retry window.

Scope. Independent review passes covered the whole diff, and every finding was fixed at the root cause before this head:

• Full diff at 335f59b0: 2 P1. Verdicts trusted mc's text over what happened, and the classifier was only tested on hand-written output. Fixed: write judged at the destination, read by local copy, exit 0 always granted; pinned mc run against a refusing stub in CI.
• Full diff at 53e8adef: 0 P0/P1, 6 P2. Fixed: IP redaction, created-object-only cleanup, probe named on timeout, wording, stub request log, required real-mc case.
• Deltas 53e8adef..4eb2eeaa and 4eb2eeaa..f37da93f: 0 P0/P1. P2s fixed: stub log readable by the runner, listing check requires a real list-type=2 request.
• f37da93f..09f6a45d72da3f8ece531a6923f884719b63e0b2 (reviewed here): the listing check now also requires the catalogue prefix and no longer pipes under pipefail; an IPv4 address glued to a hex token is redacted; the probe-prefix check fails when empty. Verified against the request lines the stub recorded in CI and against sample redaction inputs.

Evidence from the pinned client (runtime test at f37da93f, real mc against a stub answering every request with R2's 403 AccessDenied): the stub received the catalogue listing (list-type=2&prefix=cnpg%2Fwedding-db%2F), a HEAD for the object and a PUT of the probe key. mc exited 1 for all three, and the proof classified them as refusals. mc reports the copy's refusal without an error code, and prints an upload success record before the refusal; the proof relies on neither.

Safety. With the dedicated credential, the only write is the run-owned probe key. With the shared credential, the proof only lists, plus a non-recursive removal of that key when it was not refused. Operator-visible output is redacted (endpoint host, IPv4, long hex), and only the exact receipt and marker as the final two lines pass. The workflow is dispatch-only, main-only, prod environment, prod-deploy lock, contents: read.

Verdict: no P0/P1 findings

[devantler]

🤖 Generated by the Agentic Engineer

Readiness at 09f6a45d72da3f8ece531a6923f884719b63e0b2.

• Tested: CI green at this head (rollup SUCCESS). The denial suite runs twice: on the host, and inside the pinned mc and BusyBox images, where the real pinned mc also runs list, read and write against a stub that answers every request with R2's 403 AccessDenied. The stub's request log shows the catalogue listing (list-type=2, catalogue prefix), a HEAD for the object and a PUT of the probe key, and the proof classifies all three refusals as denied. More than 30 mutations of the proof and runner were each caught by the suite.
• Reviewed: local review round GREEN self@09f6a45d (local-review-verdict.sh). CodeRabbit was rate limited on this PR, and Codex and Bugbot are on usage limits; the review records the evidence for each lane. Four independent review passes found 2 P1 and several P2, all fixed at the root cause; 0 unresolved threads.
• Tried as a user: the workflow runs only from main, and starting its pod before merge would change production, so the pre-merge trial was read-only. The runner's own preconditions hold on live prod: the Cluster archives through wedding-db-dedicated, both ObjectStores match the reviewed destination, Secret and committed endpoint, and no denial-proof pod or ConfigMap exists.
• After merge: dispatch Verify Wedding Backup Denial on main with confirm=verify-wedding-backup-denial. Its receipt and DENIAL OBSERVED are the denial gate in Verify the dedicated wedding backup identity, retire shared access, and update the DR runbooks #3253.