Repository navigation
Description
π€ Generated by the Agentic Engineer
Evidence
CKV_K8S_40 (containers should run as a high UID to avoid host conflict) is the last checkov
finding on k8s/: a full local reproduction of MegaLinter's invocation reports
kubernetes β¦ Failed checks: 2, and both are the vault-backup workloads β
Job.openbao.vault-snapshot-init and CronJob.openbao.vault-snapshot. #2904 dispositioned them as a
scoped, deferred exception; this issue is the durable fix that retires it.
Three facts were measured on the live cluster while dispositioning them, and together they decide the
shape of the fix:
- User namespaces are NOT in effect at these sites, so the risk is a host UID. The
add-user-namespacesrule matches on the namespace label
pod-security.devantler.tech/user-namespaces: enabled, which live carries exactly
backstage,crossview,headlamp,homepage,umami,whoamiβ notopenbao
(kubectl get ns openbao -o jsonpath='{.metadata.labels}'returns no such label). The live
CronJob'shostUsersis unset and the running pod reportsHOSTUSERS <none>. So container
UID 100 maps straight through to host UID 100.
β οΈ This refutes the basis security(k8s): CKV_K8S_40 needs one risk acceptance, not 11 per-site constraintsΒ #2904 originally proposed for accepting the finding β the
user-namespace rollout is real, but it does not reach these two workloads. - The snapshot lands
-rw-------. From the nightly job's ownls -l:
-rw------- 1 openbao openbao 459839 Aug 17 03:30 /snapshots/openbao-20260817-033025.snap.
Owner is UID 100, group GID 1000 β and there is no group read bit. - That is why the
mirrorcontainer inherits 100.minio/mcmounts/snapshotsreadOnly
and bakes no uid-100 entry of its own, so it reads the snapshot purely as its owner. Drop it
to a high UID today and the off-cluster R2 mirror stops being able to open the file β silently,
because the container would still start.
Why this needs its own slice
fsGroup: 1000 is not enough on its own. It fixes ownership and mode at mount time; the snapshot
is created afterwards by bao operator raft snapshot save at the process umask, which is what
produces the 0600 above.
Smallest useful change, in order
- Make the snapshot group-readable so access no longer depends on being its owner β set the
umask (orchmod 0640) in the snapshot script, in bothjob.yamlandcron-job.yaml. Verify on
the next nightly run that the file lands-rw-r-----and the mirror still uploads. - Then move the UIDs, exactly as
vault-config/job.yamlalready does: pod-level
runAsUser: 65532, withrunAsUser: 100set on the openbao container only, keeping
runAsGroup: 1000/fsGroup: 1000. The threevault-snapshotswriters share one RWO PVC and
must move together βjob.yaml,cron-job.yamland.github/workflows/dr-rebuild.yaml's helper
pod.
Step 2 touches the DR restore path, so it wants a watched window: run a snapshot, confirm the mirror
uploaded, and confirm dr-rebuild can still read an existing snapshot.
Acceptance criteria
- The nightly snapshot lands group-readable and the R2 mirror still uploads it.
- All three
vault-snapshotswriters run a high pod UID, with 100 scoped to the openbao
container only. -
dr-rebuildreads a snapshot written before the change and one written after it. - The deferred
checkov.io/skipforCKV_K8S_40is removed from both manifests and checkov
still reports 0 failed checks.
Rough size: small-to-medium, two steps, second one watched.
Blocker: CLEARED 2026-08-20 β acceptance criterion 1 is met on a real snapshot, so step 2 is
now actionable. Verified live against the 2026-08-20T03:30:27Z CronJob run, the first since
#3246 reconciled:
- the snapshot lands group-readable β the job's own
ls -lreports
-rw-r----- 1 openbao openbao 459839 Aug 20 03:30 /snapshots/openbao-20260820-033021.snap,
against the-rw-------this issue recorded before the change; - the mirror still uploads it β
mctransferred449.06 KiBof that file to
backup/platform-backups/openbao-snapshots/, and the post-upload listing shows it present.
mirror container read the
snapshot as its owner (it still inherits uid 100), so what is proven is that the file is now
group-readable and the pipeline is unbroken β not yet that a high-UID mirror can read it. That
is step 2's own watched verification. The supporting mechanism was measured separately in #3258
(fsGroup supplies the group via supplementary groups and the volume setgid bit), so the remaining
risk is the sequencing rather than the premise.
Step 1 merged as #3246 at 2026-08-19T17:41Z and is already reconciled into the live cluster (the
deployed CronJob.openbao.vault-snapshot carries chmod 0640 "$SNAP"). But the last successful
snapshot was 2026-08-19T03:30:27Z β before that merge β so no group-readable snapshot exists yet
and the mirror has never been exercised against one.
The CronJob schedule is 30 3 * * *, so the first snapshot that proves criterion 1 is the
2026-08-20T03:30Z run. Starting step 2 before then would move the writers' UIDs while the only
evidence that the mirror can still read the file is the reasoning in this issue β which is exactly
the silent-failure mode described above (the mirror container would still start, and simply stop
being able to open the snapshot).
Next action on/after 2026-08-20T03:30Z: confirm the new snapshot lands -rw-r----- and that the
mirror container still uploaded it; then criterion 1 is met and step 2 becomes actionable.
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- StatusShow more project fieldsβ Done
Activity
coderabbitai commentedon Aug 18, 2026
π Related PRs
#3116 - fix(umami): run the tenant-provisioning container on a read-only root filesystem [merged]
#3203 - security(vault-backup): disposition the last two CKV_K8S_40 findings as one deferred exception [merged]
#3246 - security(vault-backup): make the raft snapshot group-readable [merged]
#3249 - security(umami): raise the provisioning CronJob above the UID/GID floor [merged]
#3257 - security(k8s): disposition trivy KSV-0020 on the vault-config Job, guarded per container [merged]
π Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.π§ͺ Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:π¬ Have feedback or questions? Drop into our discord!
devantler commentedon Aug 19, 2026
Scope note: this issue owns 14 of the 16 remaining KSV-0020/0021 findings
Measured on
main@4d2a3dc4withscripts/megalinter-scan-counts.sh --trivy-only(trivy 0.74.0),with a per-target breakdown of the same scan:
k8s/bases/infrastructure/vault-config/job.yamlk8s/bases/infrastructure/vault-backup/job.yamlk8s/bases/infrastructure/vault-backup/cron-job.yamlThe other 2 are umami's, tracked separately in #3221 (which has been rescoped to umami only β its
original text mis-stated the cause as a missing
runAsUser/runAsGroup; all these files set both,and the findings fire because the values are
<= 10000).In
vault-config/job.yamlthe 6 are per-container overrides rather than the pod default: the podruns
runAsUser: 65532/runAsGroup: 1000, andfetch-snapshot,store-keys,vault-initandvault-configrestate the lower openbao100:1000identity. So the sequencing this issue describeshas to cover the container-level fields too, not just the pod securityContext.
Expected delta when this lands: trivy 24 β 10, with KSV-0020 7 β 1 and KSV-0021 9 β 1.
chore: agent-claim #3202 nonce=6928e2af13c220116d8c8bfd0d0c3035
devantler commentedon Aug 19, 2026
Step 1 shipped; two measured notes that change step 2's scope
Step 1 is in #3246 β
chmod 0640on the snapshot in both writers, so the mirror can readit as GROUP rather than as OWNER. Deliberately not
Fixes: the UID move is still open here.1.
chmod, notumaskβ the issue offers them as alternatives, but only one worksThe issue suggests "set the umask (or
chmod 0640)". umask cannot do this job: it only everCLEARS permission bits, never adds them. The snapshot lands
0600under what is presumably a normal0022umask, which meansbaois requesting0600explicitly β and no umask value can widen anexplicit mode. Lowering the umask would have been a silent no-op that still looked like a fix.
chmodis correct under either hypothesis, so it is what shipped.2.
dr-rebuildis a third file-CREATOR, not just a third UID β step 2 needs its mode tooThe issue lists
dr-rebuild.yaml's helper pod among the three writers that must move UID together.Measured while working step 1: that helper also creates a file on the shared PVC β
.github/workflows/dr-rebuild.yaml:428doesmc cp "backup/$R2_BUCKET/openbao-snapshots/$NEWEST" "/snapshots/$NEWEST", downloading a snapshotfrom R2 for the restore path.
So it is not only a UID to raise: the snapshot it writes is read afterwards by the vault-config
restore Job, and once the UIDs diverge that read depends on the downloaded file's mode exactly the
way the mirror's read depends on the nightly one. Step 2 should set the mode there as well, or the
restore path acquires the same owner-coupling step 1 just removed from the backup path.
Acceptance criteria status
ls -l "$SNAP"in both scripts prints the mode, so the job log is the evidence.dr-rebuildreads a pre- and post-change snapshot β step 2.checkov.io/skipforCKV_K8S_40removed β step 2 (the skip is deliberately still in place; step 1 does not change any UID).devantler commentedon Aug 20, 2026
The GID question in step 2 is already answered β #3258 measured it.
The plan here keeps
runAsGroup: 1000at pod level while raising the UIDs, which leaves themirrorcontainer inheriting a low primary GID it has no image-baked reason for. That is the samedefect the UID split fixes, one field over, and it is why the four
vault-backupKSV-0021 findingssurvive this issue as written.
#3258 established, with a paired control, that cross-container access does not depend on a
shared primary GID:
fsGroupsets the volume group and the setgid bit, so files inherit group1000 whatever GID their writer runs as, and
fsGroupjoins every container's supplementary groups.With the setgid bit a file written at gid 65532 lands group 1000 and a uid-100/gid-1000 reader can
read it; without it the file lands group 65532 and that reader cannot.
So step 2 can raise
runAsGroupto 65532 alongsiderunAsUser, pinning100/1000to the openbaocontainer only β no separate investigation needed. #3264 does exactly that for
vault-config.be mirrored before the UIDs move. Only the GID design question is settled.
fix(vault-config): disposition the openbao KSV-0021 residual and guarβ¦
chore: agent-claim #3202 nonce=17152995379db305f8bb8f301ec9479a
chore: agent-claim #3202 nonce=47d7c2068e0963e21d176ca1b659cb13
10 remaining items
test(vault-backup): assert the effective GROUP of every snapshot writer
devantler commentedon Aug 21, 2026
#3279 is merged (
05:55:11Z), and here is what it actually moved β measured, not asserted.Like-for-like, one binary. Both numbers come from
scripts/megalinter-scan-counts.sh --trivy-onlyrun locally with the same trivy 0.74.0, scanned from the repository root with the ignorefile (the
helper warns that narrowing the target to
k8s/silently inflates the count β it does; my firstattempt did exactly that and reported hundreds of ignorefile-suppressed
KSV-0039rows):main@b83c9c20(pre-merge)92221a1aSo #3279 retires 2 real KSV-0020 findings. That comparison is the point of running both sides
myself: the issue's standing "8" was measured under trivy 0.73.0, and the helper explicitly warns
that a rule-set difference between versions can look like backlog movement. Comparing my 6 against
that 0.73.0 figure would have been an invalid before/after β the movement is real precisely because
both sides were measured with the same binary on the same day.
The residual 6, named per container (
--exit-code 1 --ignorefile .trivyignore.yaml --skip-dirs tests --config-data .trivy/data .):cron-job.yaml(vault-snapshot)snapshotrunAsUser> 10000cron-job.yamlsnapshot,mirrorrunAsGroup> 10000job.yaml(vault-snapshot-init)snapshotrunAsUser> 10000job.yamlsnapshot,mirrorrunAsGroup> 10000Both residual classes are load-bearing, which is why they did not fall out with the rest:
snapshotat UID 100. That is the openbao image's bakedopenbao:x:100:1000user, which
runAsUser/runAsGroupmerely restate; the same identity already carries a scopedCKV_K8S_40disposition in these files for checkov. Raising it means the openbao binary no longerruns as the user its image provisions.
runAsGroup/fsGroup1000. This is exactly the mechanism security(vault-backup): raise the snapshot writers off the host UIDΒ #3279 just built: thesnapshot is written
0640andmirrorreads it as a group member rather than by being its owner.Raising the GID above 10000 without changing the image would break the handoff this issue exists to
create.
I am deliberately NOT filing "disposition the residual 6" as ready work. Whether the GID can be
raised depends on whether the openbao image tolerates a
runAsGroupother than its baked 1000, and Ihave not tested that. That is a real fix if it holds and a scoped
.trivyignore.yamldisposition onlyif it does not β and the ladder says try the fix first. Getting that order right needs one experiment,
not an assumption.
Recording this against #2787 as well, since
REPOSITORY_TRIVYcannot leaveDISABLE_ERRORS_LINTERSuntil this residual is at zero.
chore: agent-claim #3202 nonce=dffa4f3d76bf0c656b9322fe5265417c
devantler commentedon Aug 21, 2026
Step 2 started in #3281 β the mirror half of it.
Measured, not inherited. Re-ran the repo's own scan (
scripts/megalinter-scan-counts.sh --trivy-only, trivy 0.74.0, scan root the repository root, repo flags) againstmainata0537f33: 6 misconfigurations β KSV-0021 Γ4, KSV-0020 Γ2, all onvault-backup'sjob.yamlandcron-job.yaml. After #3281: 4.The 6 split into two genuinely different causes, which is what makes half of this shippable now:
mirror(minio/mc)runAsGroup: 1000mcbakes no group, so 1000 was never its identitysnapshot(openbao)openbao:x:100:1000The mirror half needed no experiment:
fsGroupsupplies cross-container access through supplementary groups and the volume setgid bit, so reading the 0640 snapshot #3279 established never depended on the mirror's primary gid.vault-configalready does exactly this (podrunAsGroup: 65532,fsGroup: 1000) and pinned the mechanism there against a paired control (#3258).The
snapshothalf is NOT ready and I did not guess at it. Raising UID 100 means running the openbao binary as an identity its own image does not define, and the honest test is to run that pinned digest under a non-baked uid/gid and observe it. I could not do that this run β the local container runtime is not available, and I will not mutate prod to find out. So it stays open with the experiment named rather than asserted.devantler commentedon Aug 21, 2026
Correcting my previous comment. I wrote that the mirror half "needed no experiment". That was wrong, and #3281 is closed because of it.
scripts/tests/test-vault-snapshot-group-readable.shpins podrunAsGroup: 1000deliberately, and states the mechanism it is protecting: the mirror reads the snapshot on its primary GROUP entry alone β "a pod-level property, which is why this does not depend on the volume's setgid bit."My justification leaned on
vault-config, which does run podrunAsGroup: 65532withfsGroup: 1000. It differs exactly where it matters: its shared paths areemptyDir;/snapshotsis a PVC. The setgid + supplementary-group behaviour #3258 verified was verified on emptyDir, and the guard's author chose primary-GID sharing here precisely so this path would not depend on it. I treated the two as equivalent on the one point where they are not.So the residual is 4, not 2 β and the two halves are coupled, which is the substantive correction to my earlier table:
mirror(minio/mc)snapshot(openbao)Concrete next step, so this is a design rather than an open question: pod-level
supplementalGroups: [1000]grants group 1000 to every container explicitly, independent of both the primary GID and the setgid bit. That would letrunAsGrouprise above 10000 while making the invariant stronger than it is today, because the access no longer rests on an implicit mechanism.It also implies the guard should assert the property rather than the implementation β "the mirror holds group 1000", by primary or supplemental β instead of
runAsGroup == 1000. That is a widening of what the test accepts, so it only goes in together with evidence, never ahead of it.The evidence still owed is one read: a gid-65532 process opening a 0640 snapshot on that PVC. Until someone runs it, this stays open rather than shipped β the failure mode is the nightly backup silently ceasing to mirror, which is not a thing to infer.
chore: agent-claim #3202 nonce=11a9cd36504169b488a2df6293ae465f
chore: agent-claim #3202 nonce=5147de43e7351ade20712c6245fc7525
devantler commentedon Aug 21, 2026
Re-measured on
a0537f33with trivy v0.74.0, since this issue's own note says the counts drift.The residual is 6, not 8, and it splits unevenly:
snapshotonly β openbao image-defined UID 100snapshot(image-defined) andmirror(inherits pod GID 1000)Two measurement notes, both of which cost a wrong reading if skipped:
--data .trivy/data. Without it, four extra MEDIUM findings appear on these two files that CI never sees.k8s/rather than the repo root makes trivy report targets relative tok8s/, so the ignorefile'sk8s/-prefixedpaths:stop matching and already-dispositioned findings reappear β including vault-config's. That is an artifact of the invocation, not a regression.Why the two halves need different answers. The KSV-0020 premise is settled: all four openbao container references in this repository pin the identical digest
sha256:fdc6da21caβ¦171f0806, so the/etc/passwd//etc/groupmeasurement behind vault-config's accepted disposition is a measurement of this image rather than an analogy to it. Ruled on in #3282.KSV-0021 is not settled, and this is where #3281 went wrong. Its premise requires the pod default GID to be high so the non-openbao container takes it; here the pod default is 1000, so
mirrorinherits a group chosen for openbao andminio/mcbakes no such identity. vault-config can make that split because its shared paths are emptyDir, where fsGroup's setgid bit carries the group whatever gid the writer runs as (#3258). These snapshots live on a PVC, where that has not been established.The decisive missing evidence is one line: whether
/snapshotsactually carries the setgid bit. Nothing currently logs it β the snapshot script printsls -lof the file, whose group is 1000 either way, so it cannot distinguish setgid inheritance from the shared primary GID. Anls -ld /snapshotsin the snapshot container would settle it on the next nightly run.Step 1 is confirmed complete in production: the 2026-08-21 03:30 run logged
Widened 14 snapshot(s) on the shared PVC to 0640with zero warnings, so every retained snapshot is already group-readable and no longer depends on the writer owning it.ci: register the vault-backup identity guard and its reciprocal pair
devantler commentedon Aug 21, 2026
Criterion 1 re-verified live β now on a second consecutive run
Re-checked against the cluster rather than inheriting the 2026-08-20 note, because a cleared blocker
is exactly the kind of record that goes stale in the wrong direction. It has not: the
2026-08-21T03:30Z run completed successfully, and both halves hold again.
The snapshot lands group-readable:
And the mirror still uploaded it β the post-upload remote listing shows
openbao-20260821-033020.snappresent, alongside an unbroken daily series.One thing worth recording that the earlier check did not mention: the job also reports
so the backfill covers the existing snapshots on the PVC, not just newly written ones. A
high-UID reader introduced by step 2 would therefore find the whole retained set readable, not only
files created after the change β which removes a retention-window edge this issue would otherwise
have had to reason about.
Status
Criterion 1 is met on two consecutive runs (2026-08-20 and 2026-08-21), so step 2 remains
actionable and is not blocked. The residual risk is unchanged and is sequencing, not premise: the
mirrorcontainer has so far read the snapshot as its owner, so a high-UID mirror reading it bygroup is still what step 2's own watched verification has to establish. The supporting mechanism
(
fsGroupsupplying the group via supplementary groups plus the volume setgid bit) was measured in#3258.
The live CronJob confirms the pod carries
fsGroup: 1000alongsiderunAsGroup: 1000, so thesupplementary-group path is present in the deployed spec.
30 3 * * *. Step 2 should be started early enough in a run to land before a 03:30Z boundary, orexplicitly handed forward with the watch recorded β not merged and forgotten.
Part of #2787.
devantler commentedon Aug 21, 2026
Correcting the previous comment: step 2 is MERGED and LIVE, not "still actionable"
The 11:11Z comment above concluded "step 2 remains actionable and is not blocked". That was
already wrong when it was written: #3279 β which is step 2 β merged at
05:55:11Z, and thecomment at
05:56:55Zon this same issue says so. The 11:11Z status was re-derived from thisissue's body rather than from
main, which is exactly the failure modemonorepo#2967 was filed about.
Verified live, from the deployed spec β the manifest change has reconciled:
runAsUserrunAsGroup/fsGroup1000)init/snapshotctr/mirrorπ΄ But the run that has already happened under the new identity is a VACUOUS PASS β do not close on it
A
vault-snapshot-initJob ran at05:52:21Z, its frozen pod spec carries exactly the newidentity above, and it Succeeded. That is a very easy thing to mistake for this issue's
acceptance evidence, so it is worth stating plainly why it is not:
init/snapshotcontainer loggedA snapshot from today already exists β nothing to do, sono new snapshot was written;
mirrorcontainer therefore reportedTotal 0 B / Transferred 0 Band listed an unbrokenremote series.
mc mirrordecides "nothing to transfer" by listing and comparing source against destination,which needs traverse/read on the directory. It does not establish that a process at UID 65532
can open a snapshot file. The property this issue actually has to prove β the mirror reading the
snapshot by group rather than by ownership β was not exercised by that run.
The supporting mechanism is sound and measured (pod-level
runAsGroup: 1000makes 1000 the mirror'sprimary group, and the snapshot lands
-rw-r-----owned by group 1000, re-confirmed on the2026-08-21 nightly), but a mechanism argument is not the E2E observation the acceptance criteria ask
for.
Named measurement date
The first nightly that can supply the evidence is
2026-08-22T03:30Zβ the schedule is30 3 * * *and #3279 merged after today's 03:30Z run, so no nightly has yet written a snapshotwith the mirror at 65532. Until then this is skip-clause-(d) work: delivered, awaiting a named future
measurement, not unstarted.
What that run must show, all three together β a snapshot written today, a non-zero transfer by
the mirror at 65532, and a successful upload. A
SucceededJob with a 0 B transfer is the vacuouscase above and does not count.
One criterion needs a decision, separately
Acceptance criterion 4 asks for the
checkov.io/skipforCKV_K8S_40to be removed from bothmanifests with checkov still at 0 failed. As shipped, the skip is retained but narrowed β #3282
re-based it on the openbao image's own
openbao:x:100:1000identity, scoped to the single containerthat cannot avoid it. Removing it now would simply make checkov fail, since that container still runs
UID 100 by construction.
That is the "scoped, justified exception for a genuinely irreducible control" rung of the
fix-vs-except ladder rather than a retired exception, so criterion 4 as literally written is not
reachable without rebuilding the upstream image. Flagging it rather than rewriting it unilaterally,
since the delivering PR deliberately used
Part of, notFixes.Part of #2787.
devantler commentedon Aug 22, 2026
First non-vacuous nightly verification passed
The scheduled 2026-08-22 03:30 UTC run exercised the property this issue was waiting for:
This clears the watched UID-migration acceptance window. The permanent CKV_K8S_40 disposition remains correctly scoped to the OpenBao image container; removing it literally would fail Checkov without changing that image-defined identity. The four KSV-0021 group findings remain visible under #2787 and are not being folded into this verdict.
Draft #3298 updates exact-main comments that still described the UID disposition as deferred and the mirror as the file owner. Keep this issue open until that documentation correction lands; then the UID-migration leaf can close without losing the separate KSV-0021 residual.
devantler commentedon Aug 22, 2026
Completed by #3298 at reviewed head
144f680e7299c1607b277aa1a8bfb87b0198c0a1, merged as signedmain@554a3c85a5dc98e8004081bbf4bbb7ca0a423042.Merge-group CI and deployment run 32559633335 succeeded; Flux applied
latest@sha256:a81046eac9a76708a7d87c7857f93b91bb2d8c816cd0a739c00858023e9f8ca9Ready. This closes the UID/CKV_K8S_40 leaf after the accepted nightly backup proof with mode0640.The four KSV-0021 group-identity findings remain live under #2787; this closure does not disposition them.