Repository navigation
feat(sandboxpolicy): report the applied generation and a true tier in node status - #1082
Closed
AlonLiwsky wants to merge 86 commits into
Closed
AlonLiwsky wants to merge 86 commits into
AlonLiwsky wants to merge 86 commits into
Conversation
…ent library pkg/ebpf/gadgets/common is a small library for BPF LSM enforcement gadgets: the bindings map (cgroup -> policy slot), policy rank resolution, verdict base fields and counters, the arming rule, and the Go binding controller of one node-wide instance. It is taken from the enforce_exec simple engine of the pinned Inspektor Gadget fork. pkg/ebpf/gadgets/enforce_privilege enforces the SandboxPolicy privilege section with BPF LSM, no seccomp: namespaces, mounts, pivot_root/chroot, capabilities, ptrace, modules, bpf, perf, keys, io_uring, userfaultfd, mknod, setuid/setgid and capset. Its compiler merges several policies into one slot per combination. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…cy controller - CompiledPolicy carries the privilege section; the compiler accepts it - SectionBackend splits each plan per section; plans without a privilege section go to the exec backend unchanged - PrivilegeBackend owns one enforce_privilege instance per node: live bind/unbind; a new policy combination replaces the instance with overlap - slots are created when a matching pod is scheduled to the node - sandboxPolicyPrivilegeImage enables the section; empty rejects it Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an allow for one flag cannot carry another flag past a default deny - compat ptrace read-mode access is not judged - failed arming marks are counted; a deep tree arms instead of matching a wrong namespace - transient arming failures are retried and reported - a binding whose rollback failed stays tracked for Unbind - reject NUL in mountFsType and duplicate slot generation entries - the gadget build loop stops on the first failure Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an allow for one flag cannot carry another flag past a default deny - compat ptrace read-mode access is not judged - failed arming marks are counted; a deep tree arms instead of matching a wrong namespace - transient arming failures are retried and reported - a binding whose rollback failed stays tracked for Unbind - reject NUL in mountFsType and duplicate slot generation entries - the gadget build loop stops on the first failure Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The binding cgroup path comes from the container collection, which reads /proc/<pid>/cgroup in the cgroup namespace of PID 1. node-agent runs in its own cgroup namespace, where the same file shows a /../.. path, so every lease check failed with "process cgroup changed" and no container bound. Check now reads the file in PID 1's cgroup namespace when it differs. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…efix Gadget eBPF parameters are named operator.oci.ebpf.<key>. The engine set operator.ebpf.host-mntns-id, which IG ignored, so host_mntns_id stayed 0 and the program used the global init task's mount namespace. In kind the runtime runs in the node container's mount namespace, so runc init for kubectl exec was armed and its setns(CLONE_NEWNS) was denied: every exec into a bound pod failed. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- keep an unready candidate that failed to stop for a stop retry - prepare slots for ephemeral containers too - list the node's pods for slot preparation only after a Pending-pod event or every 30 s, not on every refresh Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file; the SandboxPolicy translation moves to the controller connection (#1068) - neutral names in the gadget, BPF, gadget.yaml and README - a guard test fails on imports outside Inspektor Gadget and the gadget folders, and on host-product names or outside includes - an opt-in test runs the gadget with Inspektor Gadget alone Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file; the SandboxPolicy translation moves to the controller connection (#1068) - neutral names in the gadget, BPF, gadget.yaml and README - a guard test fails on imports outside Inspektor Gadget and the gadget folders, and on host-product names or outside includes - an opt-in test runs the gadget with Inspektor Gadget alone Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…n the controller layer PrivilegeDocument (pkg/sandboxpolicy/privilege.go) replaces the gadget former FromAPI, so the gadget no longer imports the CRD types. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…active Without "bpf" in the host's active LSM list, BPF LSM programs attach but never run (for example default AKS Ubuntu and Azure Linux images), so a policy could report Ready with no enforcement. The service now reads <host>/sys/kernel/security/lsm at startup; if bpf is missing it creates no privilege backend, privilege bindings are refused, and every policy with a privilege section reports Ready=False with reason BPFLSMInactive. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot When an armed task forks and the child's task storage mark cannot be allocated, the child escaped every rule while the instance stayed ready. Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM (a clone into another cgroup too), and counts a tracking fault per slot in arming_faults. A lost mark at exec, which cannot be refused, is counted the same way. Audit-only slots and unbound cgroups allow the fork. A new thread of an armed group leader is covered through the leader and is never refused. Moving a container to another slot arms its running tasks again. The binding controller reads the count with SlotFaults. A test-only parameter injects the failure for the new kernel test. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot When an armed task forks and the child's task storage mark cannot be allocated, the child escaped every rule while the instance stayed ready. Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM (a clone into another cgroup too), and counts a tracking fault per slot in arming_faults. A lost mark at exec, which cannot be refused, is counted the same way. Audit-only slots and unbound cgroups allow the fork. A new thread of an armed group leader is covered through the leader and is never refused. Moving a container to another slot arms its running tasks again. The binding controller reads the count with SlotFaults. A test-only parameter injects the failure for the new kernel test. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…ntainer A parent cgroup bound before its containers start (a pod cgroup) also covered the OCI createContainer and startContainer hooks: runc init forks them inside the container mount namespace, so they were armed and judged as workload. A strict policy then stopped the container from starting (for example kind's own createContainer hook, a bind mount). A task is no longer armed at exec when its parent is unarmed, outside the runtime mount namespace, and in the same mount namespace and cgroup as the task: it was forked by a runtime helper inside the container. The workload and exec commands are execed by runc init itself, whose parent is in the runtime mount namespace, so they stay armed, and their children inherit the mark as before. An unknown runtime namespace trusts no parent. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # pkg/ebpf/gadgets/enforce_privilege/README.md
… of Linux 6.15 Linux 6.15 added `bool kernel` to the bpf hook and dropped the attr argument of perf_event_open, so the previous-verdict check refused every 6.15+ kernel (and the old perf_event_open program would not load there). Both programs now read `ret` at the position of the running kernel: a CO-RE type match against security_list_options picks the branch at load. The BTF check returns a result per hook (LSMHookArgMismatches) and accepts both signatures for these two hooks. The other 21 hooks have the same argument count from 6.1 to 6.18 (checked in lsm_hook_defs.h). Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
setns into a namespace checks CAP_SYS_ADMIN twice (the namespace user namespace and the caller's). Audit lets the first check pass, so one audited setns gives two identical verdicts; enforce stops at the first. A per-syscall dedupe is not reliable in an LSM hook (there is no syscall instance id, and a time window would merge real calls), so the README documents it and a kernel test pins it down. The test fixture now records the fields of each verdict. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…e-gadget Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The #1054 base moved to storage v0.0.388: PrivilegeTarget has no path, fromImage or fsType fields any more, and Rule.Origin is a pointer. The translator follows: it rejects actor and pathClass as before, now also rejects except (the gadget has no exceptions, and they were ignored before), and maps a missing origin and an empty one to no origin. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…e-gadget Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # pkg/sandboxpolicy/integration/service.go # pkg/sandboxpolicy/integration/service_test.go
The walk repeated until a pass saw no new task id, so a container that keeps starting short-lived tasks could fail the bind with "tasks still changing". A pass now counts only the tasks it had to arm: a task that already carries the mark (a child forked after its parent was armed) or that exited is not progress. The rescan for children forked before their parent was armed stays. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Review findings on the pod binding paths: - An edited pod plan whose slot is not ready keeps the last-good pod binding; the pod moves to the new slot once it exists. Only a pod that is not given any more (an authoritative removal) is unbound. - A candidate that does not fit drops only the speculative prepared slots. Active pod bindings are never dropped to make room; when the candidate still does not fit, the new container binding is refused. - Incomplete input (an invalid policy or class, a policy that does not compile, too many policies) gives the pod an empty plan, which keeps its current binding, as the container path keeps its last good plan. Containers with different policies, no privilege policy, or a policy that needs a ContainerProfile still give no pod plan. Each case has a regression test. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…origin Rule origins and the new section default origin are pointers in the policy document. The compiler gives an explicit "" its own origin id; id 0 still means no origin. Compiled.DefaultOrigins lists, per slot, the default origins of the documents that agree with the merged default, for events of the default. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
A section can set the mode of its default apart from the policy mode. The compiler ignored it, so an audit policy with an enforce default only audited its default deny. The merged default and the audit-deny raise now use each document's defaultMode, else its policy mode. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…e gadget The translator now keeps origin presence (a missing origin stays nil, an explicit "" stays "") for rules and the section default, and passes the section defaultMode, which it dropped before. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…r node Node status now carries appliedGeneration, the oldest generation still installed, so a rejected revision no longer hides which one enforces. The BPF LSM probe ran only for the privilege section, but exec also enforces through BPF LSM, so an exec policy reported Ready and tier bpf-lsm on nodes where nothing runs. The probe result now applies to every section and such nodes report tier none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Without BPF LSM, installed bindings enforce nothing, so the report no longer names an applied generation. A refused privilege section no longer reports tier none while last-good exec bindings still enforce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Temporary replace to the fork commit of kubescape/storage#485, until it merges into the sandbox policy API branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Privilege pod bindings are driven by the integration service, so the controller never saw them. When the container binding failed while a pod binding enforced, the report said no last-good enforcement exists and omitted appliedGeneration. The privilege backend now reports its bound pod plans, and the report counts them like container bindings. A pod whose plan changes only in generation records the new plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Performance Benchmark ResultsNode-Agent Resource Usage
Dedup EffectivenessNo data available. |
Performance Benchmark ResultsNode-Agent Resource Usage
Dedup EffectivenessNo data available. |
Collaborator
Author
|
Folded into #1068 to shorten the stack: the privilege-controller branch now carries these commits unchanged, and review continues there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Part of the per-node status work for sandbox policies. Two fixes to what each node reports in
SandboxPolicyNodeStatus:Applied generation. The report now fills
appliedGeneration: the oldest policy generation still installed on the node, counting only bindings whose protection is confirmed. After a rejected edit it stays on the last-good generation, soobservedGeneration: 5, appliedGeneration: 4says plainly that 4 still enforces. Before, that was only in the free-textActiveRevisionmessage.A true tier. Every section enforces through BPF LSM (exec via
bprm_check_security, privilege vialsm/*), but only the privilege section consulted the BPF LSM probe. On a node without BPF LSM (default AKS images), an exec policy reportedReady=True, tier: bpf-lsmwhile nothing ran. Now the probe runs once at startup for all sections; on such a node every report saystier: none, every enabled policy isReady=Falsewith reasonBPFLSMInactive, andappliedGenerationis absent because nothing enforces. A refused privilege section (PrivilegeNotConfigured) reportstier: noneonly when no last-good binding still enforces.Pod cgroup bindings count. Privilege pod bindings are driven by the integration service, so the controller never saw them: when the container binding failed while a pod binding enforced, the report said no last-good enforcement exists and omitted
appliedGeneration. The privilege backend now reports its bound pod plans (BoundPodPlans), and the report counts them like container bindings. File/network pod bindings from #1073/#1075 will need to report the same way.Not in this PR: kprobe-override and tripwire tiers (no implementation exists yet), and filling
degraded[](nothing in this branch weakens a single rule; the first case, unresolved classes, lives in #1073 / #1075).Additional Information
Stacked on #1068. That branch lives on my fork, so this PR targets
feat/sandbox-policy-exec-enforcementand also shows the #1067/#1068 commits until those merge. The new commits are the last four:feat(sandboxpolicy): report the applied generation and a true tier per nodefix(sandboxpolicy): keep tier and applied generation true in edge caseschore: pin storage to the node status API changefix(sandboxpolicy): count pod cgroup bindings in node statusTemporary storage pin.
go.modreplaceskubescape/storagewith the fork commit of kubescape/storage#485, which adds theappliedGenerationfield and thenonetier. I'll swap it for the upstream commit once #485 merges.How to Test
Each case in the table below has a unit test (
status_test.go,sections_test.go,controller_test.go):bpf-lsmbpf-lsmnonenonebpf-lsmbpf-lsmRelated issues/PRs
🤖 Generated with Claude Code