Repository navigation
feat(sandboxpolicy): report privilege, network and file verdicts to the exporter receivers - #1080
AlonLiwsky wants to merge 203 commits into
Conversation
…ent library pkg/ebpf/gadgets/common is a small library for BPF LSM enforcement gadgets: the bindings map (cgroup -> policy slot), policy rank resolution, verdict base fields and counters, the arming rule, and the Go binding controller of one node-wide instance. It is taken from the enforce_exec simple engine of the pinned Inspektor Gadget fork. pkg/ebpf/gadgets/enforce_privilege enforces the SandboxPolicy privilege section with BPF LSM, no seccomp: namespaces, mounts, pivot_root/chroot, capabilities, ptrace, modules, bpf, perf, keys, io_uring, userfaultfd, mknod, setuid/setgid and capset. Its compiler merges several policies into one slot per combination. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…cy controller - CompiledPolicy carries the privilege section; the compiler accepts it - SectionBackend splits each plan per section; plans without a privilege section go to the exec backend unchanged - PrivilegeBackend owns one enforce_privilege instance per node: live bind/unbind; a new policy combination replaces the instance with overlap - slots are created when a matching pod is scheduled to the node - sandboxPolicyPrivilegeImage enables the section; empty rejects it Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an allow for one flag cannot carry another flag past a default deny - compat ptrace read-mode access is not judged - failed arming marks are counted; a deep tree arms instead of matching a wrong namespace - transient arming failures are retried and reported - a binding whose rollback failed stays tracked for Unbind - reject NUL in mountFsType and duplicate slot generation entries - the gadget build loop stops on the first failure Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an allow for one flag cannot carry another flag past a default deny - compat ptrace read-mode access is not judged - failed arming marks are counted; a deep tree arms instead of matching a wrong namespace - transient arming failures are retried and reported - a binding whose rollback failed stays tracked for Unbind - reject NUL in mountFsType and duplicate slot generation entries - the gadget build loop stops on the first failure Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The binding cgroup path comes from the container collection, which reads /proc/<pid>/cgroup in the cgroup namespace of PID 1. node-agent runs in its own cgroup namespace, where the same file shows a /../.. path, so every lease check failed with "process cgroup changed" and no container bound. Check now reads the file in PID 1's cgroup namespace when it differs. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…efix Gadget eBPF parameters are named operator.oci.ebpf.<key>. The engine set operator.ebpf.host-mntns-id, which IG ignored, so host_mntns_id stayed 0 and the program used the global init task's mount namespace. In kind the runtime runs in the node container's mount namespace, so runc init for kubectl exec was armed and its setns(CLONE_NEWNS) was denied: every exec into a bound pod failed. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- keep an unready candidate that failed to stop for a stop retry - prepare slots for ephemeral containers too - list the node's pods for slot preparation only after a Pending-pod event or every 30 s, not on every refresh Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file; the SandboxPolicy translation moves to the controller connection (kubescape#1068) - neutral names in the gadget, BPF, gadget.yaml and README - a guard test fails on imports outside Inspektor Gadget and the gadget folders, and on host-product names or outside includes - an opt-in test runs the gadget with Inspektor Gadget alone Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file; the SandboxPolicy translation moves to the controller connection (kubescape#1068) - neutral names in the gadget, BPF, gadget.yaml and README - a guard test fails on imports outside Inspektor Gadget and the gadget folders, and on host-product names or outside includes - an opt-in test runs the gadget with Inspektor Gadget alone Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…n the controller layer PrivilegeDocument (pkg/sandboxpolicy/privilege.go) replaces the gadget former FromAPI, so the gadget no longer imports the CRD types. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…active Without "bpf" in the host's active LSM list, BPF LSM programs attach but never run (for example default AKS Ubuntu and Azure Linux images), so a policy could report Ready with no enforcement. The service now reads <host>/sys/kernel/security/lsm at startup; if bpf is missing it creates no privilege backend, privilege bindings are refused, and every policy with a privilege section reports Ready=False with reason BPFLSMInactive. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot When an armed task forks and the child's task storage mark cannot be allocated, the child escaped every rule while the instance stayed ready. Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM (a clone into another cgroup too), and counts a tracking fault per slot in arming_faults. A lost mark at exec, which cannot be refused, is counted the same way. Audit-only slots and unbound cgroups allow the fork. A new thread of an armed group leader is covered through the leader and is never refused. Moving a container to another slot arms its running tasks again. The binding controller reads the count with SlotFaults. A test-only parameter injects the failure for the new kernel test. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot When an armed task forks and the child's task storage mark cannot be allocated, the child escaped every rule while the instance stayed ready. Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM (a clone into another cgroup too), and counts a tracking fault per slot in arming_faults. A lost mark at exec, which cannot be refused, is counted the same way. Audit-only slots and unbound cgroups allow the fork. A new thread of an armed group leader is covered through the leader and is never refused. Moving a container to another slot arms its running tasks again. The binding controller reads the count with SlotFaults. A test-only parameter injects the failure for the new kernel test. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The first pod of a network policy after node-agent starts connected before any network instance existed: only privilege started an empty instance. - NewNetwork sets startEmpty, so an active network policy loads the gadget with no slot, with the same 30 s retry backoff as privilege. - The network compiler and runtime accept an instance with no slot. - PrepareScheduled marks the network section active only for a policy whose network section translated, so an untranslated section loads no gadget. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… origin The network translator and the network gadget now treat a missing origin and an explicit "" as different values, as the privilege gadget does: - In the gadget document, rule Origin and the egress DefaultOrigin are *string. The compiler gives no origin id 0 and an explicit "" its own id (Origins[id] == ""). - Compiled.DefaultOrigins holds, per slot, the sorted distinct default origins of the documents that agree with the merged default (a missing one adds nothing, "" is kept). Default verdicts take their origins there. - TranslateNetwork passes the rule origin and the egress defaultOrigin pointers unchanged. It already passed the egress defaultMode, which the compiler uses; tests now cover both. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… root An attach_to override of a cgroup program would make the cgroup tier judge only the cgroups below it, so the node-wide coverage would silently shrink. - The gadget runtime refuses, for the cgroup tier, an attach_to that is not the cgroup-v2 root (cgroup ID 1) with a CgroupRootError. An empty attach_to keeps the default root, which Inspektor Gadget checks itself. - node-agent maps that error to an unavailable network section with reason NetworkAttachNotRoot. The node owner keeps the reason of its last start, and SectionBackend.SectionUnavailable reads it (RuntimeAvailability), so every network policy reports Ready=False, also one with no running pod. node-agent selects the LSM tier today; the check is on the cgroup path and unit-tested without the Inspektor Gadget cgroup attach change. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…efore the first pod
The file backend sets startEmpty, but CompileSlots and validateConfig
refused zero slots ("0 policy combinations: need 1 to 64 per node").
So after a node-agent start no file instance ran until the first selected
pod was scheduled, and that pod's first command ran before its pod
binding (about 1-2 s for the first gadget load).
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
At start, and when a policy appears or changes, PrepareScheduled now adds one plan per active policy (its privilege and translated network sections) after the pod plans. Each section's node owner prepares those slots, so the first instance after a node-agent restart already has them (one load, not an empty instance and then a replacement), and a pod matched by one policy of a section binds with a live map write. A pod matched by several policies of a section still needs their combined slot (one replacement); subsets are not enumerated. Prepared slots stay speculative: they leave with the policy, are not in use, and ClearPrepared still never starts an instance. When the slots do not all fit, Prepare keeps the ones that fit (pod plans first) and logs the rest instead of failing. The service prepares right after Refresh loads new or changed policies, so the slot exists before a pod matches the policy. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # docs/features/sandbox-policy.md # pkg/sandboxpolicy/integration/service.go
The file compiler now keeps an explicit empty origin apart from no origin and lists, per slot and access kind, the default origins of the policies that agree with the merged default. A file default verdict takes them all from there. The kernel test checks that an explicit "" default origin reaches the exported event as [""] on Linux 6.8 and 6.17. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
PrepareScheduled prepares the file section too (active and slotPolicy), and the pod plan keeps passing the file plans. Tests follow the base: an untranslated file section keeps the pod binding (empty plan), and a policy gets its own prepared plan. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # pkg/sandboxpolicy/igbackend/nodeowner.go
…ault origin The privilege and network compilers now keep an explicit empty origin apart from no origin and list the default origins per slot. A default verdict takes them all from there, like file. Kernel cases check that an explicit "" exports as [""] for a privilege rule and a network default. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…rograms The cgroup tier is on hold, so the gadget attaches only lsm/socket_connect and lsm/socket_sendmsg. A node without BPF LSM reports the network section as unavailable with the BPFLSMInactive reason. - Remove the cgroup/connect4, connect6, sendmsg4 and sendmsg6 programs, the tier parameter, the tier selection and CgroupRootError. - Remove NetworkAttachNotRoot and the RuntimeAvailability plumbing that only it used. - node-agent checks BPF LSM for the network section with CheckNetworkBPFLSM. - Remove the cgroup tier tests and docs; the decision core, the earlier-deny chaining and the hook-args check stay. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # pkg/sandboxpolicy/igbackend/network_native.go # pkg/sandboxpolicy/igbackend/nodeowner.go # pkg/sandboxpolicy/integration/service.go
NewNativeNetworkEngine no longer takes a network tier since the parent removed the cgroup tier. Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
|
I followed the explicit-empty origin path through the current stacked heads: #1068 passes the privilege One small steering point: the PR description still asks whether public section dispatch should be folded into the private exporter. We already agreed it stays here, with private OTLP mapping in armosec/private-node-agent#747. Please update that open question so reviewers see the decided boundary. Keep the authorization-versus-observation mapping and backend ingestion as the separate follow-up, not a blocker on this public typed-event PR. |
…r node Node status now carries appliedGeneration, the oldest generation still installed, so a rejected revision no longer hides which one enforces. The BPF LSM probe ran only for the privilege section, but exec also enforces through BPF LSM, so an exec policy reported Ready and tier bpf-lsm on nodes where nothing runs. The probe result now applies to every section and such nodes report tier none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Without BPF LSM, installed bindings enforce nothing, so the report no longer names an applied generation. A refused privilege section no longer reports tier none while last-good exec bindings still enforce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Temporary replace to the fork commit of kubescape/storage#485, until it merges into the sandbox policy API branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Privilege pod bindings are driven by the integration service, so the controller never saw them. When the container binding failed while a pod binding enforced, the report said no last-good enforcement exists and omitted appliedGeneration. The privilege backend now reports its bound pod plans, and the report counts them like container bindings. A pod whose plan changes only in generation records the new plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Pod cgroup bindings enforce privilege, network and file rules, but the controller sees only container bindings, so node status could say that nothing enforces while pod bindings do. nodeOwner now reports its bound pod plans and SectionBackend collects them from every available section. A pod whose plan changes only in generation records the new plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
# Conflicts: # pkg/sandboxpolicy/sections_test.go
… branch Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
# Conflicts: # pkg/sandboxpolicy/igbackend/lsm.go # pkg/sandboxpolicy/igbackend/privilege.go # pkg/sandboxpolicy/igbackend/privilege_test.go # pkg/sandboxpolicy/integration/service.go # pkg/sandboxpolicy/sections_test.go # pkg/sandboxpolicy/status.go
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com> # Conflicts: # pkg/sandboxpolicy/controller.go # pkg/sandboxpolicy/integration/service.go
|
Folded into #1075 to shorten the stack: the file-gadget branch now carries these commits unchanged, and review continues there. |
Overview
This reports the privilege, network and file verdicts (enforced denies and audit wouldDeny) to the same receivers that exec authorizations use, so a private exporter (for example the OTLP mapping in armosec/private-node-agent#747) can pick them up. Until now these verdicts only went to the node-agent log.
It is stacked on #1075 (file gadget, which contains #1073, #1068 and #1067). GitHub can't use a fork branch as the base, so please review only the commits that are not in #1075, and merge after it.
Boundary (decided): the typed public section events and their queue stay here in node-agent. The private OTLP mapping lives in armosec/private-node-agent#747. Public event dispatch is not folded into the private exporter.
What changes
pkg/sandboxpolicy/section_verdict.go): one event per verdict for privilege, network and file, with its own dispatch key, the policy identity (kind, name, uid, generation, namespace) and revision, rule name, origins list, mode,wouldDeny, the target in policy words (for exampleread, not the C enum name) and the actor (process, container, pod).exporter.gois identical to feat(sandboxpolicy): enforce the file and identity sections and export section verdicts #1075), so a burst of section denies can never drop or delay an exec verdict. A test sends 1,000 section denies into a 4-entry section queue: 996 section events are dropped and counted, 0 exec verdicts.Service.SectionVerdictStats()/SectionExporterStats(). A drop is never silent.sandboxPolicyDebugExport(default off): logs one line per exported event, for local testing.setnson a mount namespace gives two). The exporter does not dedupe; the doc says so.Production code is about 950 lines (the section exporter ~150, the debug receiver ~70, the subscriber rewrite), tests about 820, docs about 35.
How to test
go test ./pkg/sandboxpolicy/...wouldDeny, rule and pod binding; for a pod with two policies, the event names the deciding policy and both in the set.Follow-up (not in this PR)
The backend accepts only exec verdicts today. Privilege, network and file need backend changes (rule family per section with an authorization phase, a phase guard on the R-NET-1 / R-FILE-1 observation views, section defaults, target keys), plus the authorization-versus-observation mapping in the private exporter. That is a separate stacked change, not a blocker for this public typed-event PR.
Tracking: SUB-9591.