Skip to content

feat(sandboxpolicy): report privilege, network and file verdicts to the exporter receivers - #1080

Closed
AlonLiwsky wants to merge 203 commits into
kubescape:feat/sandbox-policy-exec-enforcementfrom
AlonLiwsky:feat/sandbox-policy-event-export
Closed

AlonLiwsky wants to merge 203 commits into
kubescape:feat/sandbox-policy-exec-enforcementfrom
AlonLiwsky:feat/sandbox-policy-event-export

Conversation

@AlonLiwsky

@AlonLiwsky AlonLiwsky commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

This reports the privilege, network and file verdicts (enforced denies and audit wouldDeny) to the same receivers that exec authorizations use, so a private exporter (for example the OTLP mapping in armosec/private-node-agent#747) can pick them up. Until now these verdicts only went to the node-agent log.

It is stacked on #1075 (file gadget, which contains #1073, #1068 and #1067). GitHub can't use a fork branch as the base, so please review only the commits that are not in #1075, and merge after it.

Boundary (decided): the typed public section events and their queue stay here in node-agent. The private OTLP mapping lives in armosec/private-node-agent#747. Public event dispatch is not folded into the private exporter.

What changes

  • Typed section events (pkg/sandboxpolicy/section_verdict.go): one event per verdict for privilege, network and file, with its own dispatch key, the policy identity (kind, name, uid, generation, namespace) and revision, rule name, origins list, mode, wouldDeny, the target in policy words (for example read, not the C enum name) and the actor (process, container, pod).
  • Binding lookup: the event's cgroup is resolved to its container or pod binding, and the policy set comes from the judging instance's slot table (by generation and slot).
  • Own queue: section events have their own queue and worker and feed the same receivers. The exec exporter is unchanged (exporter.go is identical to feat(sandboxpolicy): enforce the file and identity sections and export section verdicts #1075), so a burst of section denies can never drop or delay an exec verdict. A test sends 1,000 section denies into a 4-entry section queue: 996 section events are dropped and counted, 0 exec verdicts.
  • Counters: kernel ring-buffer drops and section queue drops are exposed through Service.SectionVerdictStats() / SectionExporterStats(). A drop is never silent.
  • Debug receiver behind sandboxPolicyDebugExport (default off): logs one line per exported event, for local testing.
  • One verdict is one kernel check, not one syscall (an audited setns on a mount namespace gives two). The exporter does not dedupe; the doc says so.

Production code is about 950 lines (the section exporter ~150, the debug receiver ~70, the subscriber rewrite), tests about 820, docs about 35.

How to test

  • go test ./pkg/sandboxpolicy/...
  • Kernel export test on 6.8 and 6.17 (arm64): one enforced deny and one audit wouldDeny per section reach a test receiver with binding, policy identity, rule, origins, target and counters.
  • Local kind end-to-end run with the debug receiver on: every one of the 175 denied or audited actions produced its event with the right wouldDeny, rule and pod binding; for a pod with two policies, the event names the deciding policy and both in the set.

Follow-up (not in this PR)

The backend accepts only exec verdicts today. Privilege, network and file need backend changes (rule family per section with an authorization phase, a phase guard on the R-NET-1 / R-FILE-1 observation views, section defaults, target keys), plus the authorization-versus-observation mapping in the private exporter. That is a separate stacked change, not a blocker for this public typed-event PR.

Tracking: SUB-9591.

…ent library

pkg/ebpf/gadgets/common is a small library for BPF LSM enforcement gadgets:
the bindings map (cgroup -> policy slot), policy rank resolution, verdict
base fields and counters, the arming rule, and the Go binding controller of
one node-wide instance. It is taken from the enforce_exec simple engine of
the pinned Inspektor Gadget fork.

pkg/ebpf/gadgets/enforce_privilege enforces the SandboxPolicy privilege
section with BPF LSM, no seccomp: namespaces, mounts, pivot_root/chroot,
capabilities, ptrace, modules, bpf, perf, keys, io_uring, userfaultfd,
mknod, setuid/setgid and capset. Its compiler merges several policies into
one slot per combination.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…cy controller

- CompiledPolicy carries the privilege section; the compiler accepts it
- SectionBackend splits each plan per section; plans without a privilege
  section go to the exec backend unchanged
- PrivilegeBackend owns one enforce_privilege instance per node: live
  bind/unbind; a new policy combination replaces the instance with overlap
- slots are created when a matching pod is scheduled to the node
- sandboxPolicyPrivilegeImage enables the section; empty rejects it

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an
  allow for one flag cannot carry another flag past a default deny
- compat ptrace read-mode access is not judged
- failed arming marks are counted; a deep tree arms instead of matching a
  wrong namespace
- transient arming failures are retried and reported
- a binding whose rollback failed stays tracked for Unbind
- reject NUL in mountFsType and duplicate slot generation entries
- the gadget build loop stops on the first failure

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- each namespace flag of a call resolves on its own (bpf_loop), so an
  allow for one flag cannot carry another flag past a default deny
- compat ptrace read-mode access is not judged
- failed arming marks are counted; a deep tree arms instead of matching a
  wrong namespace
- transient arming failures are retried and reported
- a binding whose rollback failed stays tracked for Unbind
- reject NUL in mountFsType and duplicate slot generation entries
- the gadget build loop stops on the first failure

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The binding cgroup path comes from the container collection, which reads
/proc/<pid>/cgroup in the cgroup namespace of PID 1. node-agent runs in its
own cgroup namespace, where the same file shows a /../.. path, so every
lease check failed with "process cgroup changed" and no container bound.
Check now reads the file in PID 1's cgroup namespace when it differs.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…efix

Gadget eBPF parameters are named operator.oci.ebpf.<key>. The engine set
operator.ebpf.host-mntns-id, which IG ignored, so host_mntns_id stayed 0 and
the program used the global init task's mount namespace. In kind the
runtime runs in the node container's mount namespace, so runc init for
kubectl exec was armed and its setns(CLONE_NEWNS) was denied: every exec
into a bound pod failed.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…group

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- keep an unready candidate that failed to stop for a stop retry
- prepare slots for ephemeral containers too
- list the node's pods for slot preparation only after a Pending-pod event
  or every 30 s, not on every refresh

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file;
  the SandboxPolicy translation moves to the controller connection (kubescape#1068)
- neutral names in the gadget, BPF, gadget.yaml and README
- a guard test fails on imports outside Inspektor Gadget and the gadget
  folders, and on host-product names or outside includes
- an opt-in test runs the gadget with Inspektor Gadget alone

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
- the gadget has its own policy document with Parse and an example file;
  the SandboxPolicy translation moves to the controller connection (kubescape#1068)
- neutral names in the gadget, BPF, gadget.yaml and README
- a guard test fails on imports outside Inspektor Gadget and the gadget
  folders, and on host-product names or outside includes
- an opt-in test runs the gadget with Inspektor Gadget alone

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…n the controller layer

PrivilegeDocument (pkg/sandboxpolicy/privilege.go) replaces the gadget former
FromAPI, so the gadget no longer imports the CRD types.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…active

Without "bpf" in the host's active LSM list, BPF LSM programs attach but
never run (for example default AKS Ubuntu and Azure Linux images), so a
policy could report Ready with no enforcement. The service now reads
<host>/sys/kernel/security/lsm at startup; if bpf is missing it creates no
privilege backend, privilege bindings are refused, and every policy with a
privilege section reports Ready=False with reason BPFLSMInactive.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…path

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…sted upward includes

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot

When an armed task forks and the child's task storage mark cannot be
allocated, the child escaped every rule while the instance stayed ready.
Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM
(a clone into another cgroup too), and counts a tracking fault per slot
in arming_faults. A lost mark at exec, which cannot be refused, is
counted the same way. Audit-only slots and unbound cgroups allow the
fork. A new thread of an armed group leader is covered through the
leader and is never refused. Moving a container to another slot arms
its running tasks again. The binding controller reads the count with
SlotFaults. A test-only parameter injects the failure for the new
kernel test.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…an enforcing slot

When an armed task forks and the child's task storage mark cannot be
allocated, the child escaped every rule while the instance stayed ready.
Now an enforcing slot (any enforce entry) refuses the fork with -ENOMEM
(a clone into another cgroup too), and counts a tracking fault per slot
in arming_faults. A lost mark at exec, which cannot be refused, is
counted the same way. Audit-only slots and unbound cgroups allow the
fork. A new thread of an armed group leader is covered through the
leader and is never refused. Moving a container to another slot arms
its running tasks again. The binding controller reads the count with
SlotFaults. A test-only parameter injects the failure for the new
kernel test.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
The first pod of a network policy after node-agent starts connected before
any network instance existed: only privilege started an empty instance.

- NewNetwork sets startEmpty, so an active network policy loads the gadget
  with no slot, with the same 30 s retry backoff as privilege.
- The network compiler and runtime accept an instance with no slot.
- PrepareScheduled marks the network section active only for a policy whose
  network section translated, so an untranslated section loads no gadget.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… origin

The network translator and the network gadget now treat a missing origin
and an explicit "" as different values, as the privilege gadget does:

- In the gadget document, rule Origin and the egress DefaultOrigin are
  *string. The compiler gives no origin id 0 and an explicit "" its own id
  (Origins[id] == "").
- Compiled.DefaultOrigins holds, per slot, the sorted distinct default
  origins of the documents that agree with the merged default (a missing
  one adds nothing, "" is kept). Default verdicts take their origins there.
- TranslateNetwork passes the rule origin and the egress defaultOrigin
  pointers unchanged. It already passed the egress defaultMode, which the
  compiler uses; tests now cover both.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… root

An attach_to override of a cgroup program would make the cgroup tier judge
only the cgroups below it, so the node-wide coverage would silently shrink.

- The gadget runtime refuses, for the cgroup tier, an attach_to that is not
  the cgroup-v2 root (cgroup ID 1) with a CgroupRootError. An empty
  attach_to keeps the default root, which Inspektor Gadget checks itself.
- node-agent maps that error to an unavailable network section with reason
  NetworkAttachNotRoot. The node owner keeps the reason of its last start,
  and SectionBackend.SectionUnavailable reads it (RuntimeAvailability), so
  every network policy reports Ready=False, also one with no running pod.

node-agent selects the LSM tier today; the check is on the cgroup path and
unit-tested without the Inspektor Gadget cgroup attach change.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…efore the first pod

The file backend sets startEmpty, but CompileSlots and validateConfig
refused zero slots ("0 policy combinations: need 1 to 64 per node").
So after a node-agent start no file instance ran until the first selected
pod was scheduled, and that pod's first command ran before its pod
binding (about 1-2 s for the first gadget load).

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
At start, and when a policy appears or changes, PrepareScheduled now adds
one plan per active policy (its privilege and translated network sections)
after the pod plans. Each section's node owner prepares those slots, so the
first instance after a node-agent restart already has them (one load, not
an empty instance and then a replacement), and a pod matched by one policy
of a section binds with a live map write. A pod matched by several policies
of a section still needs their combined slot (one replacement); subsets are
not enumerated.

Prepared slots stay speculative: they leave with the policy, are not in
use, and ClearPrepared still never starts an instance. When the slots do not
all fit, Prepare keeps the ones that fit (pod plans first) and logs the rest
instead of failing. The service prepares right after Refresh loads new or
changed policies, so the slot exists before a pod matches the policy.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>

# Conflicts:
#	docs/features/sandbox-policy.md
#	pkg/sandboxpolicy/integration/service.go
The file compiler now keeps an explicit empty origin apart from no origin
and lists, per slot and access kind, the default origins of the policies
that agree with the merged default. A file default verdict takes them all
from there. The kernel test checks that an explicit "" default origin
reaches the exported event as [""] on Linux 6.8 and 6.17.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
PrepareScheduled prepares the file section too (active and slotPolicy), and
the pod plan keeps passing the file plans. Tests follow the base: an
untranslated file section keeps the pod binding (empty plan), and a policy
gets its own prepared plan.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>

# Conflicts:
#	pkg/sandboxpolicy/igbackend/nodeowner.go
…ault origin

The privilege and network compilers now keep an explicit empty origin
apart from no origin and list the default origins per slot. A default
verdict takes them all from there, like file. Kernel cases check that an
explicit "" exports as [""] for a privilege rule and a network default.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
…rograms

The cgroup tier is on hold, so the gadget attaches only lsm/socket_connect and
lsm/socket_sendmsg. A node without BPF LSM reports the network section as
unavailable with the BPFLSMInactive reason.

- Remove the cgroup/connect4, connect6, sendmsg4 and sendmsg6 programs, the
  tier parameter, the tier selection and CgroupRootError.
- Remove NetworkAttachNotRoot and the RuntimeAvailability plumbing that only
  it used.
- node-agent checks BPF LSM for the network section with CheckNetworkBPFLSM.
- Remove the cgroup tier tests and docs; the decision core, the earlier-deny
  chaining and the hook-args check stay.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
… branch

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>

# Conflicts:
#	pkg/sandboxpolicy/igbackend/network_native.go
#	pkg/sandboxpolicy/igbackend/nodeowner.go
#	pkg/sandboxpolicy/integration/service.go
NewNativeNetworkEngine no longer takes a network tier since the parent removed the cgroup tier.

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
@matthyx

matthyx commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

I followed the explicit-empty origin path through the current stacked heads: #1068 passes the privilege *string, #1073 does the same for network and allocates a nonzero origin ID for present "", #1075 does so for file, and this PR decodes ID 0 as absent versus nonzero "" as [""]. The per-slot default-origin lists also retain present empty labels. Focused Go tests on this head (6c52f1af) passed for ./pkg/sandboxpolicy/..., network/file policy, and common enforcement. The origin-presence issue I raised is addressed; this is not a claim of kernel/exporter end-to-end revalidation.

One small steering point: the PR description still asks whether public section dispatch should be folded into the private exporter. We already agreed it stays here, with private OTLP mapping in armosec/private-node-agent#747. Please update that open question so reviewers see the decided boundary. Keep the authorization-versus-observation mapping and backend ingestion as the separate follow-up, not a blocker on this public typed-event PR.

AlonLiwsky and others added 2 commits November 9, 2026 12:29
…r node

Node status now carries appliedGeneration, the oldest generation still
installed, so a rejected revision no longer hides which one enforces.

The BPF LSM probe ran only for the privilege section, but exec also
enforces through BPF LSM, so an exec policy reported Ready and tier
bpf-lsm on nodes where nothing runs. The probe result now applies to
every section and such nodes report tier none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Without BPF LSM, installed bindings enforce nothing, so the report no
longer names an applied generation. A refused privilege section no
longer reports tier none while last-good exec bindings still enforce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
@matthyx matthyx removed the needs-rebase PR has merge conflicts and needs to be rebased label Nov 9, 2026
AlonLiwsky and others added 7 commits November 9, 2026 14:16
Temporary replace to the fork commit of kubescape/storage#485, until it
merges into the sandbox policy API branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Privilege pod bindings are driven by the integration service, so the
controller never saw them. When the container binding failed while a pod
binding enforced, the report said no last-good enforcement exists and
omitted appliedGeneration. The privilege backend now reports its bound
pod plans, and the report counts them like container bindings. A pod
whose plan changes only in generation records the new plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
Pod cgroup bindings enforce privilege, network and file rules, but the
controller sees only container bindings, so node status could say that
nothing enforces while pod bindings do. nodeOwner now reports its bound
pod plans and SectionBackend collects them from every available
section. A pod whose plan changes only in generation records the new
plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
# Conflicts:
#	pkg/sandboxpolicy/sections_test.go
… branch

Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>
# Conflicts:
#	pkg/sandboxpolicy/igbackend/lsm.go
#	pkg/sandboxpolicy/igbackend/privilege.go
#	pkg/sandboxpolicy/igbackend/privilege_test.go
#	pkg/sandboxpolicy/integration/service.go
#	pkg/sandboxpolicy/sections_test.go
#	pkg/sandboxpolicy/status.go
Signed-off-by: Alon Liwsky <40373481+AlonLiwsky@users.noreply.github.com>

# Conflicts:
#	pkg/sandboxpolicy/controller.go
#	pkg/sandboxpolicy/integration/service.go
@AlonLiwsky

Copy link
Copy Markdown
Collaborator Author

Folded into #1075 to shorten the stack: the file-gadget branch now carries these commits unchanged, and review continues there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To Archive

Development

Successfully merging this pull request may close these issues.

2 participants