Description

Node agent seems to be allocating more memory than expected after the eBPF framework was moved to inspektor-gadget.

When monitoring ~20 containers the memory usage stands around 400-500Mb which seems to be a bad multiplier given users reported OOM killed (with 700Mb limit) on nodes with 30 containers.

This report is a live thread for sharing information and working on this issue.

Environment

OS: GKE
Version: 1.26

Steps To Reproduce

Steps to reproduce the behavior:

1. Install Kubescape Helm with relevancy
2. Install 20+ containers on the node
3. Monitor memory usage using Prometheus

Expected behavior

Given the above parameters and the logic of the node agent (events are written to boltdb and not kept in RAM) the memory requirements should not be more than 200Mb.

Actual Behavior

Well beyond 400Mb

Additional context

We have done some preliminary memory profiling and we got this: