Skip to content

guest: settle direct-mode execs only once their processes are reaped - #363

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/fix-issue-360-validate-implementation
Oct 5, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/fix-issue-360-validate-implementation

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Summary

Closes #360.

In direct mode, the managed agent runs as PID 1 and contains each workload in the nvx-exec cgroup. settle_exec_cgroup() returned as soon as cgroup.events reported populated 0. That field drops a killed task in cgroup_exit(), before exit_notify() turns it into a zombie that PID 1 has to reap. A sleep that finished exiting between the agent's last reap and its read of cgroup.events therefore stayed in /proc as an unreaped zombie after the agent had reported the exit. The next exec's pre-check read the same signal, so the next workload started while the zombie was still visible. That is how pgrep -x sleep in openvmm_lifecycle_on_a_real_hypervisor printed 61 on Linux/MSHV. Nothing escaped containment: every leftover had already been killed.

Changes

  • Settlement barrier (guest/common/nvx-managed-agent.c): when nvx-exec has no pids.current, prepare_exec_cgroup() enables the pids controller in the root cgroup's cgroup.subtree_control. Settlement now reaps and then waits until pids.current reads 0. The pids controller charges a task until release_task() frees it, which for a process happens when it is reaped. A zero count therefore means that every workload process has exited and been reaped. The 2 s budget and the containment-failed outcome are unchanged, and a missing or malformed pids.current also reports containment-failed.
  • Pre-check: the direct-mode pre-check reaps first and then requires the same zero count. It refuses an exec cgroup that still holds unreaped tasks. A workload that exits after its settlement timed out no longer blocks later execs as a zombie that nothing reaps.
  • Kernel guard: CONFIG_CGROUP_PIDS=y joins REQUIRED_SANDBOX_CONFIG, and kernel/config-microvm already sets it. The guard changes the kernel cache key, so CI rebuilds the kernel once for this PR.
  • Tests (scripts/test_nvx_tools.py): the fake-cgroup tests now cover:
    • strict pids.current parsing;
    • enabling the controller only while pids.current is missing;
    • that settlement waits for an exiting process and reaps it, and that the pre-check reaps before it counts, through a harness whose waitpid() drops the charge on reap, as the kernel does;
    • that a cgroup that reports populated 0 while it still charges a task does not settle.
  • Docs: the agent design document describes the new barrier.

The e2e pgrep assertions in openvmm_e2e.rs stay unchanged.

Validation

Stress harness with the real run_exec() (WSL2, Linux 6.18.33, 20 CPUs)

The harness #includes the agent and drives its real run_exec() in direct mode. It runs as a child subreaper, standing in for PID 1, in a delegated cgroup v2 scope, and doubles as the setpriv shim and the launch helper. After each detached exec, it lists its own children and reads pids.current. It then runs pgrep -x -P <harness> sleep || echo none through run_exec(). Each scenario ran 5,000 iterations:

Scenario Fixed: leftovers / pgrep hits Before: leftovers / pgrep hits
L839 workload, idle 0 / 0 0 / 0
L839 workload, 40 busy loops 0 / 0 5 / 2
L839 workload, 500 µs between the reap and the read 0 / 0 4 / 3
Detached sleeps (>/dev/null), idle 0 / 0 64 / 32
Detached sleeps, 500 µs between the reap and the read 0 / 0 1,852 / 1,725

In every scenario, the iterations with leftovers were exactly those in which pids.current was nonzero when the exec reported its exit. The sampled leftovers were zombies (Z with PF_EXITING). No live process outlived an exec in either build, and the fix caused no settlement timeouts.

Real hypervisor (Windows/WHP, single-vCPU guest)

The kernel and OpenVMM (4355c010) come from a dev release whose kernel provenance matches this branch's kernel inputs. The Alpine initramfs was built from this branch.

  • python scripts\nvx.py test-aci-edge-sandboxes --backend whp: all 7 tests pass, including openvmm_lifecycle_on_a_real_hypervisor and environments_leave_the_workload_contained.
  • A temporary in-guest stress test (not committed) ran both detached workloads 5,000 times each, or 1,000 times for the third row, and followed each with pgrep -x sleep || echo none:
Agent Delay injected between the reap and the read Detached execs Leftovers that pgrep found
Fixed none 10,000 0
Before none 10,000 0 (one vCPU rarely hits the window)
Before 500 µs 2,000 662
Fixed 500 µs 10,000 0

Local checks

  • ruff check, ruff format --check, and pyright for Linux and Windows pass.
  • The validate-nvx unit tests pass on Windows and on Linux (551 tests, run in WSL from a shallow clone of this commit).
  • The agent builds with the guest flags (-static -Os -s -Wall -Wextra -Werror).

I did not run anything locally on Linux/KVM or Linux/MSHV hardware. CI covers both.

Notes

Harness: settle_stress.c
/*
 * Stress harness for the direct-mode exec settle barrier of
 * guest/common/nvx-managed-agent.c (microsoft/nvx#360).
 *
 * The harness includes the agent and drives its real run_exec() in direct
 * mode as a child subreaper, standing in for PID 1, inside a delegated cgroup
 * v2 scope that stands in for the guest's root cgroup. Identity changes are
 * recorded instead of made, and the harness doubles as the setpriv shim and
 * as the agent's launch helper, so the agent runs unprivileged.
 *
 * Each iteration runs the detached workload of openvmm_e2e.rs, then:
 *   - lists the harness's children and reads nvx-exec/pids.current as soon as
 *     run_exec() returns, which is when the agent has reported the exit;
 *   - runs `pgrep -x -P <harness> sleep || echo none` through run_exec(), as
 *     the next exec of openvmm_e2e.rs does with `pgrep -x sleep`.
 *
 * Build with -DAGENT_SOURCE='"agent.c"' -DCGROUP_ROOT='"<delegated scope>"',
 * plus -DHARNESS_ENABLE_PIDS for an agent that does not enable the pids
 * controller itself. WORKLOAD replaces the workload; SETTLE_GAP_US delays every
 * read of cgroup.events or pids.current, widening the window between the
 * agent's reap and its read.
 */
#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <grp.h>
#include <libgen.h>
#include <limits.h>
#include <poll.h>
#include <signal.h>
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/mount.h>
#include <sys/prctl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <termios.h>
#include <time.h>
#include <unistd.h>

#ifndef AGENT_SOURCE
#error "define AGENT_SOURCE"
#endif
#ifndef CGROUP_ROOT
#error "define CGROUP_ROOT"
#endif

#define PF_EXITING_FLAG 0x00000004U

static long harness_gap_us;

static int harness_setgroups(size_t size, const gid_t *list)
{
    (void)size;
    (void)list;
    return 0;
}

static int harness_setegid(gid_t gid)
{
    (void)gid;
    return 0;
}

static int harness_seteuid(uid_t uid)
{
    (void)uid;
    return 0;
}

static int harness_open(const char *path, int flags, ...);

#define setgroups(size, list) harness_setgroups(size, list)
#define setegid(gid) harness_setegid(gid)
#define seteuid(uid) harness_seteuid(uid)
#define open(...) harness_open(__VA_ARGS__)
#define main managed_agent_main
#include AGENT_SOURCE
#undef main
#undef open
#undef setgroups
#undef setegid
#undef seteuid

static int harness_ends_with(const char *text, const char *suffix)
{
    size_t text_length = strlen(text);
    size_t suffix_length = strlen(suffix);

    return text_length >= suffix_length &&
           strcmp(text + text_length - suffix_length, suffix) == 0;
}

static int harness_open(const char *path, int flags, ...)
{
    mode_t mode = 0;

    if ((flags & O_CREAT) != 0 || (flags & O_TMPFILE) == O_TMPFILE) {
        va_list arguments;

        va_start(arguments, flags);
        mode = (mode_t)va_arg(arguments, int);
        va_end(arguments);
    }
    if (harness_gap_us > 0 && (harness_ends_with(path, "/cgroup.events") ||
                               harness_ends_with(path, "/pids.current"))) {
        usleep((useconds_t)harness_gap_us);
    }
    return open(path, flags, mode);
}

static int harness_write_text(const char *path, const char *text)
{
    int fd = open(path, O_WRONLY | O_CLOEXEC);
    ssize_t count;
    int status;

    if (fd < 0) {
        return -1;
    }
    count = write(fd, text, strlen(text));
    status = errno;
    close(fd);
    errno = status;
    return count == (ssize_t)strlen(text) ? 0 : -1;
}

static long harness_pids_current(void)
{
    char buffer[64];
    int fd = open(EXEC_CGROUP "/pids.current", O_RDONLY | O_CLOEXEC);
    ssize_t count;

    if (fd < 0) {
        return -1;
    }
    count = read(fd, buffer, sizeof(buffer) - 1);
    close(fd);
    if (count <= 0) {
        return -1;
    }
    buffer[count] = '\0';
    return strtol(buffer, NULL, 10);
}

/* Lists this process's children with their states; returns how many there are. */
static int harness_scan_children(pid_t self, char *states, size_t capacity)
{
    DIR *proc = opendir("/proc");
    struct dirent *entry;
    int count = 0;
    size_t used = 0;

    states[0] = '\0';
    while (proc != NULL && (entry = readdir(proc)) != NULL) {
        char path[300];
        char buffer[1024];
        char *open_paren;
        char *close_paren;
        char state;
        int ppid;
        unsigned int flags;
        ssize_t length;
        int fd;

        if (entry->d_name[0] < '0' || entry->d_name[0] > '9') {
            continue;
        }
        snprintf(path, sizeof(path), "/proc/%s/stat", entry->d_name);
        fd = open(path, O_RDONLY | O_CLOEXEC);
        if (fd < 0) {
            continue;
        }
        length = read(fd, buffer, sizeof(buffer) - 1);
        close(fd);
        if (length <= 0) {
            continue;
        }
        buffer[length] = '\0';
        open_paren = strchr(buffer, '(');
        close_paren = strrchr(buffer, ')');
        if (open_paren == NULL || close_paren == NULL) {
            continue;
        }
        *close_paren = '\0';
        if (sscanf(close_paren + 2, "%c %d %*d %*d %*d %*d %u", &state, &ppid, &flags) !=
                3 ||
            ppid != self) {
            continue;
        }
        ++count;
        if (used < capacity) {
            int written = snprintf(
                states + used,
                capacity - used,
                "%s%s(%s):%c%s",
                count > 1 ? "," : "",
                entry->d_name,
                open_paren + 1,
                state,
                (flags & PF_EXITING_FLAG) != 0 ? "+exiting" : "");

            if (written > 0) {
                used += (size_t)written;
            }
        }
    }
    if (proc != NULL) {
        closedir(proc);
    }
    return count;
}

struct harness_outcome {
    char output[4096];
    size_t output_length;
    char errors[4096];
    size_t errors_length;
    uint8_t final_kind;
    int32_t status;
    char category[64];
};

static uint8_t harness_session_buffer[1U << 20];

static void harness_append(char *buffer, size_t *length, size_t capacity, const uint8_t *data, size_t count)
{
    size_t room = capacity - 1 - *length;
    size_t copy = count < room ? count : room;

    memcpy(buffer + *length, data, copy);
    *length += copy;
    buffer[*length] = '\0';
}

/* Collects the frames that one run_exec() call sent to the control session. */
static void harness_collect(int fd, struct harness_outcome *outcome)
{
    size_t length = 0;
    size_t offset = 0;

    memset(outcome, 0, sizeof(*outcome));
    for (;;) {
        ssize_t count = read(
            fd, harness_session_buffer + length, sizeof(harness_session_buffer) - length);

        if (count > 0) {
            length += (size_t)count;
            continue;
        }
        if (count < 0 && errno == EINTR) {
            continue;
        }
        break;
    }
    while (offset + OUTER_HEADER_LEN <= length) {
        const uint8_t *header = harness_session_buffer + offset;
        uint32_t payload_len = read_u32(header + 40);
        const uint8_t *payload = header + OUTER_HEADER_LEN;

        if (offset + OUTER_HEADER_LEN + payload_len > length) {
            break;
        }
        if (header[6] == OUTER_DATA && payload_len >= APP_HEADER_LEN) {
            uint8_t kind = payload[5];
            uint32_t body_len = read_u32(payload + 20);
            const uint8_t *body = payload + APP_HEADER_LEN;

            if (kind == APP_STDOUT) {
                harness_append(
                    outcome->output, &outcome->output_length, sizeof(outcome->output), body, body_len);
            } else if (kind == APP_STDERR) {
                harness_append(
                    outcome->errors, &outcome->errors_length, sizeof(outcome->errors), body, body_len);
            } else {
                size_t category_length = 0;

                outcome->final_kind = kind;
                outcome->status = (int32_t)read_u32(payload + 16);
                outcome->category[0] = '\0';
                harness_append(
                    outcome->category, &category_length, sizeof(outcome->category), body, body_len);
            }
        }
        offset += OUTER_HEADER_LEN + payload_len;
    }
}

/* Stands in for setpriv: drops its options and runs the rest unchanged. */
static int harness_setpriv(int argc, char **argv)
{
    int index = 1;

    while (index < argc && strncmp(argv[index], "--", 2) == 0) {
        index += strcmp(argv[index], "--reuid") == 0 || strcmp(argv[index], "--regid") == 0 ? 2 : 1;
    }
    if (index >= argc) {
        return 127;
    }
    if (strcmp(argv[index], "/sbin/nvx-managed-agent") == 0) {
        argv[index] = "nvx-managed-agent";
        execv("/proc/self/exe", argv + index);
    } else {
        execvp(argv[index], argv + index);
    }
    return 127;
}

/* Reaps every child; returns how many are still alive after 2 s, then kills them. */
static int harness_reset(pid_t self)
{
    uint64_t started = monotonic_milliseconds();

    for (;;) {
        int status;
        pid_t pid = waitpid(-1, &status, WNOHANG);

        if (pid > 0 || (pid < 0 && errno == EINTR)) {
            continue;
        }
        if (pid < 0 && errno == ECHILD) {
            return 0;
        }
        if (monotonic_milliseconds() - started >= 2000) {
            char states[1024];
            int alive = harness_scan_children(self, states, sizeof(states));

            fprintf(stderr, "alive 2 s after the exec: %s\n", states);
            (void)kill_exec_cgroup();
            while (waitpid(-1, &status, 0) > 0 || errno == EINTR) {
            }
            return alive;
        }
        usleep(1000);
    }
}

int main(int argc, char **argv)
{
    char shim_directory[] = "/tmp/nvx-settle-shim-XXXXXX";
    char shim[PATH_MAX];
    char self_path[PATH_MAX];
    char path_variable[8192];
    char pgrep_script[128];
    const char *workload = getenv("WORKLOAD");
    const char *gap = getenv("SETTLE_GAP_US");
    pid_t self = getpid();
    int session_pipe[2];
    struct control_session session = {0};
    struct agent_config config = {
        .rootfs = "-",
        .hostname = "nvx-managed",
        .uid = "65534",
        .gid = "65534",
        .user = "nobody",
        .home = "/",
        .direct = 1,
    };
    struct exec_config exec_config = {0};
    char *detached_argv[] = {"/bin/sh", "-c", NULL, NULL};
    char *pgrep_argv[] = {"/bin/sh", "-c", pgrep_script, NULL};
    int iterations;
    int index;
    int detached_unexpected = 0;
    int settle_failed = 0;
    int leftover_after_exec = 0;
    int pids_nonzero_after_exec = 0;
    int pgrep_saw_sleep = 0;
    int precheck_refused = 0;
    int pgrep_unexpected = 0;
    int escaped_live = 0;
    int examples = 0;
    uint64_t exec_total_ms = 0;
    ssize_t self_length;

    if (argc >= 2 && strcmp(argv[1], "--exec-config-fd") == 0) {
        return launch_workload(argc, argv);
    }
    if (strcmp(basename(argv[0]), "setpriv") == 0) {
        return harness_setpriv(argc, argv);
    }
    if (argc != 2 || (iterations = atoi(argv[1])) <= 0) {
        fprintf(stderr, "usage: %s ITERATIONS\n", argv[0]);
        return 2;
    }
    detached_argv[2] =
        (char *)(workload != NULL ? workload : "setsid sleep 60 & sleep 60 & echo spawned");
    harness_gap_us = gap != NULL ? strtol(gap, NULL, 10) : 0;
    snprintf(
        pgrep_script, sizeof(pgrep_script), "pgrep -x -P %ld sleep || echo none", (long)self);

    /* A non-root cgroup enables controllers only while it holds no process itself. */
    if ((mkdir(CGROUP_ROOT "/agent", 0755) != 0 && errno != EEXIST) ||
        harness_write_text(CGROUP_ROOT "/agent/cgroup.procs", "0\n") != 0) {
        perror("agent cgroup");
        return 2;
    }
#ifdef HARNESS_ENABLE_PIDS
    if (harness_write_text(CGROUP_ROOT "/cgroup.subtree_control", "+pids") != 0) {
        perror("pids controller");
        return 2;
    }
#endif
    if (prctl(PR_SET_CHILD_SUBREAPER, 1) != 0) {
        perror("PR_SET_CHILD_SUBREAPER");
        return 2;
    }
    self_length = readlink("/proc/self/exe", self_path, sizeof(self_path) - 1);
    if (self_length <= 0 || mkdtemp(shim_directory) == NULL) {
        perror("shim");
        return 2;
    }
    self_path[self_length] = '\0';
    snprintf(shim, sizeof(shim), "%s/setpriv", shim_directory);
    if (symlink(self_path, shim) != 0) {
        perror("shim");
        return 2;
    }
    snprintf(
        path_variable,
        sizeof(path_variable),
        "%s:%s",
        shim_directory,
        getenv("PATH") != NULL ? getenv("PATH") : "/usr/bin:/bin");
    setenv("PATH", path_variable, 1);
    if (pipe2(session_pipe, O_CLOEXEC) != 0) {
        perror("session");
        return 2;
    }
    (void)fcntl(session_pipe[0], F_SETPIPE_SZ, 1 << 20);
    if (make_nonblocking(session_pipe[0]) != 0) {
        perror("session");
        return 2;
    }
    session.fd = session_pipe[1];

    for (index = 0; index < iterations; ++index) {
        struct harness_outcome detached;
        struct harness_outcome check;
        char states[1024];
        uint64_t started = monotonic_milliseconds();
        long tasks;
        int leftovers;

        if (run_exec(&session, &config, 2U * (uint64_t)index + 1U, 0, detached_argv, &exec_config) !=
            0) {
            fprintf(stderr, "iteration %d: run_exec failed\n", index);
            return 2;
        }
        exec_total_ms += monotonic_milliseconds() - started;
        /* The agent has reported the exit; nothing of the workload may be left. */
        tasks = harness_pids_current();
        leftovers = harness_scan_children(self, states, sizeof(states));
        harness_collect(session_pipe[0], &detached);
        if (detached.final_kind == APP_ERROR && strcmp(detached.category, "containment-failed") == 0) {
            ++settle_failed;
        } else if (
            detached.final_kind != APP_EXIT || detached.status != 0 ||
            strcmp(detached.category, "exit") != 0 || strcmp(detached.output, "spawned\n") != 0 ||
            detached.errors_length != 0) {
            ++detached_unexpected;
            if (detached_unexpected <= 3) {
                fprintf(
                    stderr,
                    "iteration %d: detached exec answered kind=0x%x status=%d category=%s "
                    "stdout=[%s] stderr=[%s]\n",
                    index,
                    detached.final_kind,
                    detached.status,
                    detached.category,
                    detached.output,
                    detached.errors);
            }
        }
        if (leftovers > 0) {
            ++leftover_after_exec;
        }
        if (tasks != 0) {
            ++pids_nonzero_after_exec;
        }

        if (run_exec(&session, &config, 2U * (uint64_t)index + 2U, 0, pgrep_argv, &exec_config) != 0) {
            fprintf(stderr, "iteration %d: run_exec failed\n", index);
            return 2;
        }
        harness_collect(session_pipe[0], &check);
        if (check.final_kind == APP_ERROR && strcmp(check.category, "containment-failed") == 0) {
            ++precheck_refused;
        } else if (check.final_kind != APP_EXIT || check.status != 0) {
            ++pgrep_unexpected;
        } else if (strcmp(check.output, "none\n") != 0) {
            ++pgrep_saw_sleep;
        }
        if ((leftovers > 0 || tasks != 0 || strcmp(check.output, "none\n") != 0) && examples < 5) {
            char *newline;

            ++examples;
            while ((newline = strchr(check.output, '\n')) != NULL) {
                *newline = ' ';
            }
            printf(
                "iteration %d: exec reported with pids.current=%ld; children [%s]; "
                "next exec kind=0x%x category=%s pgrep printed [%s]\n",
                index,
                tasks,
                states,
                check.final_kind,
                check.category,
                check.output);
        }
        escaped_live += harness_reset(self);
    }
    unlink(shim);
    rmdir(shim_directory);
    printf(
        "iterations=%d detached_unexpected=%d settle_containment_failed=%d "
        "leftover_after_exec=%d pids_current_nonzero_after_exec=%d "
        "pgrep_saw_sleep=%d precheck_refused=%d pgrep_unexpected=%d escaped_live=%d "
        "mean_detached_exec_ms=%.1f settle_gap_us=%ld\n",
        iterations,
        detached_unexpected,
        settle_failed,
        leftover_after_exec,
        pids_nonzero_after_exec,
        pgrep_saw_sleep,
        precheck_refused,
        pgrep_unexpected,
        escaped_live,
        (double)exec_total_ms / iterations,
        harness_gap_us);
    return 0;
}
Runner: run.sh and scenario.sh
#!/bin/bash
# Builds the settle stress harness against one agent source and runs it in a
# delegated cgroup v2 scope.
#   run.sh VERSION LABEL ITERATIONS [BUSY_LOOPS]
# VERSION is "fixed" (/tmp/nvx-settle/agent-fixed.c) or "baseline"
# (/tmp/nvx-settle/agent-baseline.c). WORKLOAD and SETTLE_GAP_US pass through.
set -euo pipefail
version=$1
label=$2
iterations=$3
busy=${4:-0}
here=$(cd "$(dirname "$0")" && pwd)
work=/tmp/nvx-settle
uid=$(id -u)
unit="nvx-settle-$version-$label-$$"
scope="/sys/fs/cgroup/user.slice/user-$uid.slice/user@$uid.service/app.slice/$unit.scope"
extra=()
case "$version" in
fixed) ;;
baseline) extra=(-DHARNESS_ENABLE_PIDS) ;;
*) echo "unknown version $version" >&2; exit 2 ;;
esac
mkdir -p "$work"
gcc -O2 -Wall -Wextra -Werror \
    "-DAGENT_SOURCE=\"$work/agent-$version.c\"" "-DCGROUP_ROOT=\"$scope\"" "${extra[@]}" \
    -o "$work/$unit" "$here/settle_stress.c"
loops=()
cleanup() {
    for pid in "${loops[@]}"; do kill "$pid" 2>/dev/null || true; done
    rm -f "$work/$unit"
}
trap cleanup EXIT
for _ in $(seq 1 "$busy"); do
    sh -c 'while :; do :; done' &
    loops+=("$!")
done
export XDG_RUNTIME_DIR=/run/user/$uid
echo "== $version $label iterations=$iterations busy=$busy workload=${WORKLOAD:-default} gap=${SETTLE_GAP_US:-0}"
systemd-run --user --scope --unit="$unit" -p Delegate=yes -p TasksMax=infinity --quiet \
    "$work/$unit" "$iterations"
#!/bin/bash
# Runs one stress scenario against one agent version.
#   scenario.sh VERSION SCENARIO ITERATIONS
# SCENARIO: l839-idle, l839-busy, detached-idle, detached-gap, l839-gap
set -euo pipefail
version=$1
scenario=$2
iterations=$3
detached="setsid sleep 60 >/dev/null 2>&1 & sleep 60 >/dev/null 2>&1 & echo spawned"
busy=$(($(nproc) * 2))
case "$scenario" in
l839-idle) bash run.sh "$version" "$scenario" "$iterations" ;;
l839-busy) bash run.sh "$version" "$scenario" "$iterations" "$busy" ;;
l839-gap) SETTLE_GAP_US=500 bash run.sh "$version" "$scenario" "$iterations" ;;
detached-idle) WORKLOAD="$detached" bash run.sh "$version" "$scenario" "$iterations" ;;
detached-gap) WORKLOAD="$detached" SETTLE_GAP_US=500 bash run.sh "$version" "$scenario" "$iterations" ;;
*) echo "unknown scenario $scenario" >&2; exit 2 ;;
esac

In direct mode, the managed agent runs as PID 1 and contains each
workload in the nvx-exec cgroup. After an exec, settle_exec_cgroup()
killed the cgroup and returned as soon as cgroup.events reported
"populated 0". That field counts only live processes: do_exit() drops a
killed task from it in cgroup_exit(), before exit_notify() turns the
task into a zombie that PID 1 must reap. A process that passed both
points between the agent's last reap and its read of cgroup.events
therefore stayed in /proc as an unreaped zombie after the agent had
reported the exit. The agent reaps only inside run_exec(), and the next
exec's containment pre-check read the same signal, so the next workload
started while the zombie was still visible. That is how
`pgrep -x sleep` in openvmm_lifecycle_on_a_real_hypervisor printed
PID 61 on Linux/MSHV. Every leftover had already been killed, so
nothing escaped containment.

When nvx-exec lacks pids.current, the agent now enables the pids
controller in the root cgroup's cgroup.subtree_control, and it settles
on that count instead. The pids controller charges a task from its
fork, or its migration into the cgroup, until release_task() frees it,
which for a process happens when it is reaped. Once the agent has
reaped what it inherited, pids.current therefore reads zero only when
every process of the workload has exited and been reaped. Settlement
keeps its 2 s budget. It still reports containment-failed when the
count stays above zero, when cgroup.kill fails, or when pids.current is
missing or malformed. The direct-mode pre-check reaps first and then
requires the same zero count. It refuses an exec cgroup that still
holds unreaped tasks, while a workload that exited after its settlement
timed out no longer blocks later execs as a zombie that nothing reaps.
Failing to enable the controller refuses the exec as launch-failed,
like the other failures to prepare the cgroup, and the kernel build now
requires CONFIG_CGROUP_PIDS=y with the other sandbox options.

Replace the fake-cgroup tests of cgroup.events parsing with tests of
pids.current parsing, and check that preparation enables the pids
controller only while nvx-exec lacks pids.current. A harness whose
waitpid() stops charging a process when it reaps it, as the pids
controller does, checks that settlement waits for an exiting workload
process and reaps it, and that the pre-check reaps an exited one before
it counts. A cgroup that reports populated 0 while it still charges a
task must not settle. Document the barrier in the agent design.

Validation outside CI:

- A harness ran the real run_exec() as a child subreaper in a
  delegated cgroup v2 scope on WSL2 (Linux 6.18.33, 20 CPUs), with
  5,000 iterations of each scenario from the issue. The fixed agent
  left nothing behind in 25,000 iterations. The previous agent left
  leftovers in 1,925 of them, and the next exec's pgrep found 1,762 of
  those: 5 leftovers for the L839 workload under load, 4 for it with
  500 us between the reap and the read, and 64 and 1,852 for the
  detached variant, idle and with the gap.
- On Windows/WHP with a single-vCPU guest, the aci_edge_sandboxes e2e
  tests pass. In 10,000 detached execs, each followed by
  `pgrep -x sleep || echo none`, the fixed agent left nothing behind.
  With a 500 us sleep injected between the reap and the read, the
  previous agent left a sleep for pgrep in 662 of 2,000 execs, and the
  fixed agent left none in 10,000.

Fixes #360.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 4, 2026 23:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused implementation addresses the race with comprehensive regression coverage and consistent documentation.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes #360 by ensuring direct-mode workloads are reaped before an execution is reported complete.

Changes:

  • Uses pids.current as the settlement and pre-check barrier.
  • Requires pids-controller kernel support and adds regression tests.
  • Updates the managed-agent architecture documentation.
File Description
guest/​common/​nvx-managed-agent.c Enables pids accounting and waits for reaping.
scripts/​test_nvx_tools.py Tests parsing, controller setup, reaping, and timeouts.
scripts/​nvx_tools/​build_constants.py Requires CONFIG_CGROUP_PIDS=y.
doc/​design/​sandbox-filesystem-and-agent-architecture.md Documents the new settlement barrier.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@ppenna
Pedro Henrique Penna (ppenna) merged commit 2508acd into dev Oct 5, 2026
27 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/fix-issue-360-validate-implementation branch October 5, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: managed agent finishes a direct-mode exec before reaping its killed processes, failing the aci_edge_sandboxes lifecycle test on Linux/MSHV

2 participants