Repository navigation
test: close end-to-end testing gaps for nvx.py sandbox - #437
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoOct 8, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The PID-limit probe miscounts container processes, and copied failure evidence is not bounded as documented.
3 open findings
What changed in this PR
Adds deterministic end-to-end coverage for the public managed sandbox lifecycle and security profile, closing #229.
Changes:
- Expands the guest security and resource-limit probe.
- Adds lifecycle acceptance and regression tests across all backends.
- Updates CI wiring, validation guards, and documentation.
| File | Description |
|---|---|
scripts/test_nvx_tools.py |
Tests CI wiring and smoke configuration. |
scripts/test_microvm_tests.py |
Tests lifecycle acceptance and cleanup. |
scripts/nvx_tools/sandbox_lifecycle.py |
Exposes runtime-process validation. |
scripts/nvx_tools/sandbox_lifecycle_tests.py |
Implements lifecycle acceptance. |
scripts/nvx_tools/microvm_tests.py |
Registers and dispatches the scenario. |
scripts/nvx_tools/managed_exec_tests.py |
Shares artifact and outcome helpers. |
guest/common/nvx-sandbox-smoke |
Expands security-profile checks. |
doc/usage.md |
Documents explicit scenarios. |
doc/run.md |
Documents lifecycle and probe contracts. |
doc/design/validation.md |
Describes validation coverage. |
doc/design/copilot-adversarial-testing.md |
Identifies the deterministic baseline. |
doc/contribute.md |
Documents scenario locations. |
doc/ci.md |
Describes backend acceptance coverage. |
.github/workflows/run-nvx-microvm-tests.yml |
Runs lifecycle and limit checks in CI. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Part of #229. /sbin/nvx-sandbox-smoke checked only the workload's UID and GID, its Ubuntu identity, and one scratch-backed /tmp write, so CI never verified most of what the sandbox profile promises: no capabilities, no_new_privs, private namespaces, the private device view, read-only sysfs, and enforced memory.max and pids.max. Profile ------- Before any share check, the probe now requires: - UID and GID 65534 as the real, effective, saved, and file-system IDs, no supplementary groups, and HOME=/nonexistent; - empty inheritable, permitted, effective, bounding, and ambient capability sets and NoNewPrivs: 1; - mount, PID, and UTS namespaces other than the initial ones, in which the agent runs. Linux 6.18 fixes the initial namespaces' inode numbers in its UAPI (enum init_ns_ino), the mount namespace's included. PID 1 of the workload's PID namespace must run as the workload, and the host name must be the configured one; - exactly one mount at /, /proc, /sys, /dev, /dev/pts, and /dev/shm, none of which propagates: an overlay root whose lower directories are the image layers and whose upper directory is on scratch, nosuid,nodev,noexec procfs, read-only sysfs, and a /dev tmpfs that holds exactly the runtime's thirteen entries, with its own devpts instance. Each managed exec enters its own namespaces before it mounts these, so a request that mounted into the agent's namespace would stack its mounts on those of earlier requests; - the agent-owned /container cgroup; and - image files that the workload cannot modify, and a /tmp write that lands in the overlay. Each failure prints an NVX-UBUNTU-SANDBOX-FAIL line that names the broken invariant and exits nonzero. Limits ------ `limits MEMORY_MAX PIDS_MAX`, given as sandbox arguments, checks that the memory controller kills an allocation of twice MEMORY_MAX with SIGKILL while a 1 MiB allocation succeeds, and that the pids controller lets the workload run PIDS_MAX processes, itself included. A subshell starts sleepers until a fork fails, and its exit trap kills and reaps them and records how many started, which must be PIDS_MAX - 2. A bound stops a subshell whose forks never fail, and twice MEMORY_MAX must fit the guest's free memory, so an unenforced limit fails the check instead of the guest. CI -- The one-shot Ubuntu smoke test on KVM, MSHV, and WHP now passes --memory-max 33554432 and --pids-max 8 and runs the probe with `limits 33554432 8`. The live-share runs repeat the profile checks. Tests ----- Shell unit tests run the probe's functions against fixture status, mountinfo, cgroup, command-line, and host-name files, with id and readlink stubbed. Valid fixtures pass, and each broken invariant fails with its message, including stacked and propagating mounts, a writable sysfs, an upper directory off scratch, a host devtmpfs, and the initial namespaces' inode numbers. They also check the limits' argument validation, that the sleeper subshell reaps what it starts, and the order in which the main body runs the profile, limit, and share checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/fix-e2e-testing-nvxpy-sandbox
branch
from
October 8, 2026 14:36
bc8ba45 to
d084db6
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new process-scan test leaks its spawned sleep process during cleanup.
1 open finding
3 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Part of #229. managed-exec-config runs provision, start, exec, stop, and deprovision through scripts/nvx.py, but it checks request-scoped execution options. Nothing exercised the lifecycle's own contract through the public interface: that a warm guest keeps its state across requests and restarts, that every invalid transition fails explicitly without leaking an OpenVMM process, control endpoint, capability, or runtime record, and that stop and deprovision remove only NVX's state while the supplied artifacts survive. The managed-lifecycle scenario covers part of this through OpenVMM's CLI, which bypasses NVX's lifecycle code. Scenario -------- The explicit sandbox-lifecycle scenario of test-microvm runs nvx.py sandbox as subprocesses against the real kernel, Alpine control initramfs, Ubuntu EROFS layer, and fresh copies of the Ubuntu scratch template. It checks, in order, that: - provision rejects a root --workload-user before it creates any state, and the guest refuses an identity that the Ubuntu image lacks, both before a one-shot workload starts and in a managed start, which then fails without leaving a runtime record, capability, socket, or OpenVMM process; - start, exec, stop, and deprovision fail on a state directory that was never provisioned and do not create it, a repeated provision keeps the configuration, and exec and stop fail before start; - a repeated start and a deprovision of a running sandbox fail without changing its runtime record, capability, or OpenVMM process, and the sandbox keeps serving requests; - arguments with leading, trailing, and embedded spaces, tabs, and newlines reach the workload intact, and a later request reads the file that an earlier one wrote; - nvx-sandbox-smoke passes its profile and `limits` checks in a sandbox provisioned with --hostname, --memory-max 33554432, and --pids-max 8, after the earlier requests; - a request that writes 2 MiB gets status 125, an output-limit report that holds only the operation, category, and status, and at most 1 MiB of output but more than 1 MiB less the agent's 32 KiB read chunk, since the agent drops the read that would exceed the bound; and a request whose --outcome-report path exists fails before its workload runs and leaves the file intact; - stop ends OpenVMM and its socket or pipe, leaves only config.json, openvmm.log, and a successful outcome.json, and leaves a cleanly unmounted scratch file system, after which exec and stop fail; - after a restart, the file written before stop is still there; - once OpenVMM is killed, exec and stop report stale runtime state and start refuses it; and - deprovision fails and keeps a foreign file in the state directory, and once that file is gone, it removes the directory, which a later exec does not recreate. After each step, exactly the OpenVMM processes that should run for the fixture do, found by their arguments in /proc on Linux and in Win32_Process on Windows. The EROFS layer's SHA-256 never changes, and the scratch image survives deprovision. On failure, the scenario stops what it started, kills every OpenVMM process that names its fixture, deprovisions both sandboxes, and preserves the fixture only when that cleanup fails. Its evidence holds bounded command observations (arguments, statuses, output sizes, and state-directory entries), the runtime record, OpenVMM logs, the VM-level and exec outcome reports, the probe transcript, and the refused one-shot run's console, each limited to its last 64 KiB. The command observations hold no workload output, and no evidence holds the control capability. Supporting changes ------------------ sandbox_lifecycle.runtime_process_running() exposes the process identity check that start, exec, stop, and deprovision already use, so the scenario applies the same definition of a running OpenVMM. managed_exec_tests shares its artifact loading and bounded-diagnostic helpers with the new scenario instead of duplicating them. Neither public scenario runs in the default suite, because packages omit the scratch template; the --scenario help and usage.md now say so for managed-exec-config as well. CI and docs ----------- The public managed sandbox acceptance step on KVM, MSHV, and WHP runs sandbox-lifecycle after managed-exec-config, and its evidence lands in the public-exec results that the job uploads on failure. run.md describes the scenario and the managed exec output bound, and ci.md, usage.md, validation.md, contribute.md, and the adversarial testing design describe the new coverage. Tests ----- Unit tests drive the scenario against a fake CLI that models the state directory, the guest's files, and OpenVMM processes. The complete run issues the expected sequence of operations and writes bounded evidence without the capability, and each of twelve injected regressions fails it with a specific message while cleanup still ends every process and removes the fixture. A failed cleanup preserves the fixture, and a noisy OpenVMM log leaves only its bounded end in the evidence. Other tests cover the scratch journal check, killing a recorded process, the control endpoint check for sockets and named pipes, the Linux process scan, the dispatcher, the default suite's exclusion of both public scenarios, runtime_process_running(), and the CI wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Part of #229. The validate-nvx action names each scripts/test_*.py module explicitly in its Linux and Windows unittest invocations. A new module that is left out never runs in CI, as scripts/test_control_session.py once ran in neither job. A unit test now requires that the modules each step names equal the modules in scripts/, so adding a module without adding it to both steps, or removing one, fails the CLI validation itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/fix-e2e-testing-nvxpy-sandbox
branch
from
October 8, 2026 14:50
d084db6 to
7d572fb
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-platform hypervisor lifecycle and security behavior requires final human review despite extensive automated coverage.
0 open findings
1 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-e2e-testing-nvxpy-sandbox
branch
October 8, 2026 15:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #229.
The public
nvx.py sandboxinterface lacked deterministic end-to-end coverage of its managed lifecycle contract and of the security profile that it promises. Since #229 was filed,managed-exec-configbegan drivingprovision -> start -> exec -> stop -> deprovisionthrough the CLI for request-scoped execution options (timeouts, nonzero exits, stdout and stderr, typed outcome reports), andscripts/test_control_session.pyjoined bothvalidate-nvxinvocations. This PR closes the remaining gaps.Three commits:
/sbin/nvx-sandbox-smokechecked only the UID/GID, the Ubuntu identity, and one/tmpwrite. Before any share check, it now requires the fixed identity as the real, effective, saved, and file-system IDs with no supplementary groups; empty inheritable, permitted, effective, bounding, and ambient capability sets andNoNewPrivs: 1; mount, PID, and UTS namespaces other than the initial ones (Linux 6.18 fixes their inode numbers in its UAPI,enum init_ns_ino, including the mount namespace's); exactly one, non-propagating mount at/,/proc,/sys,/dev,/dev/pts, and/dev/shm, with an overlay root over the image layers whose upper directory is on scratch, read-only sysfs, and a private/devwith exactly the runtime's entries; the agent-owned/containercgroup; and image files that the workload cannot modify. Withlimits MEMORY_MAX PIDS_MAX, it requires the memory controller to kill an allocation of twiceMEMORY_MAXwithSIGKILLand the pids controller to stop the workload atPIDS_MAXprocesses. The one-shot smoke step on every backend now runs it with--memory-max 33554432 --pids-max 8, and the live-share steps repeat the profile checks.sandbox-lifecyclescenario oftest-microvmrunsnvx.py sandboxas subprocesses against the real kernel, Alpine control initramfs, Ubuntu EROFS layer, and fresh ext4 scratch copies. It covers identity admission (a root identity rejected by the CLI; an identity that the image lacks refused by the guest in one-shot and managed mode), every fail-closed transition (operations on unprovisioned state, repeated provision, exec and stop before start and after stop, repeated start, deprovision while running, stale runtime state after OpenVMM is killed, and a foreign file in the state directory), warm-guest state across requests with whitespace-bearing arguments and across a stop and restart, the security probe after earlier requests, the 1 MiB output bound (the host must receive more than 1 MiB less one 32 KiB agent read and at most 1 MiB) and outcome reports, and the refusal of an existing report path before the workload runs. After each step, it requires exactly the expected OpenVMM processes for its fixture (found by their arguments) and the expected state-directory entries, and it requires that the control socket or named pipe goes away, the EROFS layer's digest never changes, and the scratch image survives deprovision. On failure, it stops what it started, kills any OpenVMM process that names its fixture, and keeps bounded evidence: command observations without workload output, the runtime record, OpenVMM logs, outcome reports, and the probe transcript, each capped at its last 64 KiB, but never the control capability. CI runs it aftermanaged-exec-configon KVM, MSHV, and WHP.validate-nvxsteps to name exactly thescripts/test_*.pymodules, so a new module cannot silently skip CI astest_control_session.pyonce did.doc/run.md,doc/ci.md,doc/usage.md,doc/design/validation.md,doc/contribute.md, and the adversarial-testing design describe the resulting coverage. #102 (adaptive adversarial testing) and #216 (sandbox virtio-fs acceptance) stay independent.Acceptance criteria
managed-exec-configandsandbox-lifecyclein the public acceptance step of every microVM job; a unit test pins the wiringnvx.py sandbox execmanaged-exec-config, plus theoutput-limitbound and the refusal of an existing report path insandbox-lifecycleno_new_privs, cgroup, lower-layer, and scratch invariantsscripts/test_control_session.pyruns in the Linux and Windows CLI validation jobsValidation
Local (Windows):
ruff check,ruff format --check, andpyright(strict) for Linux and Windows.shellcheckandshfmtimages over every script thatcheck-qualitylists, andactionlinton the changed workflow (no new findings; its four SC2012 infos predate this PR).validate-nvxcommand set. One existing test,test_materialize_kernel_provenance_inputs_bypasses_mutable_index, fails only on this workstation, whose global Git configuration requires SSH commit signing; it passes on the Linux host below.Bare metal, in fresh clones of this branch: OpenVMM, kernel, and initramfs from
v0.1.0-dev.8de93bcd226c(same OpenVMM pin,0eccda895), the Ubuntu distro layer rebuilt from this branch with CI's Docker target, and the scratch template made as CI makes it:managed-exec-config+sandbox-lifecycleAfter the review fixes, the acceptance passed again on all three hosts at
d084db699, with the KVM layer rebuilt; the final head differs from it only in a Linux unit test. The forwarded output totals were 1,032,192, 1,048,576, and 1,040,384 bytes, so an exact 1 MiB check would be flaky. prometheus32 also passed the full Linuxvalidate-nvxset (786 tests). On its guest kernel, PID 1's namespaces aremnt:[4026531832],pid:[4026531836], anduts:[4026531838], the constants that the probe compares against. No OpenVMM process or fixture remained after any run.CI on the pre-review head
bc8ba458cpassed every job, and the KVM, MSHV, and WHP microVM jobs logged thesandbox-lifecyclescenario and the probe'sLIMITS-OKandPROFILE-OKmarkers.