[Pedro Henrique Penna (ppenna)]

nvx.py sandbox now identifies OpenVMM by its process ID and start time, so a recycled PID or an unreaped OpenVMM no longer wedges stop, deprovision, and exec.

Fixes #369

Problem

sandbox start recorded only OpenVMM's PID in runtime.json, and _process_running counted any process with that PID as OpenVMM:

• On Linux, os.kill(pid, 0) succeeds for a zombie, and the PermissionError that another user's process raises also counted as running.
• On Windows, any process that the caller could open and that had not exited counted.

Once OpenVMM was gone but its PID named a live process, no CLI operation recovered the sandbox: stop waited out --timeout on the dead control endpoint, deprovision refused ("sandbox must be stopped before deprovision"), and start refused the stale record. That happens after OpenVMM crashes or the host restarts with a persistent --state-dir. Separately, when nothing reaps OpenVMM (for example, a container whose PID 1 lacks --init), it stays a zombie, so even an acknowledged STOP ended in "OpenVMM did not terminate after managed stop" with runtime.json left behind.

Fix

The implementation follows the ProcessIdentity design of the aci_edge_sandboxes backend:

• Record a start time: start writes start_time next to pid. On Linux, it is field 22 (starttime) of /proc/<pid>/stat; on Windows, the creation time from GetProcessTimes. start reads the time before it releases the child, so the PID cannot have been reused yet. If OpenVMM already exited or cannot be identified, start fails without publishing a record instead of waiting for the endpoint timeout.
• Check the identity: a process counts as OpenVMM only while its start time matches the record, and a zombie (Z/X) counts as exited. On Windows, the check now waits on the process handle (WaitForSingleObject) instead of comparing the exit code with STILL_ACTIVE, which an exit status of 259 also matches. A process that the caller cannot open still counts as gone, because OpenVMM runs as the caller.
• Older records: a runtime.json written by an earlier NVX version has no start_time and is still identified by its PID alone, as before, except that a zombie now counts as exited.
• Inconclusive checks fail the operation: if the process state cannot be read (an unexpected /proc or OpenProcess error, or a malformed stat line), the operation fails with "cannot determine whether OpenVMM process N is running" instead of guessing. deprovision therefore never removes the state of a VM that may still run. A non-integer, boolean, or negative start_time is rejected.
• Docs: doc/run.md describes the identity check and how older records are treated.

Tests

New tests in scripts/test_nvx_tools.py cover:

• the stat parser, including a command name that contains spaces and parentheses
• a start-time mismatch, and an exited child
• a zombie child (Linux; created with waitid(WNOWAIT))
• stop after an acknowledged STOP while OpenVMM stays a zombie (Linux)
• a reused PID through stop, exec, and deprovision
• older records and malformed start times
• start failures when OpenVMM exits early or cannot be identified

The two existing start tests now also check the recorded identity.

Validation

I ran the real scripts/nvx.py sandbox CLI against real OpenVMM microVMs on bare-metal Linux/KVM, Linux/MSHV, and Windows/WHP hosts. The artifacts were the v0.1.0-dev.2472412feb45 release (OpenVMM 974f356), at a3d929a with and without this change.

Scenario	Before	After
OpenVMM exits but is never reaped: a subreaper that never reaps stands in for a container init (KVM, MSHV)	stop fails with "OpenVMM did not terminate" after --timeout; start and deprovision refuse; runtime.json stays	stop succeeds in 0.2–0.4 s and removes runtime.json
After a crash, the record names root-owned PID 1, as after a reboot (KVM, MSHV)	exec and stop time out on the dead endpoint; start and deprovision refuse	exec and stop report stale state in ≤0.4 s; deprovision succeeds
Real Linux PID reuse: about 4.19M forks over 27–34 s until an unrelated sleep received OpenVMM's PID (KVM, MSHV)	Same failures	Same as the previous row
Real Windows PID reuse: an unrelated cmd.exe received OpenVMM's PID after 1.9k–23.7k process creations (WHP)	Same failures	Same as the previous row
Normal lifecycle: provision, start, exec, stop, deprovision (all three)	Passes	Passes, and runtime.json includes start_time
Sandbox started by the previous code and stopped by this code (all three; also the zombie case on Linux)	—	Passes
test-microvm --scenario managed-exec-config	—	Passes on kvm, mshv, and whp

Other checks:

• Unit tests: the validate-nvx unit suites (678 tests) pass on Windows, on the KVM host (Python 3.14), and on the MSHV host (Python 3.12). ruff check, ruff format --check, and strict pyright for both Linux and Windows are clean.
• Rebase: the branch is based on ae0c3e9, which only updates the data/ baselines on top of a3d929a. I re-ran the unit suites and quality gates there.
• Performance: in interleaved A/B runs on WHP, the median sandbox start time went from 1.943 s to 1.946 s. The identity check costs about 26–60 µs per call.

Notes

• start still refuses any existing runtime.json, stale or not, which matches the expected behavior in the issue. To recover, run deprovision (which now succeeds) and provision again. Having start or stop clear stale records automatically would be a separate behavior change.
• This change does not touch the adversarial executor's own _process_running.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the stated lifecycle requirements and includes comprehensive cross-platform coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds robust OpenVMM process identity tracking to prevent stale or recycled PIDs from wedging managed sandbox operations.

Changes:

• Records and validates process start time across Linux and Windows.
• Treats exited and zombie processes as stopped while preserving legacy-record compatibility.
• Adds lifecycle tests and user documentation.

File	Description
scripts/​nvx_tools/​sandbox_lifecycle.py	Implements PID/start-time identity checks.
scripts/​test_nvx_tools.py	Covers reuse, zombies, malformed records, and startup failures.
doc/​run.md	Documents process identity and legacy behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.