Track nvx.py managed sandbox bugs: PID-only liveness check wedges stop, deprovision, and start #369

Summary

A focused bug hunt on NVX's host-side Python tooling and guest helpers found one reproducible bug, in the managed nvx.py sandbox lifecycle. Line references are permalinks to dev at 2508acd. No repository files were changed; the reproductions ran against scratch copies of the repository. #364 tracks the separate findings for the aci_edge_sandboxes crate.

• 1. Identify OpenVMM by PID and start time, and count zombies as exited

1. A recycled or unreaped OpenVMM PID wedges stop, deprovision, and start (Medium)

Where

• sandbox start records only OpenVMM's PID in runtime.json (L466-L473).
• _process_running L272-L298 counts any process with that PID as OpenVMM. On Linux, os.kill(pid, 0) succeeds for a zombie, and PermissionError, which means another user's process holds the PID, also counts as running (L275-L282). On Windows, any process that the caller can open and that has not exited counts (L284-L298).
• Every lifecycle operation relies on that check:
  • exec and stop load the record through _load_running L301-L319.
  • After the guest acknowledges STOP, stop waits for the PID to disappear (L526-L529). If it times out, it raises before it removes the runtime record (L530-L535).
  • deprovision refuses while the PID appears to run (L541-L551).
  • start refuses whenever runtime.json exists (L378-L379).
• The aci_edge_sandboxes backend avoids this. It records the process start time, pins the process with a pidfd or handle before it acts on it, and counts zombies as exited (platform/linux.rs L22-L23, L52-L55; platform/windows.rs L121-L144).

Trigger: OpenVMM is gone, but its recorded PID still names a live process:

• Windows: OpenVMM exits without sandbox stop, because it crashes, is killed, or the host restarts with a persistent --state-dir. Windows then gives its PID to another of the user's processes. In these runs, that took 3,000 to 20,000 process creations (15 to 120 s).
• Linux: the host restarts with a persistent --state-dir, so the recorded PID belongs to whatever process received it at boot. A root-owned daemon raises PermissionError, which counts as running. PID wraparound has the same effect.
• Linux: no process reaps OpenVMM after it exits. sandbox start exits right after it launches OpenVMM, so OpenVMM is reparented to PID 1 or the nearest subreaper. In a container whose PID 1 does not reap orphans, such as docker run without --init, an exited OpenVMM stays a zombie, even after a clean sandbox stop.

Expected: the operations recognize that OpenVMM is gone, as they already do while its PID is unused. stop reports "sandbox runtime state is stale because the OpenVMM process is not running", and deprovision removes the sandbox. After the guest acknowledges STOP and OpenVMM exits, stop removes the runtime record.

Actual: the sandbox is wedged, and no CLI operation recovers it:

• stop waits --timeout for the dead control endpoint, then fails with "managed control endpoint did not become available".
• deprovision refuses: "sandbox must be stopped before deprovision".
• start refuses: "sandbox is already running or has stale runtime state".
• With an unreaped OpenVMM, even an acknowledged STOP ends with "OpenVMM did not terminate after managed stop", and runtime.json stays.

The wedge lasts until the unrelated process exits or someone deletes runtime.json by hand. No operation signals the recorded PID, so the unrelated process itself is unaffected.

Reproduction: the scripts drive the real scripts/nvx.py sandbox CLI against a provisioned sandbox. A short-lived stand-in process plays OpenVMM, and the scripts write the runtime.json and control capability that sandbox start writes.

Windows, at 2508acd86. The script starts unrelated processes until one receives the stand-in's PID:

OpenVMM stand-in PID 12160 exited; _process_running=False
Control (PID not reused yet):
  sandbox stop        -> exit 1 after  0.3s: error: sandbox runtime state is stale because the OpenVMM process is not running
After 19976 unrelated process creations (120s), PID 12160 belongs to an unrelated ping/cmd process; _process_running=True
Lifecycle with the recycled PID:
  sandbox stop        -> exit 1 after  3.3s: error: managed control endpoint did not become available: \\.\pipe\openvmm-microvm-e4b17c87751e4b5a897604a7a586a975
  sandbox deprovision -> exit 1 after  0.3s: error: sandbox must be stopped before deprovision
  sandbox start       -> exit 1 after  0.3s: error: sandbox is already running or has stale runtime state
  runtime.json still present: True
After the unrelated process exits:
  sandbox deprovision -> exit 0 after  0.2s:

Linux (WSL2), at 2508acd86, as uid 1000. The record names PID 1, a root-owned process, as it might after a reboot:

uid 1000; PID 1 belongs to uid 0; _process_running(1)=True
  sandbox stop        -> exit 1: error: managed control endpoint did not become available: /tmp/nvx-reused-pid-ov0hxjai/state/control.sock
  sandbox deprovision -> exit 1: error: sandbox must be stopped before deprovision
  sandbox start       -> exit 1: error: sandbox is already running or has stale runtime state

Linux (WSL2), at 2508acd86, with an unreaped OpenVMM. A launcher starts the stand-in in a new session and exits, as sandbox start does. The script runs under a subreaper that never reaps, as a container PID 1 without --init behaves. Because a real STOP needs a guest, the last step calls sandbox_lifecycle.stop() with a control-session stub that acknowledges STOP:

OpenVMM stand-in 451 has exited (state Z); _process_running=True
  sandbox deprovision -> exit 1: error: sandbox must be stopped before deprovision
  sandbox start       -> exit 1: error: sandbox is already running or has stale runtime state
  stop() after STOPPED   -> TimeoutError after 2.0s: OpenVMM did not terminate after managed stop
  runtime.json still present: True

Suggested fix: give the runtime record a process identity, as the crate's ProcessIdentity does.

• At start, store OpenVMM's start time next to its PID: field 22 (starttime) of /proc/<pid>/stat on Linux, or the creation time from GetProcessTimes on Windows.
• In _process_running, count the process as running only while its start time still matches. On Linux, also count a zombie (state Z or X) as exited.
• Keep the PID-only check for records written by NVX versions that do not store the start time.
• Add unit tests for a zombie child, a start-time mismatch, and stop after an acknowledged STOP while OpenVMM is a zombie.

What was explored

• The repository guidance, the design chapters on validation and on concurrency and trust boundaries, and the CI workflows and actions, including the check-required-ci expectations and the performance prepare, gate, and persist actions.
• Guest code:
  • nvx-managed-agent.c: control framing, EXEC decoding with the per-exec working directory and environment from sandbox: support managed execution environments, cwd and longer timeouts #288, the exec loop, cgroup containment, and host mappings.
  • init, nvx-init-agent, nvx-container-launch, nvx-container-enter, and nvx-snapshot.
  • nvx-port-io.c, nvx-reseed.c, nvx-mmio-write.c, and nvx-console-pending.c.
• Host Python:
  • The managed sandbox: sandbox_lifecycle.py, sandbox.py, control_session.py, the nvx.py sandbox option handling, and managed_exec_tests.py.
  • Tests and processes: scenario dispatch in microvm_tests.py, openvmm_process.py, and the process containment in the adversarial executor and oracles.
  • Performance and release: performance.py, the download path in release.py, development_release.py, archive.py, and the download and checksum helpers in common.py.
  • Other tooling: safe tar extraction in ubuntu.py, ci.py, .github/specula/ci.py, and the host-profile resolver of the nvx-host-connect skill.
• Cross-checks:
  • The EXEC encoding in control_session.py against the agent's decoder. The layout, flags, and limits match: 64 arguments, 256 environment entries, 4096-byte entries and working directory, and a 64 KiB payload.
  • The options, choices, and defaults in doc/usage.md against the argparse definitions. No documented option is missing, and no choice list differs.
  • --mount and --microvm-sandbox-block in sandbox.py against OpenVMM's parsers at the pinned revision.
• Checked but not filed:
  • performance gate raises IndexError instead of PerformanceError when a target CSV has a header but no rows. performance collect never writes such a file, and the gate still fails closed.
  • Duplicate rows already in a baseline CSV each count toward --minimum-history. performance persist deduplicates rows, so only a hand edit can introduce duplicates.
  • download reads only the newest 100 releases (per_page=100, without pagination). It fails only if none of them is a published release with the platform's package.
• Ruled out: download sends the GitHub token as a bearer token and drops it on cross-origin redirects.

Validation and limitations

• Baseline: the python -m unittest suites from validate-nvx (test_performance, test_nvx_tools, test_microvm_tests, test_development_release, test_egress_policy, test_control_session, and test_managed_agent) pass on Windows at acd647200: 529 tests, 28 skipped. Under WSL2, the Linux-only test_managed_agent and test_control_session suites pass (22 tests).
• No VM ran on KVM, MSHV, or WHP. The reproductions replace OpenVMM with a stand-in process and exercise the real CLI and sandbox_lifecycle code. A record that names PID 1 simulates the reboot case; no host was rebooted. A non-reaping subreaper stands in for a container.
• The Windows and zombie reproductions ran at both acd647200 and 2508acd86, and the root-owned PID reproduction ran at 2508acd86. Between those commits, the lifecycle code changed only by a new exec_workload parameter.