test: close end-to-end testing gaps for nvx.py sandbox

.github/workflows/run-nvx-microvm-tests.yml

@@ -109,22 +109,24 @@ jobs:
           ${{ inputs.debug-kernel && '--debug-kernel' || '' }}
           --output-dir "build/test-results/microvm-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}"
 
-      - name: Run public managed execution configuration on Linux
+      - name: Run public managed sandbox acceptance on Linux
         if: runner.os != 'Windows' && !inputs.debug-kernel
         shell: bash
         run: >-
           python3 scripts/nvx.py test-microvm
           --backend "${{ inputs.backend }}"
           --scenario managed-exec-config
+          --scenario sandbox-lifecycle
           --output-dir "build/test-results/public-exec-${{ inputs.backend }}"
 
-      - name: Run public managed execution configuration on Windows
+      - name: Run public managed sandbox acceptance on Windows
         if: runner.os == 'Windows' && !inputs.debug-kernel
         shell: powershell
         run: >-
           python scripts\nvx.py test-microvm
           --backend "${{ inputs.backend }}"
           --scenario managed-exec-config
+          --scenario sandbox-lifecycle
           --output-dir "build/test-results/public-exec-${{ inputs.backend }}"
 
       - name: Run Ubuntu guest smoke tests on Linux
@@ -217,12 +219,19 @@ jobs:
           )
           scratch="build/ubuntu-smoke-${{ inputs.backend }}.ext4"
           cp --reflink=auto build/ubuntu-smoke-scratch.ext4 "${scratch}"
+          # The probe checks the security profile and that the 32 MiB memory
+          # and 8-process limits hold.
           python3 scripts/nvx.py sandbox \
             --hypervisor "${{ inputs.backend }}" \
             --layer "distro,build/ubuntu-distro.erofs,${uuid}" \
             --scratch "${scratch}" \
             --entrypoint /sbin/nvx-sandbox-smoke \
+            --arg limits \
+            --arg 33554432 \
+            --arg 8 \
             --workload-user 65534:65534 \
+            --memory-max 33554432 \
+            --pids-max 8 \
             --memory-mib 256
 
       - name: Run Ubuntu sandbox live-share smoke test on Linux
@@ -687,12 +696,19 @@ jobs:
               ConvertFrom-Json
           $scratch = "build\ubuntu-smoke-${{ inputs.backend }}.ext4"
           Copy-Item build\ubuntu-smoke-scratch.ext4 $scratch
+          # The probe checks the security profile and that the 32 MiB memory
+          # and 8-process limits hold.
           python scripts\nvx.py sandbox `
             --hypervisor "${{ inputs.backend }}" `
             --layer "distro,build\ubuntu-distro.erofs,$($manifest.uuid)" `
             --scratch $scratch `
             --entrypoint /sbin/nvx-sandbox-smoke `
+            --arg limits `
+            --arg 33554432 `
+            --arg 8 `
             --workload-user 65534:65534 `
+            --memory-max 33554432 `
+            --pids-max 8 `
             --memory-mib 256
 
       - name: Run Ubuntu sandbox live-share smoke test on Windows

doc/ci.md

@@ -221,7 +221,26 @@ descriptor modules, converter implementation, and Dockerfile. Artifact upload
 retains the Alpine filenames and adds the distinct Ubuntu filenames. Each
 backend also boots the Ubuntu initramfs and runs
 `/sbin/nvx-sandbox-smoke` from the Ubuntu EROFS layer as UID/GID 65534 over a
-fresh ext4 scratch copy. The same entrypoint then verifies a live virtio-fs
+fresh ext4 scratch copy, with a 32 MiB `--memory-max` and an 8-process
+`--pids-max`. The probe verifies the
+[sandbox security profile](run.md#experimental-single-workload-sandbox): the
+fixed identity, empty capability sets and `no_new_privs`, private mount, PID,
+and UTS namespaces, the private device view and read-only sysfs, the
+container cgroup, the overlay root over read-only image layers with its upper
+directory on scratch, and that the memory controller kills an allocation of
+twice the limit and the pids controller stops the workload at eight
+processes. The live-share runs below repeat the profile checks. Each
+backend also runs the public `managed-exec-config` and `sandbox-lifecycle`
+scenarios of `test-microvm`, which drive `nvx.py sandbox provision`, `start`,
+`exec`, `stop`, and `deprovision` as subprocesses (see
+[Run](run.md#managed-lifecycle)): request-scoped working directories and
+environments, timeouts, nonzero exits, stdout and stderr, bounded output and
+outcome reports, warm-guest state across requests and restarts, identity
+refusal, fail-closed lifecycle transitions, a crashed OpenVMM's stale runtime
+state, and teardown that leaves no OpenVMM process, control endpoint,
+capability, or runtime record while the layer and scratch artifacts survive.
+Their bounded evidence is part of the backend's failure logs. The same
+entrypoint then verifies a live virtio-fs
 share inside the container: a read-write `/workspace` share with a denied
 subdirectory must round-trip guest writes to the host, and a read-only
 `/opt/hostedtoolcache` share must reject writes and symbolic links. In the

doc/contribute.md

@@ -22,3 +22,7 @@ guest helpers, SMP behavior, or virtio devices. Add those scenarios to
 `scripts/nvx_tools/microvm_tests.py` and run them with
 `scripts/nvx.py test-microvm --backend BACKEND`. Keep guest workloads in
 `scripts/nvx_tools/microvm_test_scripts` and retain complete failure logs.
+The scenarios that drive the public `nvx.py sandbox` commands live in
+`scripts/nvx_tools/managed_exec_tests.py` and
+`scripts/nvx_tools/sandbox_lifecycle_tests.py`, and `microvm_tests.py`
+registers them.

doc/design/copilot-adversarial-testing.md

@@ -85,6 +85,8 @@ The initial workload catalog exercises the same managed control and isolation
 mechanisms through `test-microvm`; it does not yet drive the public persistent
 `sandbox provision -> start -> exec -> stop` command sequence. That lifecycle
 remains required before the workload-adversary mode is considered complete.
+The deterministic `sandbox-lifecycle` scenario already drives that sequence,
+so it is the regression baseline for such a primitive.
 
 Model-generated scripts are intentionally unsupported. A future generator
 must place its output inside the disposable guest or a separate disposable

doc/design/validation.md

@@ -39,7 +39,15 @@ localhost-to-guest forwarding under allow, and pre-boot rejection when both
 directions cannot be enforced. Managed lifecycle coverage authenticates the
 dedicated control channel, runs multiple workloads in one warm VM across
 reconnects, preserves guest state, reports execution timeout, and stops the VM
-cleanly. Sandbox coverage adds deterministic active block-I/O drain, paired
+cleanly. Public sandbox coverage drives `nvx.py sandbox` itself: request-scoped
+execution options, identity refusal, fail-closed lifecycle transitions,
+scratch-backed state across requests and restarts, bounded output and outcome
+reports, a crashed OpenVMM's stale runtime state, and teardown that leaves no
+OpenVMM process, control endpoint, capability, or runtime record while the
+supplied layer and scratch survive. Its guest probe verifies the sandbox's
+identity, capability, `no_new_privs`, namespace, device, sysfs, cgroup,
+resource-limit, image-layer, and scratch invariants. Sandbox coverage adds
+deterministic active block-I/O drain, paired
 scratch publication, two private restores, fresh scratch replacement, and
 pre-entry rejection of missing, corrupt, mismatched, or wrong-geometry media.
 Denied-filesystem coverage verifies listing suppression, allowed writes,

doc/run.md

@@ -411,9 +411,36 @@ python3 scripts/nvx.py sandbox \
   --memory-mib 256
 ```
 
-CI uses `/sbin/nvx-sandbox-smoke` as the entrypoint to verify Ubuntu identity,
-the fixed non-root account, and a scratch-backed `/tmp` write before clean
-guest exit. With `--arg TARGET --arg ro|rw`, it also checks a live share at
+CI uses `/sbin/nvx-sandbox-smoke` as the entrypoint to verify the security
+profile that this section describes before clean guest exit. Before any other
+check, it requires:
+
+- the fixed `65534:65534` identity as the real, effective, saved, and
+  file-system IDs, no supplementary groups, and `HOME=/nonexistent`;
+- empty inheritable, permitted, effective, bounding, and ambient capability
+  sets and `NoNewPrivs: 1`;
+- mount, PID, and UTS namespaces other than the guest's initial ones, whose
+  inode numbers Linux 6.18 fixes in its UAPI, with the workload as PID 1 of its
+  PID namespace and the `--hostname` value as its host name;
+- exactly one mount at each of `/`, `/proc`, `/sys`, `/dev`, `/dev/pts`, and
+  `/dev/shm`, none of which propagates to another namespace: an overlay root
+  over the EROFS layers whose upper directory is on scratch, `nosuid,nodev,noexec`
+  procfs, read-only sysfs, and a private `/dev` tmpfs that holds only `fd`,
+  `full`, `null`, `ptmx`, `pts`, `random`, `shm`, `stderr`, `stdin`, `stdout`,
+  `tty`, `urandom`, and `zero`, with its own devpts instance;
+- the agent-owned `/container` cgroup; and
+- Ubuntu image files that the workload cannot modify, and a scratch-backed
+  `/tmp` write that lands in the overlay.
+
+A managed workload that mounted into the agent's namespace would stack its
+runtime mounts on those of earlier requests, so the single-mount check also
+covers repeated managed execs. With `--arg limits --arg MEMORY_MAX --arg
+PIDS_MAX`, for a sandbox started with the same `--memory-max` and
+`--pids-max`, it also requires the memory controller to kill an allocation of
+twice `MEMORY_MAX` with `SIGKILL` while a 1 MiB allocation succeeds, and the
+pids controller to stop the workload at `PIDS_MAX` processes, itself included;
+keep twice `MEMORY_MAX` within the guest's free memory. With
+`--arg TARGET --arg ro|rw`, it also checks a live share at
 `TARGET` as described below, including symbolic links in an `rw` share; the
 share needs a host-created, world-writable `nvx-links` directory for them.
 Repeat the pair to check several shares; with an `rw` and an `ro` share, it
@@ -692,7 +719,10 @@ request to retain VM state.
 Environment files are limited to 1 MiB of UTF-8 JSON. Environments contain at
 most 256 unique, nonempty names. Each `KEY=VALUE` entry and working-directory
 path is limited to 4096 UTF-8 bytes; the combined execution request must also
-fit the existing 64 KiB control-payload bound.
+fit the existing 64 KiB control-payload bound. A managed `exec` forwards at
+most 1 MiB of combined standard output and standard error; the guest agent
+kills a workload that writes more, and `exec` exits with status 125 and reports
+the `output-limit` category.
 
 The explicit `test-microvm --scenario managed-exec-config --backend BACKEND`
 scenario is the authoritative acceptance for these public options. It invokes
@@ -718,6 +748,56 @@ synthesize its own variables. The scenario retains bounded subprocess argument
 and status observations, typed exec outcomes, and OpenVMM logs. Inline
 environment values are redacted from the retained command observations.
 
+The explicit `test-microvm --scenario sandbox-lifecycle --backend BACKEND`
+scenario is the authoritative acceptance for the managed lifecycle itself. It
+also runs the public commands as subprocesses against the real kernel, Alpine
+control initramfs, Ubuntu EROFS layer, and fresh copies of the same scratch
+template, so it has the same artifact requirements and also runs only when
+named. It checks that:
+
+- `provision` rejects a root `--workload-user` before it creates any state, and
+  the guest refuses an identity that the Ubuntu image lacks before a one-shot
+  workload starts, and in a managed `start`, which then fails without leaving
+  a runtime record, capability, control socket, or OpenVMM process;
+- every operation fails on a state directory that was never provisioned, a
+  repeated `provision` leaves the configuration unchanged, and `exec` and
+  `stop` fail before `start`;
+- a repeated `start` and a `deprovision` of a running sandbox fail without
+  changing its runtime record, capability, or OpenVMM process, and the
+  sandbox keeps serving requests;
+- managed arguments keep leading, trailing, and embedded spaces, tabs, and
+  newlines, a later request reads the file that an earlier one wrote, and the
+  file survives `stop` and a new `start`, because the overlay's upper directory
+  is on scratch;
+- `/sbin/nvx-sandbox-smoke` passes its security profile and resource-limit
+  checks in a sandbox provisioned with `--hostname`, `--memory-max`, and
+  `--pids-max`, after earlier requests;
+- a request that writes more than the 1 MiB output bound exits with status
+  125 and an `output-limit` outcome report after the host has received more
+  than 1 MiB less one 32 KiB read chunk and at most 1 MiB, and a request whose
+  `--outcome-report` path exists fails before its workload runs and leaves the
+  file intact;
+- `stop` ends OpenVMM and its control socket or pipe, leaves only
+  `config.json`, `openvmm.log`, and a successful `outcome.json`, and leaves a
+  cleanly unmounted scratch file system, after which `exec` and `stop` fail;
+- after OpenVMM exits without a `stop`, `exec` and `stop` report the stale
+  runtime state and `start` refuses it; and
+- `deprovision` removes only NVX's files: it fails and keeps a foreign file in
+  the state directory, and once that file is gone, it removes the directory,
+  which a later `exec` does not recreate.
+
+Throughout, no OpenVMM process whose arguments name the scenario's fixture
+outlives its sandbox, and the EROFS layer keeps its digest and the scratch
+image its file. On failure, the scenario stops and deprovisions what it
+started, ends every OpenVMM process that names its fixture, and preserves the
+fixture for recovery only if that cleanup fails. Its evidence, written to the
+output directory with a `sandbox-lifecycle-` prefix, holds bounded command
+observations (arguments, statuses, output sizes, and state-directory entries),
+the runtime record, the OpenVMM logs, the VM-level and exec outcome reports, the
+guest probe's transcript, and the console of the refused one-shot run, each
+limited to its last 64 KiB. The command observations hold no workload output,
+and no evidence holds the control capability.
+
 Decoder, helper, and direct control-session tests remain useful supplemental
 coverage for protocol boundaries and guest implementation details. They do not
 replace or establish support through the public `nvx.py sandbox` interface.

doc/usage.md

@@ -275,12 +275,16 @@ python3 scripts/nvx.py test-microvm
 
 Runs NVX-owned Linux, SMP, virtio, sandbox, and snapshot correctness scenarios
 against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without
-it, every scenario supported by the selected guest runs, except `smp-lapic`,
-which runs only when named: it repeats `smp` and also asserts that every CPU
-uses the one-shot counting LAPIC. Alpine remains the
+it, every scenario supported by the selected guest runs, except three that run
+only when named. `smp-lapic` repeats `smp` and also asserts that every CPU
+uses the one-shot counting LAPIC. `managed-exec-config` and
+`sandbox-lifecycle` drive the public `sandbox` lifecycle commands (see
+[Run](run.md#managed-lifecycle)) and also require the guest-artifact build's
+Ubuntu layer and scratch template, which packages omit. Alpine remains the
 default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject
 the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and
-the sandbox-control-dependent `snapshot-tiers` scenario. Ubuntu also rejects
+the sandbox-control-dependent `snapshot-tiers`, `managed-exec-config`, and
+`sandbox-lifecycle` scenarios. Ubuntu also rejects
 the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots
 the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task
 detectors the guest's time ABI watcher reports; without `--scenario`, it runs