Repository navigation
nvx-rs: host cold integration for image-backed sandboxes #622
Description
Parent: #535. Phase: A4. PR 1 (below) comes first. Runtime integration needs A1-host; the boot contract is agreed with A3. Delivers: #552, the host library part of #558 (with A3), #570's VMM check, and the cold block part of #548.
Goal. One composable cold-launch implementation for NVX's CLI and external orchestrators. Planning, attachment preparation and supervision are shared; authorization, image resolution, application transport and product policy stay with the adapter.
Today (dev ff95973):
host::sandboxplans one EROFSdistro(nvx_layer=distro,MMIO,UUID) and a required ext4 scratch, and passes block paths to OpenVMM (--microvm-sandbox-block distro:file:PATH,ro).- The planner has no GPT and no RAM upper. OpenVMM d0bdbed8 already accepts a cold RAM-upper layout with one read-only
distro. - The CLI refuses overlapping host paths among a sandbox's own shares (
host/sandbox/mount.rs:742-804), but not shares that reach its state directory, scratch, layers or bundle. host::filesystem::planrecords the identities of objects inside read-write mappings and rechecks them at start. A read-only path inside a read-write mapping is refused only for a Windows file; elsewhere only the guest's bind enforces it (nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552).- The public network policy has no proxy-only mode.
- OpenVMM accepts a command line of up to 64 KiB, but the guest kernel reads 2,048 bytes, NUL included (microVM: reconcile the 64-KiB command-line limit with the guest kernel's limit #570).
Scope:
- Typed plan and prepared launch:
- A typed, persistable cold plan: block format and upper, guest boot inputs and their expected activation-configuration identity, one export plan with ordered mapping actions, network and proxy policy, and launch settings.
- A separate prepared launch that owns the checked files, the handoff guards, the descriptor allowlist and the rendered arguments.
- Adapters keep their own boot console, CPU profile, guest control and identity choices within A3's rules. A root profile is not encoded as a zero workload identity for the VMM.
- Images and uppers:
- One read-only
distro, EROFS or GPT, from explicitly raw backing. An adapter whose images use a container format (for example a fixed VHD) stages a raw file; the runtime never converts formats or reads footers. - The boot token names the format:
nvx_layer=distro,MMIO,erofs[:UUID]ornvx_layer=distro,MMIO,gpt[:DISK-GUID]. Partition GUIDs have no boot transport. Malformed, duplicate, conflicting and old forms are refused, with no probing fallback. - The upper is a whole-device ext4 scratch, or A3's tmpfs: exactly one
nvx_overlay_upper=ramfstoken and no scratch block or token.
- One read-only
- Checked handoff:
- distro: the A4 PR 1 seal, checked by metadata only. On Linux, the checked descriptor is inherited and named
/proc/self/fd/N. On Windows, a handle denying write and delete is held until authenticated Ready, or until a failed launch is torn down. - scratch: a separate mutable binding (identity, kind, size and exclusive ownership by one sandbox), since the VMM writes it. On Windows its guard shares read and write and denies delete.
- Only allowlisted descriptors pass (A1-host). Descriptor paths are never persisted.
- distro: the A4 PR 1 seal, checked by metadata only. On Linux, the checked descriptor is inherited and named
- Mappings and network:
- One planner lowers exports, mapping tables and network policy, including proxy-only networking, for the CLI and external orchestrators.
- Today's identity record at provision and recheck at start stay; opened-object receipts are meta B.
planrefuses a read-only path inside a read-write mapping on every host (nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552). A read-write path inside a read-only mapping stays valid.- Managed launch order, through NVXC and through the test adapter's transport alike: authorize and install policy, boot, authenticated Ready, ordered MAPS, then Start completes.
- Runtime identity:
- Reattach needs a verified process in the same host boot with the same runtime identity, which A1-host records.
- On a different identity the sandbox is stopped by identity-verified forced termination, without speaking a possibly different guest protocol. If exit can't be confirmed, the record is kept and the failure reported.
- The CLI's ordinary
stopreports a timeout without forced termination.
- Command line (microVM: reconcile the 64-KiB command-line limit with the guest kernel's limit #570):
- OpenVMM's microVM profile rejects an effective command line over 2,048 bytes including NUL (2,047 bytes of text). The check runs after the VMM appends its own tokens, and restore validates saved strings without rewriting them. This is a VMM PR.
- Launchers keep their 1,024-byte caller budget, which refuses early; the VMM check is authoritative.
- CLI:
provisionrecords the PR 1 binding andstartchecks it.
GPT images, the RAM upper and protected paths in nvx-rs sandbox, and the test adapter's end-to-end runs, moved to #816 (#818, #822).
bundled keeps the old guest until #624 re-pins it.
Not in scope: guest root assembly (A3), exec v2 (#821), restore and opened-object receipts (meta B), flat ext4, image slots, image management, and protected roots chosen by the library.
Acceptance:
- The test adapter boots all four format and upper combinations on KVM, MSHV and WHP. Tests assert the selected filesystem and upper, the expected identities, the block size, and no scratch in RAM mode. The CLI's combinations moved to nvx-rs: GPT images, the RAM upper and protected paths in
nvx-rs sandbox#818. - Fault injection replaces the image between the check and the VMM's open: the checked object is used, or the launch fails. This covers writable scratch, Windows sharing, the Linux allowlist and unrelated-descriptor leaks. A Windows launcher that ends before the VMM opens the images is nvx-rs: keep a Windows launch's checked images guarded when its launcher ends before OpenVMM opens them #780.
- A staged launch reads no artifact content; verification reads are measured separately from VMM and guest disk I/O.
- Tests cover zero, ordered and failed MAPS, mapping identity changes, network allow and deny, proxy-only networking, and failed-start teardown.
- Tests cover a same-identity restart, a changed identity, an interrupted launch, an earlier host boot, PID reuse and an indeterminate identity.
- nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552 is refused on Linux and Windows, and the inverse nesting stays allowed.
- Tests cover command-line boundaries after the VMM's tokens, duplicate tokens, wrong formats or identities, and conflicting upper settings.
- Unchanged cases keep their goldens. Intended changes are limited to the format token, nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552 and the records.
- Consumer-contract scenarios C3, C4 and C5 pass, and C7's host part.
PR stack:
- PR 1: verification split and staged binding.
- Typed plan, blocks and uppers, and the boot contract agreed with A3.
- Checked handoff on Linux and Windows (needs A1-host's allowlist).
- Shared export composition and nvx-rs: enforce read-only mappings nested in read-write mappings on the host #552.
- Shared network and proxy composition.
- Runtime-identity reattach.
- GPT bindings and the adapter plan, with registered images in
aci_edge_sandboxes(sandbox: bind GPT disks, plan adapter launches, and stage registered images #813).
The CLI's integration and the test adapter end to end (formerly PRs 7 and 8) moved to #818 and #822.
In parallel, a VMM PR in nanvix/openvmm adds the command-line limit. It moves to microsoft/openvmm feature/microvm/dev before the NVX PR that updates the gitlink.
Carries over: #538 (host planning and lifecycle), the host part of #541, the cold block part of #548, #552, #558 (with A3), and #570.
PR 1: separate distro verification from launch planning
Parent: #535. This is the first PR of A4; A4 hands the checked file to the VMM.
Goal. Shared provision, start, exec and restore paths never hash large content. Explicit verification stays available at build, install, staging and library load, and for opt-in diagnostics. NVX's CLI keeps its preflight: run and provision verify the distro when a manifest sits beside it.
Today:
- The CLI calls
validate_sandbox_systemd_policyonrun, dry runs included, and onprovision. - It rejects the systemd entrypoints first. With a manifest present, it checks the manifest's format, hashes the whole layer, compares the digest, then rejects a
systemdpackage. - A persisted
startdoesn't call it, and the planners read no layer content. - The digest binds the manifest to the layer, but doesn't authenticate the manifest.
Scope:
- Split three checks apart: the cheap entrypoint check, explicit image and manifest verification, and a metadata-only check of a staged binding.
- A versioned file seal: platform scheme, volume or device, file ID, kind, length, modification time and, on Linux, change time.
- A distro binding that ties the image seal to:
- the manifest's seal, or its explicit absence;
- the accepted policy revision and result;
- the staged format and expected identities.
- A binding comes from explicit verification, or is imported from an authorized staging caller without hashing again. Verification compares metadata before and after through the opened files.
checkcompares the opened image and manifest with the binding by metadata only, and returns a checked file that A4 hands to the VMM.- A missing or stale binding fails. No runtime operation registers, verifies or rehashes on its own.
- Staged files live in a directory only the orchestrator's account can write, are read-only, and are never rewritten in place. A seal detects accidental change, not deliberate tampering by that account.
- A layer without a manifest passes; launch still checks the entrypoint, without rereading the package list.
- The CLI's
provisionrecords the binding andstartchecks it. Diagnostics stay separate operations.
Acceptance:
- CLI outcomes, error order and goldens are unchanged; a digest mismatch is still reported before package errors.
- With a manifest present, and with open and stat allowed but every content read or mapping failing, the staged check succeeds. The CLI's verification is the positive control and demonstrably reads content.
- A change of identity, kind, length or timestamps, an added, removed or replaced manifest, a new policy revision or a rejected result each fail without hashing.
- A missing manifest doesn't skip the entrypoint check.
- Ownership of the checked file is tested here; later A4 PRs extend the replacement tests through the VMM's open.
Meta A: #535. Phases: A1-guest #619, A1-host #620, A3 #621, A4 #622, A5 #623. NVX's binaries and tools, and the final acceptance: #816.
Activity
nvx-rs: record the distro binding at provision and check it at start
microVM: derive the command-line limit tests from the shared check
openvmm: refuse kernel command lines that the guest kernel would trun…
microVM: check the effective command-line test against the profile to…
openvmm: refuse kernel command lines that the guest kernel would trun…
microVM: build the command-line limit tests' expectations from the pr…
openvmm: refuse kernel command lines that the guest kernel would trun…
openvmm: refuse kernel command lines that the guest kernel would trun…
microVM: build the discovery limit test's expected line from its parts
openvmm: refuse kernel command lines that the guest kernel would trun…
microVM: derive the limit error and the tests' padding from the limit
openvmm: refuse kernel command lines that the guest kernel would trun…
nvx-rs: separate distro verification from the launch check
125 remaining items
filesystem: check the write paths and pins of recorded plans
filesystem: compare nested targets by name and reserve the whole root
filesystem: refuse recorded host paths that are not resolved
sandbox: check a one-shot run's shares again for overlap
filesystem: refuse host paths that only a verbatim path names
guest: map host paths beneath a managed sandbox's root
sandbox: deliver managed live shares as a host mapping table
guest: map host paths beneath a managed sandbox's root
sandbox: deliver managed live shares as a host mapping table
guest: map host paths beneath a managed sandbox's root
sandbox: deliver managed live shares as a host mapping table
esaurez commentedon Oct 23, 2026
PRs 1–6 merged (#653, #676, #752, #765 with #794, #753, #751), and #813 added GPT bindings and the adapter plan. The CLI's integration and the test adapter end to end moved to #816; #780 tracks a Windows gap in the checked handoff.
nvx-rs: bind GPT disks as distro images
nvx-rs: plan launches for an adapter's own guest