[Enrique Saurez (esaurez)]

Stacked on #884; this PR retargets when its parent merges.

Chain: #932 → #884 → this PR → #933, on dev; #879, #881, #885, #886, and #877 have merged.

Part of #841 (PR 2 of 3: capture and publication).

Updates, as fast-forward commits on aebe922:

1. Each capture now launches its VM from its transaction, the VM's own sandbox store, rather than from its slot. At the pinned OpenVMM, a VM that captures a snapshot must have its control socket, on Linux, in the directory that holds its --snapshot-destination, and refuses to start otherwise; the captures of nvx-rs: lower, capture and publish platform templates #841 PR 3, which run VMs through the runtime adapter, found that on Linux. The slot keeps the VMM's log and outcome report of its class's latest capture, the fake OpenVMM now enforces the rule, and the tests check it.
2. The fake checks every control console on Unix as OpenVMM does: the socket's directory must be a directory, not a link, of the user with exactly the mode 0700, and, if the VM restores or captures a snapshot, the canonical directory that holds it, the one of --restore-snapshot first, else of --snapshot-destination.
3. The capture tells its block devices apart by the identity that their seals record, FileSeal's device and file ID, as clones do in nvx-rs: plan the clones of templates, and check their block geometry #888. On Windows that is the volume and the 128-bit file ID of FILE_ID_INFO, where it was the volume serial number and the 64-bit file index of BY_HANDLE_FILE_INFORMATION, which Microsoft doesn't guarantee unique on ReFS, the file system of Dev Drives: there, another file at the placeholder's or the scratch's path could have passed as the one that the capture checked. Linux keeps st_dev and st_ino.

What and why

create and refresh capture a tenant-free template through the supervisor, and publish it into the template store as a new generation.

• Store::create_template(&Capture, now: impl Fn() -> Result<HostTime>, deadline) -> Result<Created> serves the capture class with a generation that the store holds, Created::Existing: the latest that the index lists of the class, of the key of the class's latest generation, that no tombstone quarantines, and that is eligible at the host's time, as Store::select chooses among a key's generations by Generation::eligibility. Without one, it captures one, Created::Captured. It calls now once, after it holds the class's slot and has recovered it and the store's index, so a generation that a racing creation captured while it waited counts, and creations that race for a class make one generation between them. Callers pass HostTime::now, or a closure that adds a margin.
• Store::refresh_template(&Capture, deadline) -> Result<Generation> always captures a new generation, which later selections of its key take first. Older generations and their leases are untouched.
• Store::recover_captures(deadline) -> Result<Vec<Generation>> ends the captures that a crash interrupted, as an adapter does once as it starts.

The issue calls the first two Store::create and Store::refresh; #881's Store::create(dir) already creates the store itself, so they take a _template suffix. Capture holds a capture's inputs: its CaptureClass, the adapter's SandboxPlan, the placeholder's DistroBinding, the distro policy revision that the caller accepts, and the runtime's Artifacts.

Serialization and the transaction

Each class has a slot, captures/<the first 16 hex digits of its ClassIdentity>/: a sandbox store whose lock serializes the class's captures, and which keeps the VMM's log and outcome report of the class's latest capture. A capture works in a private transaction directory in its slot, on the store's volume, which is also the sandbox store of its VM, whose launch journal records the VMM: at the pinned OpenVMM, a VM that captures a snapshot must have its control socket, on Linux, in the directory that holds its --snapshot-destination, and a sandbox store's VM listens in the store's directory. The capture holds the transaction's lock, under its slot's, until the VM has ended, and releases it before it publishes. The capture writes capture.json (the store, the class, and the storage generation), creates the dormant scratch there as a sparse file, lowers the adapter's plan into a TemplatePlan (#884) with a fresh StorageGeneration, and cold-launches the template's VM through supervisor::launch from the transaction, as a launch of its own (LaunchId::generate, since a transaction holds no lease). It opens no application session and waits for no Ready: the guest captures itself, and OpenVMM publishes the snapshot into the transaction and exits. A template's VM never becomes a sandbox: its launch is never committed. Once the VM has ended, the capture moves the VMM's log and outcome report to the slot.

Launching a capture: block devices by path

At the pinned OpenVMM, OpenVMM opens snapshot-capable block devices itself, by path, whenever --snapshot-destination is given: without following a link (O_NOFOLLOW), and on Windows without following a reparse point (FILE_FLAG_OPEN_REPARSE_POINT), sharing only reads and deletion. A PreparedLaunch's /proc/self/fd/N paths and inherited scratch claim therefore don't serve a capture, so, as decided for #841, the capture hands OpenVMM paths, not handles:

• Before launch, it checks the transaction, which must be private to the user; the placeholder, through its DistroBinding (identity and length), and then at its path: a regular file, not a link or other reparse point, the file that the binding's check found, owned by the user or an administrator, and that no one else may change; and the scratch at its path, the same way: the file that the capture created, of the template's length.
• It tells files apart by the identity that their FileSeals record: the device or volume, and the file ID, on Windows the 128-bit one of FILE_ID_INFO, which ReFS keeps unique where the 64-bit file index is not.
• As OpenVMM starts, the host holds no writable handle on either file: it closes the scratch, and keeps only the binding's read-only open of the placeholder, which on Windows denies others write access, until the VM has ended.
• The directories above the files must be ones that only the user or an administrator may change, since OpenVMM resolves each path again. The placeholder's path is absolute, as nvx-rs: lower adapter plans into the plans of template captures #884's TemplatePlan::new requires, so the file that the host checks is the one that OpenVMM opens from the transaction, its working directory. Store's documentation states the rule, under "Block devices by path", so that a restore of a clone (nvx-rs: restore clones from a template through Activate, Ready and Start #844) takes the same one.
• Everything else of a launch is as for a sandbox: supervisor::launch runs and supervises the process, and the devices and the NIC are rendered by SandboxPlan::openvmm_arguments, through TemplatePlan::openvmm_arguments, which is what a PreparedLaunch renders besides its handle substitution. The capture therefore builds no PreparedLaunch.

Checks after the source exits, before publication

• The verified exit: the VM's outcome report, which the capture moved to the slot once the VM had ended, and whose predecessor it removed before the VM started, records the operation capture, the category success, and the status 0.
• The snapshot: a private directory whose manifest TemplateManifest::read reads, whose artifacts are exactly manifest.bin, memory.bin, and state.bin, private regular files of the lengths that the manifest records, checked without reading their content, and whose key, TemplateKey::new, is its class's. A snapshot that is missing, or that lacks its manifest, fails with ErrorKind::Io, as a lost capture, not NotFound.

Publication in two commits

1. OpenVMM publishes the snapshot into the transaction. Once the capture has checked it, it closes the transaction's lock and removes the lock's file, the scratch, and what OpenVMM left beside the snapshot (the file that backed the RAM, staging directories), and writes key.bin, durably, which marks the transaction as checked.
2. Under the store's lock, the host renames the envelope, OpenVMM's directory with capture.json and key.bin beside it, to generations/<the next store generation> without replacing anything (renameat2 with RENAME_NOREPLACE on Linux, MoveFileExW without MOVEFILE_REPLACE_EXISTING on Windows), makes the rename durable, and commits the index with the full key, syncing the store's directory and then its parent, since Store::create syncs neither.

Only indexed generations can be acquired, and a committed payload is never rewritten. A capture that fails before its transaction is checked tears its VM down and retires the transaction once the VMM has ended; a transaction whose launch may still run stays. Recovery, by the next capture of the class or by recover_captures, ends the launch that the transaction's sandbox store records, as LockedStore::recover does, then publishes a checked transaction and retires any other, once it has moved its VMM's log and outcome report to the slot; every capture, creation or refresh, also drops the generations whose payloads a crash lost, which then take none of the store's capacity, and completes the store's interrupted publications, before it looks for a generation or counts the store's against its capacity. A creation that serves a generation that the store holds syncs the store's directory and its parent first, since the commit that listed it may have been interrupted. A payload at the index's next store generation is indexed first, once the recovery has made its rename durable as the publication would have (it syncs the payload, generations/, and the slot that the payload left), and a generation whose payload a crash lost is dropped from the index. A checked transaction is synced again before recovery publishes it, since the crash may have come before the capture synced it. recover_captures returns every generation that it published or whose publication it completed.

Lock order

A capture holds its slot's lock until its VM has ended, and takes the store's lock under it to publish. It also holds its transaction's lock, the lifecycle lock of the VM's sandbox store, until its VM has ended, and releases it before it publishes; so does a recovery, which ends the transaction's launch. The supervisor's lock order (#886) now starts with a capture slot's lock: the slot's, then the template store's, then a sandbox's or a capture transaction's, then the scratch claim. Nothing takes a slot's lock while it holds another.

Fix to #881: Store::snapshot_dir and the snapshot directories that OpenVMM creates

On Windows, OpenVMM creates the snapshot directory and each artifact with a protected access control list of their own, D:P(A;;FA;;;SY)(A;;FA;;;OW), with no entries that files inherit, and snapshot_dir refused that directory, since check_owner_only_dir requires such entries. The crate-private platform::check_snapshot_dir(path: &Path) -> io::Result<Option<String>> keeps every other rule (a plain directory, not a link or reparse point; a trusted owner; a list that is present, with only allow and deny entries, for trusted principals or the owner's aliases), and drops the inheritance rule alone: OpenVMM gives each artifact its own list, and the directory's list already lets no one else create a file in it. On Linux and other hosts it is check_owner_only_dir. Only the snapshot directory's check changes: the directories of the generations and of the payloads, and every directory that the host creates, keep check_owner_only_dir.

Other changes

• Public constants for the layout's new entries: CAPTURES_NAME (captures), TRANSACTION_NAME (transaction), and CAPTURE_RECORD_NAME (capture.json).
• platform::foreign_writer_problem, make_sparse, and rename_no_replace, crate-private, on Linux, Windows, and other hosts.
• The fake OpenVMM runs the VMs of captures: it parses a capture's command line as the host renders it, opens the block devices as OpenVMM does, without following a link and, on Windows, sharing only reads and deletion, and records in its blocks dump whether a handle could write them (on Linux, through a read lease); it backs the RAM with a sparse file beside the destination, and then publishes the snapshot of a given manifest, fails, never ends, or crashes, as fake-openvmm-capture.json next to the kernel asks. On Unix, it checks where every control console listens, as OpenVMM does: in a directory, not a link, of the user with exactly the mode 0700, which holds the snapshot that the VM restores or captures, if any.
• Docs: Store's layout and "Captures and publication", the template module, the supervisor's lock order, and nvx-rs/README.md.

Tests

With OpenVMM emulated in the test process, through hooks that fail or crash a capture at each point (src/host/template/store/capture/tests.rs):

• captures publish generations that creations then serve, and a creation captures unless its class has an eligible generation that restores take;
• refreshes leave older generations and their leases untouched;
• failures around each commit: before the check (the transaction is retired; a snapshot without its manifest or its RAM is refused as Io), after the check (the transaction outlives the failure until recovery publishes it), and after the rename (the publication completes); crashes, which recovery ends; payloads lost in a crash are dropped from the index;
• a recovery makes what it completes or publishes durable before it indexes it, and a capture makes the store's own entry durable before it reports a generation (each sync is recorded with the index at that moment); a recovery returns the generations whose publications it completed as well as its own;
• a creation completes an interrupted publication before it looks for a generation, serves no generation whose payload a crash lost, and captures in a store whose index is full of lost generations; a recovery drops lost generations before it indexes an interrupted publication, so that a full index of them does not block it; a creation makes the store durable before it serves a generation, even when no recovery changed the index;
• what OpenVMM leaves beside the snapshot is removed; a transaction stays while it records a launch; each VM launches from its transaction, with its control socket beside the snapshot's destination; the slot keeps the outcome report of the latest capture, and none of an earlier one;
• links, reparse points, and unsafe ownership at the placeholder, the scratch, and the transaction are refused before the VM starts, and no handle through which the host could write a block device is open as the VM starts;
• another file of the scratch's length, renamed over the scratch before the VM starts, is refused, and the scratch's check compares the whole of a 128-bit file ID: an ID that differs from the scratch's only above its low 64 bits, as two files' IDs on ReFS may, is refused;
• captures of one class are serialized, and creations that race make one generation between them even when both start before the capture's anchor; recovery refuses entries of captures/ that are not slots; stores too deep for a capture's control socket are refused (Linux).
• snapshot_dir accepts a snapshot directory with OpenVMM's exact descriptor, and still refuses a payload directory without inheritable entries (Windows).

Through supervisor::launch, with nvx-fake-openvmm (tests/template_capture.rs):

• a creation captures, the VMM receives the template's command line and storage generation, listens and reports in the transaction (the fake refuses, on Unix, a capture whose control socket is not in the directory that holds the destination, as OpenVMM does), opens the placeholder and the scratch at their paths, and finds no handle that could write them, and a second creation serves the generation without a VM;
• a refresh publishes a new generation and leaves the older one;
• VMs that end without a capture (a guest failure, a VMM that fails after it published, a crash) publish nothing;
• VMs that never end are torn down at the deadline;
• captures of one class are serialized;
• an adapter that crashes while its capture's VM runs leaves the VMM, which recovery ends, publishing nothing, also when the VM had published its snapshot.

The fake OpenVMM's own tests (src/testing/fake_openvmm.rs), on Unix:

• a capture whose control socket is in its slot, beside its transaction, as at aebe922, is refused with OpenVMM's message;
• so is a socket's directory of another mode than 0700, a link to a private directory, or a missing directory, and a control console that names no endpoint after listen=;
• a restore listens only in the directory of the snapshot that it restores.

Validation

No VM is needed for this change: everything below ran without one, at this head, with the OpenVMM submodule at e68871cc.

Rebase: this head is rebased onto #884's new head 77e1660, on #932's f28fdca, which turns 3e421b7, 408c06c, bd33b3d, and 0a25211 into be318c2, ea2532b, 8396bea, and 0b23113. #886, now merged, moved the helpers that the template and sandbox stores share into nvx::host::store, so the capture takes open_regular and read_at_most from nvx::host::store::regular, the store re-exports Lease and LeaseId from nvx::host::store::lease, and nvx-rs/README.md lists the template-capture tests beside the base's host-module layering test.

• Windows: in nvx-rs, with NVX_BUNDLE=skip: cargo fmt --all --check; cargo clippy --locked --all-targets --all-features -- -D warnings; cargo clippy --locked --lib -- -D warnings, also with --no-default-features; cargo clippy --locked --all-targets --no-default-features --features <feature> -- -D warnings for host, cli, testing, serde, and bundled; cargo test --locked --all-features; cargo test --locked; cargo build --locked --lib --no-default-features and cargo test --locked --lib --no-default-features, also with --features host; RUSTDOCFLAGS="-D warnings" cargo doc --locked --no-deps --all-features; cargo +1.89 check --locked --all-targets --all-features; and cargo test --locked --test host_modules. From the root, the ignored cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored. Then the commands of check-nvx-rs for tests/runtime-adapter, protocol-only and host-only, and python scripts/check_markdown_links.py.
• Linux: the same, plus --features guest, cargo check --locked --all-targets --all-features --target aarch64-apple-darwin, python3 scripts/nvx.py check-guest-binaries, and the runtime adapter's static musl guest configuration.
• At each of the four commits, on both hosts: cargo fmt --all --check; Clippy with -D warnings: --all-targets --all-features, --all-targets, --lib, and --lib --no-default-features, plus --target x86_64-unknown-linux-musl on Windows; cargo test --locked --test host_modules; and on Linux, cargo check --locked --all-targets --all-features --target aarch64-apple-darwin.
• tests/template_capture.rs passed 5 times in a row on each host.
• The fake's control-console checks: on Linux, at 8396bea, this head's parent, with be318c2's capture.rs, whose VM listened in its slot, in place of 8396bea's, all 6 tests of tests/template_capture.rs fail: the fake refuses each capture before it starts, as OpenVMM does.
• VMs: tests: capture templates in VMs through the runtime adapter #933, nvx-rs: lower, capture and publish platform templates #841 PR 3, which builds on this PR, captured templates in VMs through the runtime adapter at its head 8c127e4: its 9 template tests passed, with the adapter's other VM tests, on Windows/WHP, and on Linux/MSHV with OpenVMM's --cpu-profile host, since that development host's CPU has no built-in profile, and in CI on KVM, MSHV, and WHP.
• CI: run 40574331834, a workflow_dispatch run of CI at tests: capture templates in VMs through the runtime adapter #933's head, which holds this PR, passed: every job that a dispatch runs, among them the microVM, platform, and OpenVMM tests on KVM, MSHV, and WHP.

Left to CI: the full matrix, once this PR targets dev; a stacked PR runs none.

Follow-up

Older identity checks share the weakness on ReFS. They predate this stack, and this PR leaves them as they are:

• a sandbox store's lock, whose taker checks that the path still names the file that it locked, as does its holder before it removes an unprovisioned lock file;
• the claim of a launch's log, whose identity the launch journal records (LogClaim), and which log_claim_held checks on Windows;
• the template store's lock (nvx-rs: add the template store #881).

They compare platform::opened_file_identity and platform::file_identity, which on Windows are the volume serial number and the 64-bit file index. A follow-up can move them to FileSeal identities; for the log's claim, that changes the journal's record.

[Copilot]

🟡 Changes recommended

Index recovery, eligibility timing for queued creations, and store-parent durability remain unresolved.

3 open issues

• Recover and persist slots before generation lookup and capacity checks · New
• Select eligibility time after recovering the shared capture slot · New
• Sync the store parent directory before reporting durable capture · New

What changed in this PR

Adds template capture and publication to NVX’s Rust host library, building on template planning and storage to support future clone restores.

Changes:

• Adds create, refresh, and recovery APIs with supervised capture and two-stage publication.
• Adds platform-specific file checks, sparse-file support, and non-replacing renames.
• Extends fake OpenVMM and tests for capture failures, recovery, concurrency, and file safety.

File	Description
nvx-rs/​tests/​template_capture.rs	Adds end-to-end capture and recovery tests with fake OpenVMM.
nvx-rs/​src/​testing/​fake_openvmm.rs	Emulates capture launches, snapshots, and failure scenarios.
nvx-rs/​src/​host/​template/​store/​tests.rs	Tests Windows snapshot directory permissions.
nvx-rs/​src/​host/​template/​store/​capture/​tests.rs	Tests publication, recovery, serialization, and file checks.
nvx-rs/​src/​host/​template/​store/​capture.rs	Implements capture, publication, and recovery APIs.
nvx-rs/​src/​host/​template/​store.rs	Exports capture APIs and adjusts snapshot directory validation.
nvx-rs/​src/​host/​template/​mod.rs	Re-exports capture types and documents their role.
nvx-rs/​src/​host/​sys/​mod.rs	Registers the Linux filesystem helper.
nvx-rs/​src/​host/​sys/​fs.rs	Implements Linux non-replacing rename.
nvx-rs/​src/​host/​supervisor/​mod.rs	Documents capture slots in the lock order.
nvx-rs/​src/​host/​store/​files.rs	Makes private file creation available within the crate.
nvx-rs/​src/​host/​platform/​windows.rs	Adds snapshot permission checks, sparse files, and safe renames.
nvx-rs/​src/​host/​platform/​other.rs	Adds unsupported-platform implementations.
nvx-rs/​src/​host/​platform/​linux.rs	Adds capture file checks and filesystem helpers.
nvx-rs/​README.md	Documents capture APIs, behavior, and testing.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform crash recovery, publication durability, and file-access protections warrant final human review alongside the pending CI matrix.

No open issues 🎉

3 resolved since last review

• Sync the store parent directory before reporting durable capture
• Select eligibility time after recovering the shared capture slot
• Recover and persist slots before generation lookup and capacity checks

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform durability and security-sensitive file handling warrant human review, and the published-snapshot recovery test needs reliable synchronization.

1 open issue

• Wait for snapshot publication before killing adapter · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform publication durability, process recovery, and Windows permission handling warrant final human review despite extensive test coverage.

No open issues 🎉

1 resolved since last review

• Wait for snapshot publication before killing adapter

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform crash durability, process recovery, and access-control changes warrant final human review despite extensive test coverage.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform publication durability, filesystem permissions, and orphan-process recovery warrant final human review despite no confirmed blocking findings.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform crash consistency, process recovery, and filesystem permission enforcement warrant final human review.

No open issues 🎉

_{Review effort: Balanced}