Repository navigation
openvmm: let platform snapshots activate vCPUs at restore - #801
Open
Pedro Henrique Penna (ppenna) wants to merge 6 commits into
Open
Pedro Henrique Penna (ppenna) wants to merge 6 commits into
Pedro Henrique Penna (ppenna) wants to merge 6 commits into
Conversation
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-platform snapshot and authentication changes require human review of complete exact-head CI evidence.
No open issues 🎉
What changed in this PR
Addresses #796 by updating the OpenVMM pin to support restore-time vCPU activation for platform snapshots and correct authentication timeout handling during gated restores. Adds NVX coverage and measurement tools on top of #797.
Changes:
- Adds a default microVM scenario covering CPU activation, platform repair, invalid targets, and snapshot integrity.
- Adds an explicit platform-snapshot benchmark for 1/2/4/8 online vCPUs, reporting latency, peak memory, and optional lifecycle phases.
- Rebalances CI shards and documents the new behavior and upstream changes.
| File | Description |
|---|---|
| scripts/test_nvx_tools.py | Tests benchmark arguments, capture reuse, targets, and metadata. |
| scripts/test_microvm_tests.py | Tests scenario execution, repair checks, VP bindings, and snapshot integrity. |
| scripts/nvx_tools/microvm_tests.py | Implements the platform restore-processor scenario. |
| scripts/nvx_tools/microvm_shards.json | Adds the scenario and rebalances backend shards. |
| scripts/nvx_tools/benchmark.py | Implements and registers the platform vCPU restore benchmark. |
| doc/usage.md | Documents scenario restrictions, reporting, and suite selection. |
| doc/run.md | Clarifies platform snapshot CPU activation support. |
| doc/openvmm-upstream-roadmap.md | Records the new pin and upstream fixup mapping. |
| doc/ci.md | Describes the new scenario’s correctness checks. |
| doc/benchmarks.md | Documents benchmark usage and inherited Windows capture behavior. |
Review effort: Balanced
Pin microsoft/openvmm#4734 at 8a0d945edca13b301a5152f225702057337b2a56, one commit on 79a501b63, the head of feature/microvm/dev and feature/microvm/release/v2.3, which #788 pins. A Windows microVM capture flushes guest RAM to the backing file that OpenVMM creates at launch. NTFS zero-fills such a new file below any write past its valid data length, and OpenVMM flushed it with up to eight concurrent FlushViewOfFile calls over disjoint ranges, so ranges were written twice: a 128 MiB guest wrote about 135 to 180 MiB per capture, a different amount each time. That spread capture.mapped_memory_flush within a lifecycle series even on a steady scratch volume and left the stability check of #169 less margin for the volume's throughput. OpenVMM now marks every page of the RAM view modified and flushes it in one ascending pass, so a capture writes at most 128 MiB, and fragmented dirty pages no longer turn the flush into seconds of small writes. The runners measured the identical patch on 7ff31ed7a, the v2.2 pin. At CI's Windows acceptance settings with F: scratch, on azure-windows-1, -3, and -4, snapshot generation fell from a p50 of 1056 to 1188 ms to 836 to 878 ms, with every series' p50 within 1.2% of its p25, and cold boot and restore stayed within their run-to-run variation. Linux hosts are unaffected. The change does not remove the external trigger of #169: when another writer halves the scratch volume's throughput mid-series, the series of both builds still split and are rejected. doc/benchmarks.md said that a 128 MiB capture writes about 150 MiB; it now describes the single pass. The roadmap records the commit as position 338, a PR02 fixup to fold into position 010. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin microsoft/openvmm#4735 at 2b681564dd63483f2f9c9f9bba3b98a4c00e9680, two commits on 8a0d945ed, the head of microsoft/openvmm#4734 that #797 pins, which is one commit on 79a501b63, the head of feature/microvm/dev. The first commit lets a platform snapshot opt into restore-time vCPU activation. Restore-time memory expansion already worked on every tier, but a platform capture booted with maxcpus=B failed to publish, because the platform tier's command-line allowlist did not accept maxcpus=, and one captured without it refused --restore-processors at every restore. OpenVMM now accepts one maxcpus= token on a platform command line, the canonical form of the recorded boot-online count, as it accepts nr_cpus=, so each clone of one platform capture can bring its own vCPU prefix online at restore. The second stops the control-console broker from timing a host's authentication while host input is gated. A host that connected as soon as it launched a gated restore, and authenticated right away, was detached whenever the guest's repair outlasted --microvm-control-auth-timeout-ms, although the broker had not read its authentication yet. The timeout is now suspended while input is gated and restarts when the input is released. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the two commits that the previous commit pins as positions 339 and 340 of the microVM chain, and move the NVX pin to 2b681564d. Position 339 lets platform snapshots opt into restore-time vCPU activation. It fixes position 120, "openvmm: bring a restore-time processor prefix online", which added the activation without extending the platform allowlist that position 116 had introduced, so it folds into 120 in PR20. Position 340 times host authentication only while the control-console broker reads host input. It fixes position 136, "virtio_console: attach the broker to serial I/O with receive credits", so it folds into 136 in PR23. Extend the scope, the chain, the fold instructions, the table of the commits after the mapped tip, the PR summary, and the commit mapping, and count the fixups in the PR20 and PR23 nodes of the graph. The pin is now 183 commits past the mapped tip. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The pinned OpenVMM accepts restore-time vCPU activation on platform snapshots. Say in the run guide's restore-readiness section that a platform snapshot opts in as other snapshots do, by capturing with a canonical maxcpus= boot-online prefix, so each clone of one platform capture can bring its own prefix online. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No test covered restore-time vCPU activation on a platform snapshot, which OpenVMM refused until the pinned fix. Add the platform-restore-processors microVM scenario to the default suite. The scenario captures a platform snapshot of capacity 4 whose only caller token is maxcpus=1, with a distro layer and a scratch disk, so the guest's nvx-snapshot takes the platform tier from the host. It restores that snapshot at each requested --restore-processors count up to the capacity, 1, 2, and 4 by default, and once without a target, each with a fresh scratch disk. Each restore must run the platform repair and pass the checks of restore-processors: the online CPU prefix, the warp probe over every pair of online CPUs, the time ABI's restore line, and the VP-binding profile, which on MSHV must bind only the target's prefix. OpenVMM must refuse the counts 0 and 3, which no microVM supports, and 8, above the capacity, before the guest runs, and no restore may change the snapshot. The scenario needs sandbox blocks, so it runs only with the Alpine sandbox-control guest, like snapshot-tiers. It took 10 to 17 s on bare-metal KVM, MSHV, and WHP hosts, so the shards are planned again with an estimate of 16 s, which CI's duration reports replace once CI has run it. Unit tests cover the capture and restore commands, the refused counts, the platform repair check, the MSHV prefix check, and the snapshot's integrity, and the runner tests cover its processor counts and its duration record. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the platform-snapshot-restore-vcpu workload suite, which restores one platform snapshot of capacity 8 at the online targets 1, 2, 4, and 8. Like snapshot-restore-vcpu, it reports each target's restore latency and OpenVMM's peak RSS, and its lifecycle phases with --snapshot-profile. A platform command line carries no tenant tokens, so the capture boots with maxcpus=1 as its only caller token, a distro layer, and a scratch disk, and the guest captures from its shell instead of with the pre-banner capture. Each restore reattaches the blocks and reaches its marker after the guest's platform repair, which onlines the target's prefix, refreshes the clone's identity, and acknowledges the restore gate. The restored guest then waits for the host to end it: a sandbox guest's virtio console can deliver the marker after a prequeued exit has started OpenVMM's teardown, too late to sample its peak RSS. The suite is explicit, so it changes no performance baseline. On bare-metal hosts, its medians are close to snapshot-restore-vcpu's at every target: for 1, 2, 4, and 8 vCPUs, 82, 136, 154, and 232 ms on Linux/MSHV, 68, 152, 200, and 231 ms on Linux/KVM, and 178, 385, 486, and 530 ms on Windows/WHP. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/fix-issue-796-implement-tests-benchmarks
branch
from
October 23, 2026 20:19
2f4bba1 to
f593442
Compare
This was referenced Oct 24, 2026
Pedro Henrique Penna (ppenna)
requested a review
from Enrique Saurez (esaurez)
October 25, 2026 01:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #796. Pins microsoft/openvmm#4735 at
2b681564dd63483f2f9c9f9bba3b98a4c00e9680, which lets platform snapshots bring a different vCPU prefix online in each clone at restore, as they can already grow their RAM. It adds a microVM scenario and a benchmark that cover that gap.Stacked on #797. This branch is #797's head (
aaf30ed88) plus five commits. Until #797 merges, the diff here includes its OpenVMM repin commit, so review only the last five. microsoft/openvmm#4735 is likewise stacked on microsoft/openvmm#4734, the8a0d945edthat #797 pins.Change
47361e3f8openvmmopenvmm8a0d945ed→2b681564d: the two commits of microsoft/openvmm#473544924e545docdoc/openvmm-upstream-roadmap.md82984fb5adocdoc/run.md--restore-processorsafter amaxcpus=capturedee57d4efscriptsscripts/nvx_tools/microvm_tests.py,scripts/test_microvm_tests.py,scripts/nvx_tools/microvm_shards.json,doc/ci.md,doc/usage.mdplatform-restore-processors; shards replannedf593442bescriptsscripts/nvx_tools/benchmark.py,scripts/test_nvx_tools.py,doc/benchmarks.md,doc/usage.mdplatform-snapshot-restore-vcpuOpenVMM fixes
maxcpus=(2f51e8bab). The platform tier's command-line allowlist acceptednr_cpus=but notmaxcpus=. A platform capture booted withmaxcpus=Btherefore failed to publish withplatform snapshot command line contains tenant or unsupported configuration, and one booted without it refused--restore-processors. OpenVMM now accepts onemaxcpus=token on a platform command line: the canonical form of the recorded boot-online count, supported and within the capacity.maxcpus=01,maxcpus=+1, duplicates, and mismatches stay rejected.2b681564d), the issue's lower-priority item. The broker enforced the authentication deadline even while a gated restore kept it from reading the host's authentication, so a host that connected at launch was detached whenever repair outlasted--microvm-control-auth-timeout-ms. The deadline is now suspended while input is gated and restarts at the release for a host that has not authenticated.Tests and benchmark
platform-restore-processorscaptures a platform snapshot of capacity 4 whose only caller token ismaxcpus=1, with sandbox blocks. It restores the snapshot at each requested count up to 4 (1, 2, and 4 by default) and once without a target, each with fresh scratch. Each restore must run the platform repair and pass the checks ofrestore-processors: the online CPU prefix, the warp probe, the time ABI's restore line, and the VP bindings (MSHV binds only the target's prefix). OpenVMM must refuse 0, 3, and 8 before the guest runs, and the snapshot must not change. It runs with the Alpine sandbox-control guest only, likesnapshot-tiers, and joins shard 1 on every backend with a 16 s estimate.platform-snapshot-restore-vcpurestores one capacity-8 platform snapshot at the online targets 1, 2, 4, and 8 and reports latency, peak RSS, and, with--snapshot-profile, the lifecycle phases. It is explicit, so no performance baseline changes.Evidence
On the bare-metal hosts prometheus30 (Linux/MSHV), prometheus32 (Linux/KVM), and prometheus28 (Windows/WHP), each in a separate worktree:
79a501b63binaryplatform snapshot command line contains tenant or unsupported configuratione2eacceptance benchmarkRestore medians in ms at 1, 2, 4, and 8 online vCPUs (capacity 8, 512 MiB, one warmup, five runs):
platform-snapshot-restore-vcpusnapshot-restore-vcpuEquivalence and validation
79a501b63and these NVX changes on openvmm: pin microVM v2.3, rebased onto upstream main #788's head. That is patch equivalence with this branch, not tree identity: the OpenVMM diff has the same patch-id (174b483ff) on79a501b63and on8a0d945ed, and the NVX diff, without the gitlink and the roadmap, has the same patch-id (2b42f7345) onee2e3c7e5and onaaf30ed88.8a0d945edadds only [DO NOT REVIEW] microvm: flush writable file views whole, in one ascending pass openvmm#4734's commit, the Windowssparse_mmapflush change and its guide text, which openvmm: write each page of a new microVM RAM file once at capture #797's CI validated.cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agentforopenvmm_helpers,virtio_console, andopenvmm_entry(404 passed), andcargo xtask fmt(clean).nvx.py verify,ruff check,ruff format --check,pyrightfor Linux and Windows, the markdown link check, thevalidate-nvxunittest list (1,220 tests, 45 skipped),test_adversarial.py,test_hosts.py, and the CLI--helpchecks. The scenario commitdee57d4efalso passes its tests on its own.scripts/test_setup_azure.py, whose shell cases need Linux, and the OpenVMM VMM tests, which CI runs.Exact NVX head submitted to CI:
f593442bea11d7cc3c60b39b594c6d7854849024. It replays the five commits of2f4bba171, whose CI passed (run 39964812987), onto #797's rebased head;git range-diffshows each commit patch-identical.