openvmm: refuse kernel command lines that the guest kernel would truncate by esaurez · Pull Request #661 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-20T20:57:56Z

Note

The openvmm gitlink points at 9f04d05c5441a9b31f7f81effa0ce7c4fc66a0d3, the head of nanvix/openvmm#138. Before this PR merged,
nanvix/openvmm main and microsoft/openvmm feature/microvm/dev fast-forwarded to it from d0bdbed8, so it needed no repin.

The x86 guest kernel keeps at most 2,048 bytes of the boot command line, including the terminating NUL, and silently drops the rest. OpenVMM appends the machine profile's own tokens after the caller's, so they were the first to be lost, while OpenVMM accepted effective command lines of up to 64 KiB (#570).

What changed

OpenVMM pin (microVM: refuse command lines that the guest kernel would truncate nanvix/openvmm#138): OpenVMM refuses, before boot, an effective microVM command line longer than 2,047 bytes or one with an embedded NUL, with an error that names the length and the limit, such as microVM kernel command line is 2048 bytes; the limit is 2047 bytes plus the terminating NUL. One shared check replaces the profile's and the snapshot contract's 64-KiB limits, and a restore refuses a saved command line over the limit instead of rewriting it.
Boot test: the command-line-limit microVM scenario, in the default test-microvm set that CI runs on every backend. With a NIC attached, it boots Alpine once with a one-byte padding token, cmdline_limit.padding=x, a module parameter that the kernel passes to neither init nor its environment, to learn the effective command line from the guest. It then pads the caller's tokens until the effective command line holds 2,047 bytes, and checks that the guest's /proc/cmdline holds all of it, token for token, and that the guest installed the IPv6 default route that the profile's last token, virtnet_gw6=, names. OpenVMM must refuse a command line one byte longer, before boot and with its error. Unit tests cover the scenario's command, guest script, and dispatch, and doc/ci.md describes it.
Callers: nothing else changes. nvx-rs keeps its 1,024-byte caller budget (COMMAND_LINE_BUDGET) and its early refusals, including of embedded NULs, and the CLI and launch goldens are unchanged.

Every NVX launcher fits

Launcher	Worst-case effective command line	Margin to 2,047 bytes
nvx-rs managed launch (`launch.rs`, which `aci_edge_sandboxes` also uses): the 1,024-byte caller budget, OpenVMM's tokens, and the fixed 85-byte virtio-fs fragment	about 1,619 bytes	about 428 bytes
nvx-rs sandbox: its 1,024-byte budget, which includes the virtio-fs fragment, then the block devices, network, and control console	about 1,692 bytes	about 355 bytes
`nvx.py run` with an aggregate share	about 1,731 bytes	about 316 bytes

OpenVMM's own tokens add at most 43 bytes of base tokens, 10 for nr_cpus=8, 56 for the workload identity, 23 for the lifecycle, 38 for the snapshot tier, and 336 for device discovery, of which the network bootstrap takes up to 201. nvx.py run with a single directory share or a plain --cmdline forwards the caller's tokens without a budget of its own, as before; OpenVMM is the authority there, and a command line that does not fit now fails before boot with OpenVMM's error instead of booting truncated.

The boot protocol's design text, doc/design/5.1-boot-protocol.md, still states 64 KiB; draft #591 rewrites that paragraph to the 2,048-byte rule, so this PR leaves it alone.

Part of #622.
Part of #570.

OpenVMM revision

Source: microVM: refuse command lines that the guest kernel would truncate nanvix/openvmm#138, branch esaurez/microvm-command-line-limit-20261020
Pinned: 9f04d05c5441a9b31f7f81effa0ce7c4fc66a0d3, six commits that the nanvix/openvmm maintainer signed, directly on d0bdbed8f6e3b86271277dc22658109e2e633b2d, the commit of microsoft/openvmm feature/microvm/dev that dev pins: f894fbdd1, the change, and five test-only commits from review, 88d4832ba, 452ab29a4, e6df0c3ea, d0a1b820c, and 9f04d05c5, which derive the tests' expected errors, lengths, and effective command lines from the shared check, the limit, and the profile's own token functions
Scope: the command-line limit only; the loader's 64-KiB command-line region and the guest kernel are unchanged
Dependencies: none
Equivalence: the six commits re-create the reviewed fee2eec7e, 03171584c, c4bb0256b, c3c61d184, 3eebb3976, and 2a9556f3e, in order, on the same base, with the maintainer's signature. Each keeps its original's tree, author, author date, and message, and git range-diff reports all six as identical, so the pinned tree, be9604c40855174ee0116bbd84f4db264b91624e, is byte-for-byte that of 2a9556f3e, and the validation below, which names the original commits, holds for the trees of their re-created counterparts

Validation

The runtime runs used OpenVMM built from fee2eec7e; 03171584c, c4bb0256b, c3c61d184, 3eebb3976, and 2a9556f3e change only OpenVMM tests, which pass on Windows and Linux (nanvix/openvmm#138). Most NVX runs used a008a92f, this patch on f7882746, before dev moved. At ec9a0ecc, the same patch pinned to 3eebb3976 on 401c95a3, whose newer commits change nvx-rs's guest spawn (#629), the adversarial executor (#657), and performance-baseline data, none of which this patch or its scenario uses (the scenario starts OpenVMM directly), I reran verify, the quality checks, the NVX CLI tests, the nvx-rs checks, and the new scenario on WHP. The previous head, 7f7e963b, differed from ec9a0ecc only in 2a9556f3e, a test-only OpenVMM commit, and two performance-baseline data commits of dev; at it I reran verify and test_performance.py. The last paragraph covers this head.

On Windows 11 (Windows/WHP):

python scripts\nvx.py verify
python scripts\nvx.py build-openvmm at the pin, then python scripts\nvx.py test-microvm --backend whp: the full default set, including command-line-limit, passes with the guest artifacts of v0.1.0-dev.e0398adc52af and nvx-rs built from the checkout; command-line-limit also passes with --guest ubuntu and --guest azurelinux, and again at ec9a0ecc
ruff check and format, pyright for Linux and Windows, and the Markdown link check: clean; shellcheck and shfmt, with check-quality's images, on the new guest script: clean
validate-nvx's unittest modules: 1,064 tests, none failing and 34 skipped on this platform; test_hosts.py passes; the four process-containment tests of test_adversarial.py that fail on this host fail identically on dev, and of its other 73 tests, 67 pass and 6 are skipped
nvx-rs cargo fmt --all --check: clean
OpenVMM at the pin, as microVM: refuse command lines that the guest kernel would truncate nanvix/openvmm#138 lists: its workspace unit tests (4,937 of 4,938; lxutil tests::stat_fs fails on this host without the change too), its doc tests, and the 29 VMM tests of test-openvmm --backend whp

On Linux/MSHV (Azure Linux 3.0, a nested virtual machine), with OpenVMM built by nvx.py build-openvmm --backend mshv at the pin:

the nvx-rs checks with NVX_BUNDLE=skip, at ec9a0ecc: both Clippy runs with -D warnings, cargo test --locked --all-features, cargo test --locked, and cargo doc with -D warnings
command-line-limit, guest-boot, virtio-net, snapshot-core, and network-snapshot pass. This host cannot present the built-in CPU profile that --cpu-profile auto selects (amd.genoa.v1 needs tsa_l1_no_support and tsa_sq_no_support), and test-microvm takes no profile, so they ran with OpenVMM wrapped to add its development --cpu-profile host

On Linux/KVM (WSL2 Ubuntu 22.04, nested), with the pin's static musl OpenVMM:

command-line-limit, guest-boot, virtio-net, network-snapshot, and snapshot-core pass on --cpu-profile host, for the same reason (intel.alderlake.v1 needs MSR 0x10a bits that this KVM does not expose); snapshot-core failed once with a 2-second clock skew under host load, then passed twice, as it did twice with dev's OpenVMM
validate-nvx's unittest modules pass, test_microvm_tests.py with NVX_RS naming an nvx-rs built from the checkout; test_adversarial.py's sigkill containment case fails here as it does on dev

In each backend's command-line-limit run, the guest's /proc/cmdline held exactly 2,047 bytes ending with virtnet_gw6=fd00::a00:1, and OpenVMM refused the 2,048-byte command line before boot with its error.

Left to CI: the full default test-microvm set on KVM and MSHV with --cpu-profile auto; the other guests', debug-kernel, and public sandbox sets; test-openvmm on KVM and MSHV; the Linux OpenVMM unit tests; the guest artifact builds; and the benchmarks.

This head, 494ee980, re-creates 7f7e963b on dev at f2e7c984, with the pin moved to the re-signed 9f04d05c5. Its patch matches 7f7e963b's apart from the gitlink and the pin line of its message, and doc/ci.md and scripts/test_microvm_tests.py merge cleanly with dev's newer changes to them. On Windows 11, at this head, verify and test_performance.py pass. At ecf7dca7, the same commit on 5418be9a, whose tree differs from this head's only in dev's performance-baseline data, ruff check and format, compileall, pyright for Linux and Windows, and the Markdown link check are clean, and validate-nvx's unittest modules ran 1,090 tests, 35 of them skipped. Their only failures also fail on dev on this host: the same 18 tests of test_setup_azure.py, and a 10-second timeout of Git for Windows' sh.exe in test_host_loopback_allow_script_checks_publication_and_traffic, which failed in two of three runs on dev as well.

NVX head submitted to CI: 494ee980a70b31dcf09e1417d0188c0a7ce2d661.

Copilot

🔵 Needs a closer look

The review-only OpenVMM revision still requires upstream promotion, repinning, equivalence verification, and final CI validation before merge.

No open issues 🎉

What changed in this PR

Adds NVX regression coverage for OpenVMM’s guest-kernel command-line limit, addressing #570 without changing launcher budgets.

Changes:

Tests successful boot at 2,047 bytes and rejection at 2,048 bytes.
Verifies the complete guest command line and IPv6 default route.
Adds unit coverage and CI documentation.

File	Description
scripts/test_microvm_tests.py	Tests scenario commands, validation, and dispatch.
scripts/nvx_tools/microvm_tests.py	Adds the default command-line boundary scenario.
scripts/nvx_tools/microvm_test_scripts/command-line-limit.sh	Reports the command line and checks the IPv6 route.
doc/ci.md	Documents boundary-test behavior.

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The explicitly blocked OpenVMM revision requires upstream integration, repinning, and exact-head validation before human approval.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The review-only OpenVMM revision still requires upstream landing, repinning, and exact-head validation before merge.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The explicit merge prerequisite requires upstream OpenVMM promotion, repinning, and validation of the final revision.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The provisional OpenVMM revision still requires upstream integration, repinning, and final validation before merge.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The explicitly blocked OpenVMM revision still requires upstream integration, repinning, and validation before merge.

No open issues 🎉

_{Review effort: Balanced}

…cate Part of #622. Part of #570. Pin nanvix/openvmm#138 at 9f04d05c5441a9b31f7f81effa0ce7c4fc66a0d3, six commits on d0bdbed8. The x86 guest kernel keeps at most 2,048 bytes of the boot command line, including the terminating NUL, and silently drops the rest. OpenVMM accepted effective microVM command lines of up to 64 KiB, and it appends its own tokens after the caller's, so the machine profile's tokens were the first to be lost. OpenVMM now refuses, before boot, an effective command line longer than 2,047 bytes or one with an embedded NUL, with an error that names the length and the limit. A restore refuses a saved command line over the limit instead of rewriting it. Add the command-line-limit microVM scenario. With a NIC attached, it boots once with a one-byte padding token, a module parameter that the kernel passes to neither init nor its environment, to learn the effective command line from the guest. It then pads the caller's tokens until the effective command line holds 2,047 bytes, and checks that the guest's /proc/cmdline holds all of it and that the guest installed the IPv6 default route that the last token, virtnet_gw6=, names. OpenVMM must refuse a command line one byte longer. Describe the scenario in the CI guide. Every NVX launcher fits. nvx-rs keeps its 1,024-byte caller budget and its early refusals, and the worst-case effective command lines of its managed launches and sandboxes, and of nvx.py run with an aggregate share, stay more than 300 bytes under the limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54816c68-c749-4008-8ae9-b3da19544730

Copilot

🔵 Needs a closer look

The explicit merge hold requires upstream OpenVMM integration, repinning, and validation of the promoted revision.

No open issues 🎉

_{Review effort: Balanced}

The default Alpine suite of test-microvm runs its 36 scenarios one after another: 457.6 seconds on KVM, 416.4 on MSHV, and 596.3 on WHP, as the medians of the scenario duration reports (#694) of the 13 CI runs of October 21 and 22 add up. On WHP its job bounds every run that skips both OpenVMM suites (#696). No scenario depends on another, so CI can run the suite in parallel shards, as long as every scenario runs in exactly one of them and the shards take about as long as each other, whatever number of scenarios each runs. scripts/nvx_tools/microvm_shards.json is that partition: for each backend, the median duration of every default scenario, the CI runs whose reports gave them, and each shard's scenarios with their total. The backends differ too much for one partition: console-snapshot takes 49 seconds on KVM but 1 on MSHV and 4 on WHP, and console-exit 25 on KVM but 87 on WHP. The best partition common to all three that a search found left the longest KVM and MSHV shards about 5% above an even split, where one per backend balances each to within a second, so each backend has its own. The new plan-microvm-shards command writes the file. It takes each backend's median of every default scenario over the reports that it is given, sharded or not, of the Alpine guest on the default kernel with the default processor counts, from runs that ended; a report counts for a scenario only when every execution of it passed. Each scenario starts in its current shard, and one without a shard goes, longest first, to the shortest. A shard that is still empty, as one that --shards adds can be, takes the longest scenario of the longest shard that has more than one, since moving a scenario that takes no time into it would shorten no shard. The plan then moves a scenario, or swaps two of different shards, while that shortens the longest shard by a second or more, so that new medians that move within the run-to-run noise leave the shards alone. A scenario that no report measures keeps its duration, or takes an estimate from --seconds, and --shards changes a backend's number of shards. The first plan started from the two shards that #696 proposed, the restore, snapshot, and time ABI scenarios and the network, filesystem, and lifecycle ones, which were up to 18% apart on KVM, and a single swap per backend balanced each: 228.9 and 228.7 seconds on KVM, 208.0 and 208.4 on MSHV, and 298.3 and 298.0 on WHP. test-microvm --shard INDEX/COUNT runs one shard of the default suite. It fails before it runs a scenario unless the file puts every default scenario in exactly one shard of every backend and splits the backend's suite into COUNT shards. A scenario that a later change adds, as #661 added command-line-limit, therefore cannot drop out of a sharded run, and a run that counts a different number of shards than the file fails. The option takes no --scenario, --debug-kernel, or other guest. Only the microVM tests use the new module and its data, so the classifier leaves them out of the platform jobs' inputs, and doc/ci.md says so. New tests check that the checked-in file puts each default scenario in exactly one shard of each backend and is the planner's own plan of its durations, and that the shards of each backend together run the default suite that an unsharded run selects, each scenario once. They check that --shard fails before any scenario runs for a new, removed, doubled, or missing scenario, an empty shard, or a count that differs from the file's, and that it rejects other selections. They also cover the planner's balancing, its threshold, its placement of new scenarios and added shards, which it fills even with scenarios that take no time, its determinism, which reports it counts, the command's options and errors, and malformed files. The classification tests cover the new files. doc/usage.md documents the command and the option, and doc/ci.md how the shards are planned and how to plan them again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The default Alpine suite of test-microvm runs its 36 scenarios one after another: 457.7 seconds on KVM, 416.0 on MSHV, and 601.5 on WHP, as the medians of the scenario duration reports (#694) of the 18 CI runs of October 21 and 22 that the partition lists add up. On WHP its job bounds every run that skips both OpenVMM suites (#696). No scenario depends on another, so CI can run the suite in parallel shards, as long as every scenario runs in exactly one of them and the shards take about as long as each other, whatever number of scenarios each runs. scripts/nvx_tools/microvm_shards.json is that partition: for each backend, the median duration of every default scenario, the CI runs whose reports gave them, and each shard's scenarios with their total. The backends differ too much for one partition: console-snapshot takes 49 seconds on KVM but 1 on MSHV and 4 on WHP, and console-exit 25 on KVM but 87.5 on WHP. The best partition common to all three that a search found left the longest KVM and MSHV shards about 5% above an even split, where one per backend balances each to within a second, so each backend has its own. The new plan-microvm-shards command writes the file. It takes each backend's median of every default scenario over the reports that it is given, sharded or not, of the Alpine guest on the default kernel with the default processor counts, from runs that ended; a report counts for a scenario only when every execution of it passed. Each scenario starts in its current shard, and one without a shard goes, longest first, to the shortest. A shard that is still empty, as one that --shards adds can be, takes the longest scenario of the longest shard that has more than one, since moving a scenario that takes no time into it would shorten no shard. The plan then moves a scenario, or swaps two of different shards, while that shortens the longest shard by a second or more, so that new medians that move within the run-to-run noise leave the shards alone. A scenario that no report measures keeps its duration, or takes an estimate from --seconds, and --shards changes a backend's number of shards. The first plan, from 13 of the runs, started from the two shards that #696 proposed, the restore, snapshot, and time ABI scenarios and the network, filesystem, and lifecycle ones, which were up to 18% apart on KVM, and a single swap per backend balanced each. Planned again from all 18, which add two later dev pushes, another pull request's run, and two runs of this change, every scenario stayed in its shard: 228.8 and 228.9 seconds on KVM, 208.0 and 208.0 on MSHV, and 300.4 and 301.1 on WHP. test-microvm --shard INDEX/COUNT runs one shard of the default suite. It fails before it runs a scenario unless the file puts every default scenario in exactly one shard of every backend and splits the backend's suite into COUNT shards. A scenario that a later change adds, as #661 added command-line-limit, therefore cannot drop out of a sharded run, and a run that counts a different number of shards than the file fails. The option takes no --scenario, --debug-kernel, or other guest. Only the microVM tests use the new module and its data, so the classifier leaves them out of the platform jobs' inputs, and doc/ci.md says so. New tests check that the checked-in file puts each default scenario in exactly one shard of each backend and is the planner's own plan of its durations, and that the shards of each backend together run the default suite that an unsharded run selects, each scenario once. They check that --shard fails before any scenario runs for a new, removed, doubled, or missing scenario, an empty shard, or a count that differs from the file's, and that it rejects other selections. They also cover the planner's balancing, its threshold, its placement of new scenarios and added shards, which it fills even with scenarios that take no time, its determinism, which reports it counts, the command's options and errors, and malformed files. The classification tests cover the new files. doc/usage.md documents the command and the option, and doc/ci.md how the shards are planned and how to plan them again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 20, 2026 20:57

Enrique Saurez (esaurez) mentioned this pull request Oct 20, 2026

microVM: refuse command lines that the guest kernel would truncate nanvix/openvmm#138

Merged

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 20, 2026 20:58

View session