nvx-rs: measure line, branch, and mutation coverage on Linux and Windows by ppenna · Pull Request #806 · microsoft/nvx

Pedro Henrique Penna (ppenna) · 2026-10-23T14:44:19Z

Closes #726. Part of #722 (step 0).

#722 retires the Python-parity golden tests of the nvx crate one family at a time. Each retirement is gated on showing that the tests that remain still reach every line and branch, and catch every mutant, that only the retired family did, on Linux and Windows. This PR adds those measurements.

Commits

nvx-rs: return exit statuses from main so coverage builds write profiles. On Windows, std::process::exit skips the C runtime's exit handlers, and an instrumented build writes its LLVM profile from one of them. So no nvx-rs process, and no one-shot or run nvx-fake-openvmm process, ever wrote a coverage profile. Both now return their status from main as an ExitCode when it fits in 0–255, and still exit otherwise, so callers see the same statuses. Under cargo-llvm-cov on Windows, the test that runs nvx-rs run against the fake reached none of nvx::host::cli::main before; now its 13 nvx-rs processes reach all of it.
ci: measure what each golden-test family of nvx-rs alone reaches.
- scripts/nvx_rs_coverage.py runs the crate's tests under cargo llvm-cov --branch from one build, once per selection: all tests, each family alone, all but each family, and all but the families.
- It writes one LCOV report per selection and summarizes, per family and OS, the production lines and branches that only the family reaches, with where they are.
- nvx-rs-coverage.yml runs it on ubuntu-latest and windows-latest weekly, on dispatch, and for pushes that change it, and combines both systems into one summary.
- Artifacts have run-scoped names that a rerun overwrites, so after "re-run failed jobs" the summary takes the latest measurement of each OS.
ci: run cargo-mutants on files of nvx-rs on demand. nvx-rs-mutants.yml takes the files to mutate, test filters, cargo test target arguments, the runner, and 1–16 shards, and merges the latest outcomes of every shard. It runs cargo-mutants in place in each job's disposable checkout, because the CLI's tests need the repository around the crate, which cargo-mutants' default copy lacks. Comparing two runs, one with a family's tests and one with its replacement, gives the mutants that only the family catches.

doc/ci.md documents both ("nvx-rs coverage" and "nvx-rs mutation testing"), and doc/project-structure.md and the nvx-rs README point to them.

No required check changes. Neither workflow runs for pull_request. Like copilot-setup-steps.yml, each runs for pushes that change it, which is how this branch exercised them on its final head: the coverage run and the mutants smoke run, which caught 24 of 30 mutants of src/gpt/crc.rs, with none missed and 6 unviable.

Found while measuring on Linux

llvm-profdata crashed (SIGILL) on unreadable profiles. tests/store.rs's a_restrictive_umask_leaves_the_store_usable runs children under umasks 400, 200, and 777. They write their profiles under the same umask, so two of them aren't readable by their owner. The script now restores the owner's read and write permission on every profile before it reports a selection.
guest::sys::signal::tests::a_thread_that_blocks_sigpipe_keeps_only_a_signal_that_was_pending is flaky. It failed in 2 of 14 instrumented full test runs in WSL, and in none on the CI runners so far. Each time its write succeeded after it dropped the pipe's reader, apparently because another test's fork()ed child had inherited the read end. A measurement now runs a selection whose tests fail once more, and reports the retry. The test itself is untouched here; it deserves its own issue.

Baseline

From the coverage run on the final head. These are the production lines and branches that only each family reaches: all tests reach them, and the tests outside the family don't.

Family	Linux lines	Linux branches	Windows lines	Windows branches
Python semantics	12	12	6	10
Launch profile and options	3	2	1	0
Egress policies	6	3	0	1
CLI scenarios	297	98	296	97
All families	309	112	306	110

Beyond the tests outside the four families, which is #726's reference definition, the families reach these lines and branches on Linux, then on Windows:

Python semantics: 8 and 14, then 6 and 12.
Launch profile: 1 and 1, then 1 and 1.
Egress policies: 3 and 3, then 3 and 2.
CLI scenarios: 300 and 97, then 299 and 97.
All families: 309 and 112, then 306 and 110.

The tests reach 40,171 of 46,369 production lines on Linux and 22,694 of 24,330 on Windows.

On Linux, part of each small family's count is timing-dependent paths that a full run happened to reach, and they differ from run to run. In this run they are a BrokenPipe in guest/exec/binding.rs, a wait in guest/process.rs, and a wait loop in a guest/cgroup.rs test. That's why the summary lists where every line is, and why the gates should read those lists, not only the counts.

Validation

Windows, locally: cargo fmt --check, Clippy (--all-targets --all-features and --lib --no-default-features, -D warnings), the tests with all, default, and no default features, rustdoc with -D warnings, the 1.89 check, and the aci_edge_sandboxes tests all pass.
Linux (WSL Ubuntu), locally: full measurements, which exercised the profile repair and the retry. The script's tests pass, including the mutants workflow's bash steps run with a stub Cargo.
Ruff, Pyright (Linux and Windows), check_markdown_links.py, and actionlint with ShellCheck are clean.
The validate-nvx unit-test list passes, except for 18 test_setup_azure subtests on my Windows host, where bash resolves to the WSL launcher. They're unrelated, and they pass on GitHub's runners.

Open item for #722

#722 still has to decide whether the weekly schedule stays. The workflow has one for now, so that the next steps start from a fresh baseline.

Copilot

🟡 Changes recommended

Mutation jobs fail on clean runners, and both workflows exclude earlier successful artifacts during partial reruns.

3 open issues

What changed in this PR

Adds coverage and mutation measurements to support retiring nvx-rs Python-parity golden tests under #722 and #726, without adding required checks.

Changes:

Return representable exit statuses from Rust entry points so Windows coverage builds write profiles.
Measure each golden-test family’s line and branch coverage on Linux and Windows, and add configurable mutation-testing jobs.
Add tooling tests, validation integration, and usage documentation.

File	Description
`scripts/test_nvx_rs_coverage.py`	Tests coverage analysis, command execution, and workflow behavior.
`scripts/nvx_rs_coverage.py`	Collects and summarizes coverage by golden-test family.
`nvx-rs/src/testing/fake_openvmm.rs`	Returns compatible exit statuses to permit profile writing.
`nvx-rs/src/bin/nvx-rs.rs`	Returns compatible CLI exit statuses from `main`.
`nvx-rs/README.md`	Links to coverage and mutation-testing documentation.
`doc/project-structure.md`	Lists the new measurement tooling and workflows.
`doc/ci.md`	Documents measurements, artifacts, limitations, and comparisons.
`.github/workflows/nvx-rs-mutants.yml`	Adds configurable, sharded mutation testing and merged outcomes.
`.github/workflows/nvx-rs-coverage.yml`	Adds Linux and Windows coverage collection and summaries.
`.github/actions/validate-nvx/action.yml`	Includes coverage-tooling tests in validation.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Relative coverage-target overrides make profile-permission repair scan the wrong directory, potentially breaking Linux coverage reporting.

3 open issues

1 resolved since last review

Create cargo-mutants output directory before running

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Relative coverage-target overrides resolve against the wrong directory, bypassing profile permission repair and potentially breaking Linux measurements.

1 open issue

Resolve relative target-directory overrides against the crate directory · New

3 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Default unfiltered mutation runs fail their baseline because the temporary workspace lacks the checkout layout required by an existing CLI integration test.

1 open issue

Default cargo-mutants baseline fails in temporary workspace · New

1 resolved since last review

Resolve relative target-directory overrides against the crate directory

_{Review effort: Balanced}

Copilot

🟢 Approval recommended

No unresolved blocking issues remain in the exit-status changes, measurement tooling, workflow artifact handling, or supporting tests.

1 open issue

Default cargo-mutants baseline fails in temporary workspace

_{Review effort: Balanced}

On Windows, std::process::exit ends a process with ExitProcess, which skips the C runtime's exit handlers, and LLVM's profiler runtime writes the coverage profile of an instrumented build from one of them. The nvx-rs binary passed the CLI's status to std::process::exit, so under cargo llvm-cov no nvx-rs process wrote a profile, and nvx-fake-openvmm, which ends its one-shot and run lifecycles the same way, wrote none for them either. Every line and branch that only those processes run, such as the CLI's sandbox lifecycle and all of egress compile, read as unreached on Windows, which leaves the measurements that #726 adds for the retirement of the golden tests (#722) without them. Both now return the status from main as an ExitCode when it fits one, so that the C runtime runs its exit handlers. A status outside 0 to 255, such as the status of an OpenVMM that a Windows exception ended, or a negated signal on Unix, where exit runs the handlers anyway, still ends the process with std::process::exit, so callers see the same statuses as before. The fake's fake_crash_after_ms crash still exits from another thread, as the crash that it stands for would. Under cargo-llvm-cov 0.9.1 on Windows, the test that runs nvx-rs run against the fake OpenVMM reached none of nvx::host::cli::main and none of the fake's run_microvm before this change; now the profiles of its 13 nvx-rs processes reach both. rustfmt, Clippy with warnings denied, the crate's tests with every feature, with the default ones, and without them, rustdoc, the 1.89 check, and the aci_edge_sandboxes tests pass on Windows. Part of #722. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The golden tests of the nvx crate retire one family at a time (#722), and each retirement has to show that the tests that remain reach every line and branch that only the retired family reached, on Linux and Windows (#726). Nothing measured that. scripts/nvx_rs_coverage.py measures it on the host that runs it. Its measure command runs the crate's tests with every feature under cargo-llvm-cov, with branch coverage, from one build, once for each selection: every test, each of the four families alone (Python semantics, the launch profile and options, egress policies, and the CLI scenarios), every test but each family, and every test outside the families. It first lists the tests and fails if a family's filter selects none, so that a renamed golden test cannot empty its family unnoticed. It writes the LCOV report of each selection, with paths relative to the crate, the list of tests, and measurement.json, which records the selections, the statuses and duration of their runs, the toolchain, the compiler, the commit, the families, and the crate's test-only modules, which it finds from the crate's cfg(test) module declarations. Two things that a measurement meets on Linux needed handling: - A test may fail only sometimes, as a guest signal test whose pipe another test's forked child held open did once in six runs, so a selection whose tests fail runs once more, and its report holds what its last run reached. The measurement exits with status 1 after the last selection if the tests of a selection failed in both runs. - A process writes its profile under its umask, and the store tests run processes whose umasks clear the owner's permissions, on whose profiles llvm-profdata crashed with SIGILL. Before it reports a selection, the script lets the owner read and write every profile in cargo-llvm-cov's target directory. Its summarize command reads one or more measurements, checks that their reports instrument the same lines and branches, and prints a Markdown summary, and JSON on request. For each family, it counts and locates the lines and branches of production code that only the family reaches, which every test reaches and the tests outside the family don't, and that a retirement has to reach with other tests or delete. It also counts those that the family reaches and the tests outside every family don't, as the reference measurement of #726 does, and names the selections whose tests failed, or passed only when they ran again. .github/workflows/nvx-rs-coverage.yml runs the script on GitHub-hosted Ubuntu and Windows runners, weekly, on demand, and for pushes that change the workflow or the script, never for pull requests, so no required check changes. It pins the nightly toolchain that branch coverage needs and cargo-llvm-cov 0.9.1, uploads each system's reports and measurement, also when tests failed, and combines the measurements that finished into the run's summary, which gives the baseline of both systems that #722 records. The unit tests cover the selections, the families against the crate's sources, the test-only modules of a synthetic crate and of the crate itself, the LCOV reader, the analysis and its consistency checks, the summary, and the commands that measure runs, with a fake in place of Cargo, including the second run of a selection whose tests fail and the repair of a profile that its owner can't read. They also check that the workflow never runs for pull requests and pins its tools. validate-nvx runs them with the other script tests. doc/ci.md describes the measurement, its artifacts and summary, what counts as production code, what failed tests and killed processes leave out, and how to measure on a development host; doc/project-structure.md and the nvx-rs README point to it. Measured locally on this branch with the workflow's toolchain, a full measurement took 4 minutes on Linux, in WSL, and 6 on Windows. Only the CLI scenarios reach 301 lines and 97 branches on Linux, and 296 and 97 on Windows; the four families together reach 313 and 114, and 307 and 111. Ruff, Pyright for Linux and Windows, the script's tests on Windows and Linux, the CLI validation tests, the Markdown link check, and actionlint with ShellCheck pass. Part of #722. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Besides coverage, the gate of each retirement of #722 asks that cargo-mutants, run on the retired family's files, misses no mutant with the new tests that the family's tests catch (#726). Nothing in CI ran cargo-mutants. .github/workflows/nvx-rs-mutants.yml runs cargo-mutants 27.1.0 on GitHub-hosted runners on a manual dispatch, whose inputs name the files to mutate, as globs relative to nvx-rs, the arguments of the test binaries that select the tests, such as a family's filters, the arguments of cargo test that select the test targets, the runner, and the number of jobs, from 1 to 16, that share the mutants through --shard. Its plan job refuses inputs that span lines, an unknown runner, and a number of jobs out of range before any job starts. Each shard job passes the inputs to cargo-mutants as separate arguments, without expanding their globs, and treats missed mutants and timeouts as outcomes rather than failures. It runs cargo-mutants in place, in its disposable checkout: in the copy of the crate that cargo-mutants makes by default, artifacts_default_to_the_checkout_that_holds_the_executable finds no repository around the crate, so the unmutated baseline of every run that includes the CLI's tests failed. It also creates the parent of cargo-mutants' output, which cargo-mutants doesn't. The summarize job merges the shards' caught, missed, timeout, and unviable lists into one artifact, counts them in the run's summary with the missed mutants, and fails if a shard reported nothing. The artifacts' names hold the run but not its attempt, and their uploads overwrite, so that after a rerun of the jobs that failed, the merge takes the latest outcomes of every shard, each once. Comparing the caught lists of two runs, one with the family's tests and one with the tests that replace it, gives the mutants that only the family catches. Like the coverage workflow, it never runs for pull requests, so no required check changes; a push that changes it checks it on src/gpt/crc.rs with the gpt::crc:: tests, in two jobs. The script tests now also check that the workflow never runs for pull requests, pins cargo-mutants, and names its artifacts so that reruns replace them, and, on POSIX hosts, run its plan, shard, and merge steps with a stub of Cargo: the inputs that pass and those that fail, the arguments that cargo-mutants gets with and without test and target arguments, the parent of its output, the statuses that fail a shard, and the merge of complete and missing shards. doc/ci.md describes the inputs and outputs and how to compare two runs, and doc/project-structure.md and the nvx-rs README point to it. Locally on Windows, an unfiltered run on one of 30 shards of src/gpt/crc.rs failed its baseline in a copy of the crate and caught its mutant in place; the push-triggered smoke run caught 24 of the 30 mutants, with none missed and 6 unviable. The script tests pass on Windows and on Linux, where the step tests run, and so do Ruff, Pyright for Linux and Windows, the Markdown link check, and actionlint with ShellCheck. Part of #722. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

No unresolved blocking issues remain, and focused regression coverage plus successful final-head coverage and mutation smoke runs support approval.

No open issues 🎉

1 resolved since last review

Default cargo-mutants baseline fails in temporary workspace

_{Review effort: Balanced}

The nvx-rs coverage workflow runs the crate's tests built with -C instrument-coverage, and has failed on Linux since #806 merged: managed_agent_helper_applies_inherited_exact_and_layered_environments fails in every selection of tests, as in dev's run 39990930082. The LLVM profiling runtime of an instrumented executable sets __LLVM_PROFILE_RT_INIT_ONCE in the environment of its own process when it starts, and the instrumented nvx-managed-agent hands that environment on to a workload that inherits the default environment, alone or beneath the request's entries. The test compared the environment that /usr/bin/env printed exactly, so it met an entry that no uninstrumented build has. The test now leaves that one variable out of the environments of the inherited and layered cases, the two that take the helper's own environment, and still compares the rest of them exactly. The cases that replace the environment are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ff659e8f-6bed-4a2c-8c84-1835fc7d4834

Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:44

Copilot started reviewing on behalf of Pedro Henrique Penna (ppenna) October 23, 2026 14:44

View session

Pedro Henrique Penna (ppenna) force-pushed the agents/issue-722-fix-and-validate-commits branch from 0f053e0 to 5f0a468 Compare October 23, 2026 14:48