Skip to content

nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726

Description

Goal

Measure the coverage of nvx-rs's tests on Linux and Windows, for each family of golden tests, so that each step of #722 can show that the tests it adds reach what the tests it deletes reached.

Problem

On Windows, nvx-rs writes no coverage profile. src/bin/nvx-rs.rs exits through std::process::exit, which on Windows ends the process without running the exit handler with which LLVM's profiler writes its profile. In the analysis for #722, no nvx-rs process wrote a profile until the binary wrote it explicitly, while nvx-fake-openvmm, which returns from main, wrote them.

Tasks

  • Make nvx-rs write its profile before it exits, for example by returning from main where the status allows it, or by calling __llvm_profile_write_file under the cfg(coverage) that cargo llvm-cov sets, declared in [lints.rust] unexpected_cfgs.
  • Add a workflow that runs cargo llvm-cov with branch coverage on ubuntu-latest and windows-latest, with NVX_BUNDLE=skip, on demand and optionally on a schedule, but never as a required check. It uploads LCOV for all tests, for each family alone, and for all tests but the family.
  • Add an on-demand cargo mutants job that takes files and test filters.
  • Record the baseline of both OSes in nvx-rs: retire the Python-parity golden tests and make Rust the reference #722: the lines and branches that only each family reaches.
  • Document both in doc/ci.md.
Family Tests
Python semantics host::python::golden::
Launch profile and options host::sandbox::golden::
Egress policies every_fixture_case_compiles_as_python_compiled_it, unreadable_files_fail_as_in_python, and paths_print_as_pathlib_prints_them
CLI scenarios nvx_rs_reproduces_the_recorded_scenarios, in tests/cli.rs

Reference measurement

At 8b585d7, on Windows, with a scratch build that wrote the profile explicitly, the lines and branches of production code that each family reaches and that the tests outside the four families don't:

Family Lines Branches
CLI scenarios 533 130
Egress policies 9 6
Launch profile and options 7 1
Python semantics 6 12
All four 546 147

The wire-format byte tests, measured against all other tests, reach no line or branch that the others miss.

Acceptance criteria

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions