You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726
Measure the coverage of nvx-rs's tests on Linux and Windows, for each family of golden tests, so that each step of #722 can show that the tests it adds reach what the tests it deletes reached.
Problem
On Windows, nvx-rs writes no coverage profile. src/bin/nvx-rs.rs exits through std::process::exit, which on Windows ends the process without running the exit handler with which LLVM's profiler writes its profile. In the analysis for #722, no nvx-rs process wrote a profile until the binary wrote it explicitly, while nvx-fake-openvmm, which returns from main, wrote them.
Tasks
Make nvx-rs write its profile before it exits, for example by returning from main where the status allows it, or by calling __llvm_profile_write_file under the cfg(coverage) that cargo llvm-cov sets, declared in [lints.rust] unexpected_cfgs.
Add a workflow that runs cargo llvm-cov with branch coverage on ubuntu-latest and windows-latest, with NVX_BUNDLE=skip, on demand and optionally on a schedule, but never as a required check. It uploads LCOV for all tests, for each family alone, and for all tests but the family.
Add an on-demand cargo mutants job that takes files and test filters.
every_fixture_case_compiles_as_python_compiled_it, unreadable_files_fail_as_in_python, and paths_print_as_pathlib_prints_them
CLI scenarios
nvx_rs_reproduces_the_recorded_scenarios, in tests/cli.rs
Reference measurement
At 8b585d7, on Windows, with a scratch build that wrote the profile explicitly, the lines and branches of production code that each family reaches and that the tests outside the four families don't:
Family
Lines
Branches
CLI scenarios
533
130
Egress policies
9
6
Launch profile and options
7
1
Python semantics
6
12
All four
546
147
The wire-format byte tests, measured against all other tests, reach no line or branch that the others miss.
Acceptance criteria
The workflow's LCOV includes nvx-rs's profiles on Linux and Windows.
Activity
ci: run cargo-mutants on files of nvx-rs on demand
ci: measure what each golden-test family of nvx-rs alone reaches
ci: run cargo-mutants on files of nvx-rs on demand
ci: measure what each golden-test family of nvx-rs alone reaches
ci: run cargo-mutants on files of nvx-rs on demand
nvx-rs: return exit statuses from main so coverage builds write profiles
ci: measure what each golden-test family of nvx-rs alone reaches
ci: run cargo-mutants on files of nvx-rs on demand
nvx-rs: return exit statuses from main so coverage builds write profiles
ci: measure what each golden-test family of nvx-rs alone reaches
ci: run cargo-mutants on files of nvx-rs on demand
scripts: count every test that replays the egress fixture in its family
scripts: retire the nvx-rs coverage workflow
scripts: retire the nvx-rs coverage workflow
scripts: retire the nvx-rs coverage workflow