Repository navigation
nvx-rs: replace the golden CLI scenarios with structured scenario tests #728
Copy link
Copy link
Closed
Copy link
Description
Goal
Replace tests/fixtures/cli.json and its replay, nvx_rs_reproduces_the_recorded_scenarios in tests/cli.rs, with scenario tests that check what callers rely on: exit statuses, error codes and stable fragments, the state directory and its parsed JSON files, file modes, and the arguments that OpenVMM receives, rather than transcripts recorded from Python.
Background
cli.jsonholds 141 POSIX and 140 Windows scenarios, in 426 steps, recorded fromnvx.py sandbox,egress compile, andartifacts, withnvx-fake-openvmmas OpenVMM. Each step compares the exit status, standard output and error, and state files byte for byte, after replacing the root, process IDs, and random identifiers with placeholders.- On Windows, only the replay reaches 533 lines and 130 branches, mostly in
src/host/cli/sandbox/andsrc/testing/fake_openvmm.rs, and all ofegress compile. nvx.py sandboxpasses the streams and status ofnvx-rsthrough, and the hypervisor suites match message fragments and parseruntime.jsonand outcome reports.
Tasks
Replacement tests, in a pull request that doesn't change behavior:
- A harness in
tests/cli/that reuses the root and fake-OpenVMM setup oftests/cli.rs, and checks exit statuses, error codes (nvx-rs: give CLI errors stable machine-readable codes #724) or stable fragments, the entries of the state directory, its parsed JSON files, POSIX file modes, and the arguments that the fake OpenVMM received. - Table-driven tests:
usage_errors_exit_2_and_name_the_optionvalidation_errors_exit_1_before_any_side_effect: the state directory is unchanged, and OpenVMM didn't run.dry_runs_print_the_openvmm_command_and_launch_nothing: the>>line, split into arguments, holds the sandbox blocks, workload identity, shares, egress rules, and kernel command line.one_shot_runs_exit_with_the_workload_or_openvmm_status: statuses such as 0, 3, 7, 124, 127, and 137, the workload's output byte for byte, and the schema of the outcome report.lifecycles_of_every_configuration: the state after each ofprovision,start,exec,stop, anddeprovision.refusals_of_lifecycle_operationsconfigurations_of_every_format_start_and_exec, from hand-writtenconfig.jsonfiles of formats 1 to 6 intests/fixtures/state/.egress_compile_prints_the_library_loweringartifacts_list_every_guest_in_order- A property that the
>>line splits back into the arguments, withCommandLineToArgvWon Windows and a shell-word splitter on POSIX.
- A temporary table that maps each of the 281 scenarios, and the regressions of Track nvx.py managed sandbox bugs: PID-only liveness check wedges stop, deprovision, and start #369, Track managed sandbox bugs: --dry-run performs managed operations, and layered exec --cwd leaves PWD at / #373, Track managed sandbox lifecycle bugs: on Windows, provision waits out its timeout for a lock file that refuses access for good #464, and Track Rust sandbox exec bugs: async output collection bypasses timeouts #513, to tests, and a test that fails on a scenario that maps to none.
- Keep the other tests of
tests/cli.rs, including the Ctrl-C tests.
Retirement, in the next pull request:
- Delete
cli.json, the replay and its normalization helpers, and the mapping test. - Write only the newest
config.jsonformat, and keep reading formats 1 to 6. Drop the README's promise that an oldernvx.pycan operate a sandbox thatnvx-rsprovisioned. - Docs: replace the README's "Golden scenarios" section, and update its Testing section and
doc/project-structure.md.
Gate
- The lines and branches that only the replay reaches, on Windows and Linux (nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726), are reached by the new tests or deleted.
-
cargo mutantsonsrc/host/cli/sandbox/mod.rs,lifecycle.rs, andstate.rsmisses no mutant with the new tests that the replay catches. - Every scenario maps to a test.
Coordination
nvx-rs run(nvx-rs: make the nvx-rs CLI the backend of nvx.py run #638) gets its scenarios as tests of this harness, not as records from Python.- Until
cli.jsonis deleted, changes such as those of nvx-rs: share the VMM supervisor and the sandbox store #620 and nvx-rs: host cold integration for image-backed sandboxes #622 may edit its records by hand, but add no recorded scenarios.
Acceptance criteria
-
cli.jsonis gone, and the scenario tests pass on Linux and Windows.
Metadata
Metadata
Assignees
Labels
No labels
Activity
nvx-rs: build the roots of the CLI tests in a shared harness
nvx-rs: test the usage and validation errors of the CLI
nvx-rs: test the dry runs of the CLI and the splitting of their commands
nvx-rs: test the one-shot runs of the CLI
nvx-rs: test the lifecycles of managed sandboxes
nvx-rs: test the sandbox configurations of every format
nvx-rs: test egress compile and artifacts against the library
nvx-rs: map each recorded CLI scenario to the tests that replace it
nvx-rs: build the roots of the CLI tests in a shared harness
nvx-rs: test the usage and validation errors of the CLI
33 remaining items
nvx-rs: retire the golden CLI scenarios
nvx-rs: write sandbox configurations in format 7 alone
nvx-rs: test egress compile and artifacts against the library
nvx-rs: map each recorded CLI scenario to the tests that replace it
nvx-rs: pin the help of run in a fixture that nvx.py checks too
scripts: stop measuring the coverage of the CLI scenarios
nvx-rs: retire the golden CLI scenarios
nvx-rs: write sandbox configurations in format 7 alone
nvx-rs: pin the help of run in a fixture that nvx.py checks too
scripts: stop measuring the coverage of the CLI scenarios
nvx-rs: retire the golden CLI scenarios
nvx-rs: write sandbox configurations in format 7 alone
doc: state the testing policy that replaced recorded output
nvx-rs: describe the tests that replaced the golden tests