Repository navigation
ci: cut OpenVMM and CLI work from the critical path and the WHP pool - #436
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoOct 8, 2026
Merged
Conversation
The openvmm-unit-tests matrix ran the same Linux suite on a KVM runner
and on an MSHV runner. The OpenVMM unit tests that need a hypervisor
device are #[ignore]d ("requires /dev/kvm", "requires /dev/mshv"),
test-openvmm-unit runs nextest without --run-ignored, and no other unit
test or doctest probes the device. Run 37635396339 shows the result:
both Linux legs ran the identical 4,820 tests across 414 binaries, with
the same 28 skipped. The second leg added no coverage but held a
hypervisor runner for 7 to 8 minutes of every run that tests OpenVMM.
Run the suite once per OS instead, on the MSHV pool and on WHP. The MSHV
pool is the largest (nine runners, against six KVM and five WHP), so the
remaining Linux leg lands where capacity is most available, and it still
validates its runner as MSHV. The legs are now named "Linux" and
"Windows", since neither stands for a backend any more; the dev ruleset
requires only "Required status check", so renaming them needs no
ruleset change.
The OpenVMM VMM tests keep one leg per backend, because Petri boots real
guests on each hypervisor.
Refs #430
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The change detector exposed only run-tests and run-workloads, so every pull request that touched any NVX script, guest, or workflow file ran the OpenVMM unit and VMM tests, although those suites test a pinned OpenVMM revision that the change did not alter. On WHP they are the longest jobs in the matrix: the VMM tests compile OpenVMM, its tests, pipette, and the UEFI test image for about 15 minutes before testing, and the unit tests compile for about 13. Together they hold a WHP runner for about 40 minutes per run, and the WHP pool has only five runners. Move the classification from inline bash into a tested Python classifier, `nvx.py classify-ci-changes`, and add two outputs: - run-openvmm-unit-tests: the OpenVMM pin, rust-toolchain.toml, the runner setup scripts, ci.yml, the actions that the unit-test job uses, nvx.py, and the modules that ci.py imports. - run-openvmm-vmm-tests: those inputs plus every input of the kernel and Alpine initramfs cache keys, which the VMM tests boot, and the build-guest-artifacts action that produces them. Pull requests run each suite only when one of its inputs changed, and the suite outputs imply run-tests. Pushes set both to run-tests, so every dev push keeps running the full required matrix. Of the 29 merged pull requests among the last 40 that ran tests, 11 changed no VMM-test input and 13 no unit-test input, so these skip one or both suites, along with the critical path that the WHP VMM tests set. The detector stays fail-safe: it runs the full matrix when the base or a commit is missing, when the diff or the classifier fails, or when the classifier omits a suite or reports it twice, since a missing output would otherwise skip that suite's jobs silently. It forwards only well-formed name=true|false lines to GITHUB_OUTPUT. Required status check passes the new outputs to check-required-ci, which now takes the classification as one CiChanges value and expects each OpenVMM job to succeed exactly when its suite is selected. The classifier also adds azurelinux/ to the test inputs. The Azure Linux package lock is an input of the Azure Linux initramfs that the microVM tests boot, but a change to it alone skipped those tests. Tests cover each path class for pull requests and pushes, the job summary, the CLI command, and the required-result policy for the new outputs. A bash test (POSIX only) runs the detector script with stubbed git and classifier output through every fallback, and once with the real classifier. A drift test derives each suite's inputs from the actions its jobs use, the modules that ci.py imports, and the guest cache key inputs, and fails when a pattern misses one. Refs #430 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Every hypervisor job ran the validate-nvx action: the OpenVMM unit and VMM tests, the microVM tests, and the platform benchmarks, on all three backends. That is up to 15 runs of the same backend-independent unit tests per workflow, about 37 s on a Linux runner and 50 s on a WHP runner, or 10.4 runner-minutes of self-hosted time per dev run, and each run sat on the critical path of its job. Validate the CLI once per OS instead, in the GitHub-hosted nvx-cli-linux and nvx-cli-windows jobs, and remove the action from the hypervisor jobs and the two reusable workflows. The jobs start with the workflow, since they do not depend on change detection, and keep the coverage of the runs they replace: - They check out OpenVMM first, because a test compares NVX's CPU profile catalog with the profiles of the pinned submodule revision and skips without it. - They use Python 3.12, the release that the self-hosted runners provide. - The Windows job installs the Rust release that rust-toolchain.toml pins, with the minimal profile, for the test that compiles the Windows curl shim; rustup would otherwise install the default profile on first use. - Like the hypervisor jobs, they run only for pushes and same-repository pull requests, because fork pull requests lack the deploy key that the OpenVMM checkout uses. The microVM and platform jobs of each OS now wait for that OS's CLI job, so a broken CLI no longer occupies the hypervisor runners. They already wait for their OpenVMM producer, which a hosted CLI job usually outlasts. The OpenVMM unit and VMM tests do not wait: they start right after change detection and set the critical path. Required status check, the development release, and performance persistence require both CLI jobs to succeed. With validate-nvx gone from every test job, classification no longer treats changes to it as test inputs. The CI documentation now describes the job graph, records the runner pools (five WHP, six KVM, and nine MSHV runners) in place of the stale "three runners per backend", and explains why the WHP pool bounds the time of overlapping runs. Refs #430 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Dev pushes remain path-classified instead of running the full matrix required by the stated acceptance criteria.
1 open finding
What changed in this PR
This PR restructures NVX CI to reduce duplicated validation and conserve self-hosted runner capacity.
Changes:
- Adds per-suite change classification for OpenVMM tests.
- Runs unit and CLI validation once per OS.
- Updates CI policy tests and documentation.
| File | Description |
|---|---|
.github/workflows/ci.yml |
Revises job topology, classification, and gating. |
.github/workflows/run-nvx-microvm-tests.yml |
Removes duplicated CLI validation. |
.github/workflows/run-platform.yml |
Removes duplicated CLI validation. |
scripts/nvx_tools/ci.py |
Implements suite classification and result policy. |
scripts/nvx.py |
Exposes classification and expanded CI checks. |
scripts/test_nvx_tools.py |
Tests classification, topology, and fallback behavior. |
doc/ci.md |
Documents the new graph and runner capacity. |
doc/usage.md |
Documents the new CLI interface. |
doc/benchmarks.md |
Documents CLI gating for persistence and releases. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-ci-turnaround-time-issue-430
branch
October 8, 2026 13:28
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #430. This PR targets the two constraints that the investigation in #430 identified: build-heavy OpenVMM jobs on the critical path, and the scarce WHP pool. It frees self-hosted capacity on every run, and it takes the WHP OpenVMM jobs off the critical path whenever a pull request cannot affect them. It keeps every required suite, backend, guest, and gate. It doesn't narrow
devpushes: they keep the classification they had before this PR, so both OpenVMM suites still run on every push that changes a test input. Only pull requests use the per-suite outputs. (As before this PR, a documentation-only push still skips the hypervisor matrix.)Three commits:
#[ignore = "requires /dev/kvm"]or#[ignore = "requires /dev/mshv"],test-openvmm-unitnever passes--run-ignored, and no other test probes the device. In run 37635396339, both legs ran the same 4,820 tests across 414 binaries, with the same 28 skipped. One leg now runs on the MSHV pool, the largest one, and one on WHP.nvx.py classify-ci-changes, replaces the inline bash detector and addsrun-openvmm-unit-testsandrun-openvmm-vmm-testsoutputs. A pull request runs a suite only when it changes one of that suite's inputs: the OpenVMM pin, the Rust pin, runner setup,ci.yml, the job's actions,nvx.py, and the modules thatci.pyimports. The VMM tests also depend on every input of the kernel and Alpine initramfs cache keys, since they boot those guests. Pushes keep both suites onrun-tests, as before this PR. The detector runs the full matrix whenever it can't classify a change. That includes a missing base or commit, a failed diff or classifier, and a missing or duplicated output. This commit also fixes a gap: a change only toazurelinux/packages.lock.jsonused to skip the microVM tests that boot that guest.validate-nvxran in up to 15 hypervisor jobs per workflow: about 37 s per Linux job and 50 s per WHP job, adding up to 10.4 self-hosted runner-minutes perdevrun. It now runs innvx-cli-linuxandnvx-cli-windowson GitHub-hosted runners, after the OpenVMM checkout that a CPU-profile test needs. Those jobs use Python 3.12 (the runners' release) and, on Windows, the pinned Rust release with the minimal profile. The microVM and platform jobs of each OS wait for that OS's CLI result, as the issue asks. The OpenVMM unit and VMM tests don't wait, because they set the critical path. Required status, release, and performance persistence require both CLI jobs.doc/ci.mdnow describes the job graph and change classification. It records the runner pools (5 WHP, 6 KVM, 9 MSHV) in place of the stale "three runners per backend".Evidence from the current pipeline
Measured over 29 recent full-matrix runs (job API data):
The WHP pool (5 runners) carries about 71 runner-minutes per run, against about 45 for KVM (6 runners) and 40 for MSHV (9 runners). Runs that overlap other PRs took 40 to 45 minutes, mostly waiting for WHP runners: queue times of 10 to 20 minutes per WHP job.
Expected impact
ci.yml, so it runs the full matrix. It should show the per-job savings, but not the classification win.Measured results
Run 37768232087 on head
79966a6ce, against pre-change full-matrix runs without contention (37637843120, 37508330384, 37527959862, 37635396339, 37503416598):devpushValidation
scripts/test_nvx_tools.py(430 tests) on Windows. New tests cover:ci.pyimport graph, and the guest cache keys.validate-nvxcommand set on Windows and on Linux (WSL), including the gcc-based guest tests.ruff check,ruff format --check, andpyright(strict) for Linux and Windows;actionlinton the four changed workflows.Not in this PR (remaining #430 work)
vmm-tests-run --build-onlywithvmm-tests-run-target. Earlier target-directory caching was deliberately replaced by sccache, so this needs bundle-size measurements first.Running ...lines through buffered stdout, so per-scenario durations can't be recovered from logs yet.Cache size 10 GiB / Max cache size 10 GiB, so sccache is full and evicting. That likely explains the 74 to 80% WHP hit rate, against 97 to 99.7% on Linux. RaisingSCCACHE_CACHE_SIZEis a provisioning change (andvalidate-runnerchecks the value), so it's left for a follow-up once disk headroom is known.