Repository navigation
microvm tests: stream OpenVMM transcripts to disk as they are read - #659
Merged
Pedro Henrique Penna (ppenna) merged 5 commits intoOct 21, 2026
Merged
Pedro Henrique Penna (ppenna) merged 5 commits into
Pedro Henrique Penna (ppenna) merged 5 commits into
Conversation
OpenvmmProcess kept the transcript in memory and wrote it only from wait or close, so a run terminated before cleanup left no transcript, or a truncated one, even though output had been read. Open the transcript before OpenVMM starts, append each consumed chunk and flush it, and close the file at wait or close. Bytes are written in the order received, with no prefixes and no duplication at cleanup. An error creating the transcript stops the run before OpenVMM starts. Output still queued at close is written too. The in-memory output and marker search are unchanged. Adds tests for growth while the process runs, a transcript that cannot be created, and output delivered after the last wait. Fixes #176 Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>
… release the transcript close drained the reader's queue before the interaction had stopped, so a final chunk queued during shutdown was lost from both output and transcript. A write failure during that drain also skipped closing the interaction and the transcript. Stop the interaction first, then drain, and release the transcript in a finally block. Adds regression tests for output queued during interaction close and for a transcript write failure. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>
If InteractiveProcess.close raised, the drain was skipped and chunks still queued were missing from both output and the transcript. Drain in a finally around the interaction shutdown, so the transcript is complete whichever step fails. Adds a regression test with queued output and a failing interaction shutdown. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>
_drain_available stopped at the first failed transcript write, so chunks still queued behind it never reached process.output. Keep collecting them in memory and re-raise the first error once the queue is empty. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Signed-off-by: jay23606 <4602056+jay23606@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
Static review identified no blocking issues; runtime tests were not executed.
No open issues 🎉
What changed in this PR
Updates OpenvmmProcess to persist transcripts incrementally, preserving diagnostics if a run ends before normal cleanup.
Changes:
- Opens the transcript before launch and flushes each consumed chunk.
- Drains queued output during cleanup and releases resources on write failures.
- Adds regression tests for transcript growth and cleanup failures.
| File | Description |
|---|---|
| scripts/test_microvm_tests.py | Tests incremental capture, delayed output, and failure handling. |
| scripts/nvx_tools/openvmm_process.py | Streams transcript bytes and strengthens cleanup handling. |
Review effort: Balanced
The signature of the test that keeps every queued chunk when the transcript write fails reached 90 columns, past the 88-column limit, so ruff format --check failed the Lint, typing, and formatting job. Wrap it as ruff format does, like the file's other long signatures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
deleted the
ppenna/stream-openvmm-transcript-20261020
branch
October 21, 2026 12:31
This was referenced Oct 21, 2026
Pedro Henrique Penna (ppenna)
added a commit
that referenced
this pull request
Oct 21, 2026
The Alpine, Ubuntu, Ubuntu distro, and Azure Linux cache keys hashed all of nvx-rs but its target directory, and the toolchain pin, and the OpenVMM VMM tests took every nvx-rs path as an input, because they boot the Alpine initramfs. The images carry only the crate's Rust guest binaries, though, so a change to its host side or tests rebuilt every guest image and ran the VMM tests, the longest WHP jobs. Of the 42 dev merges through #659, the VMM tests ran on 33, 26 of them only because nvx-rs changed, and each of the 27 that changed nvx-rs changed every guest image key. A cold guest-artifact job takes a median 5.3 minutes instead of 0.4, and every job that boots a guest waits for it. The keys already end with the guest binaries' source identity, which now covers exactly the files that their build reads, and which every guest build checks against Cargo's dep-info. Drop nvx-rs and rust-toolchain.toml from the hashFiles of the image keys, so that the identity alone covers them, and select the VMM tests by the same pattern, GuestRustBuildConstants.SOURCE_PATTERN, instead of by every nvx-rs path. run-tests still covers all of nvx-rs, because the NVX microVM tests run the nvx-rs CLI. Replaying the classifier over those 42 merges, 11 of the 26 no longer run the VMM tests, and 12 of the 27 keep every guest image key: they changed only the host side, protocol tests, integration tests, or the loopback harness. The other 15 changed files that the guest binaries compile, such as guest and protocol modules, lib.rs, Cargo.toml, and Cargo.lock, and still run both. The drift guard that derives the VMM tests' inputs from the guest cache keys now also takes every file of the identity, and checks that the Alpine key ends with it. A new test checks every file under nvx-rs: those in the identity select the VMM tests, the others neither select them nor match a hashFiles pattern of an image key, and all of them select the microVM tests. New modules count unless they belong to the host side or the tests. For changes to the host side, the CLI and fake binaries, protocol tests, the loopback harness, integration tests, and artifacts.json, it checks that pushes and pull requests run the microVM tests but skip the VMM tests, and that the identity, and with it every image key, stays as it is. For changes to the manifest, lockfile, build script, lib.rs, protocol, a guest module, and a guest binary, it checks that they run both and change the identity. doc/ci.md, doc/usage.md, and the nvx-rs README describe the inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Oct 21, 2026
Pedro Henrique Penna (ppenna)
added a commit
that referenced
this pull request
Oct 23, 2026
The VMM tests run on most dev merges, but what determines their build rarely changes: replaying the change classifier over the 42 dev merges through #659, the tests would have run on 33, and the OpenVMM pin changed in none. Each run still built them: in run 39954081539, the build took 330 s of the Linux producer's 6.2 minutes and 858 s of the Windows producer's 16.2, which bounds the WHP lane (#692). Cache each producer's test content under the key that openvmm-test-cache-key computes, and restore it instead of building it: - A producer computes the key from the gitlink before it checks out OpenVMM and, on pushes and pull requests, restores the newest entry of the key into the directory that it uploads. reuse-openvmm-tests keeps the content only if it is exactly what a fresh build of the key makes, and removes what it refuses, with a warning, or what a partial restore left. The restore continues on error, so that a failed restore falls back to the build as a miss does. - On a hit, the producer skips the OpenVMM checkout, the retrying curl, the OpenVMM input and Cargo caches, sccache, and the build, and uploads the restored content as the same-run artifact that the test jobs download as before. The test jobs never read the cache. - After a build, check-openvmm-tests fails the producer unless the build passes the same check, so that neither the cache nor the test jobs get content that a reuse would refuse. - Only dev pushes save the content, as they save the other OpenVMM caches, and only after a build: under the key followed by the run ID and attempt, so that each build is an entry of its own and the restore takes the newest. An entry that a later reuse refuses then cannot block a newer build, as it would under an exact key, which a save never replaces. The OpenVMM input and Cargo caches also save only after a build, since a hit restores neither. - Scheduled and manually dispatched full runs neither restore nor save the content, so the nightly run still builds the tests every day and catches a breakage of the build that the key does not cover, such as a toolchain change on the runners. A miss, an eviction, a failed restore, or a refused entry therefore only costs the build. A test checks each producer's steps, conditions, keys, paths, and order; the full-run test counts the two new cache saves and the two restores that name the push event; and the guest artifact test registers the new commands as booting no guest. doc/ci.md describes the cache, its key, and its effect on the WHP runners, and doc/usage.md links to it. OpenVMM is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
that referenced
this pull request
Oct 23, 2026
The VMM tests run on most dev merges, but what determines their build rarely changes: replaying the change classifier over the 42 dev merges through #659, the tests would have run on 33, and the OpenVMM pin changed in none. Each run still built them: in run 39954081539, the build took 330 s of the Linux producer's 6.2 minutes and 858 s of the Windows producer's 16.2, which bounds the WHP lane (#692). Cache each producer's test content under the key that openvmm-test-cache-key computes, and restore it instead of building it: - A producer computes the key from the gitlink before it checks out OpenVMM and, on pushes and pull requests, restores the newest entry of the key into the directory that it uploads. reuse-openvmm-tests keeps the content only if it is exactly what a fresh build of the key makes, and removes what it refuses, with a warning, or what a partial restore left. The restore continues on error, so that a failed restore falls back to the build as a miss does. - On a hit, the producer skips the OpenVMM checkout, the retrying curl, the OpenVMM input and Cargo caches, sccache, and the build, and uploads the restored content as the same-run artifact that the test jobs download as before. The test jobs never read the cache. - After a build, check-openvmm-tests fails the producer unless the build passes the same check, so that neither the cache nor the test jobs get content that a reuse would refuse. - Only dev pushes save the content, as they save the other OpenVMM caches, and only after a build: under the key followed by the run ID and attempt, so that each build is an entry of its own and the restore takes the newest. An entry that a later reuse refuses then cannot block a newer build, as it would under an exact key, which a save never replaces. The OpenVMM input and Cargo caches also save only after a build, since a hit restores neither. - Scheduled and manually dispatched full runs neither restore nor save the content, so the nightly run still builds the tests every day and catches a breakage of the build that the key does not cover, such as a toolchain change on the runners. A miss, an eviction, a failed restore, or a refused entry therefore only costs the build. A test checks each producer's steps, conditions, keys, paths, and order; the full-run test counts the two new cache saves and the two restores that name the push event; and the guest artifact test registers the new commands as booting no guest. doc/ci.md describes the cache, its key, and its effect on the WHP runners, and doc/usage.md links to it. OpenVMM is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #588 by Jay (@jay23606), whose commits are carried here unchanged (head
2634167ff). It is opened from amicrosoft/nvxbranch so that CI runs on a stable head: automated rebases of the original kept superseding its runs.Fixes #176.
Summary
OpenvmmProcesskept the transcript in memory and wrote it only fromwaitorclose. A run that was terminated before cleanup left no transcript, or a truncated one, even though its output had been read.waitandcloseonly close the file. Bytes are written in the order received, with no prefixes and no rewrite at cleanup, so nothing is duplicated.closeis written too.OpenvmmProcessonly.TcpConsolehas no transcript path, and the benchmark helpers inbenchmark.pyare not changed. Those are follow-ups.Test
wait), a transcript that cannot be created (fails before the change), and output delivered after the last wait.BOOT, thenLATEthree seconds later. The transcript containedBOOTwhile the child was still running, and the final transcript matched the output.test_microvm_tests: 193 tests. The dev baseline (190 tests) has the same nine failures and errors in host-loopback, private network ingress and SMP timer tests. They are unrelated to the transcript code and present on untoucheddev.🤖 Generated with Claude Code