Repository navigation
ci: remeasure unstable Windows lifecycle benchmarks - #117
Merged
Pedro Henrique Penna (ppenna) merged 1 commit intoSep 18, 2026
Merged
Conversation
Classify bimodal OpenVMM lifecycle results as a retryable validation outcome without weakening the existing 25% p50-over-p25 stability guard. Validate Windows e2e results immediately and rerun the acceptance benchmark once only for that typed instability. Preserve normal failure behavior for malformed results and fail if the second measurement remains unstable. Add regression coverage from workflow run 35382585623 and document the pooled-runner behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cf6bb716-116d-4646-8228-1177e711a06d
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Instability is classified before later malformed-data checks, allowing invalid results to trigger remeasurement.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
What changed in this PR
Adds selective Windows lifecycle benchmark remeasurement for transient snapshot-generation instability.
Changes:
- Introduces validation with exit status 75 for unstable samples.
- Retries Windows lifecycle measurement once.
- Adds tests and CI behavior documentation.
| File | Description |
|---|---|
.github/actions/run-benchmark/action.yml |
Adds selective lifecycle retry logic. |
scripts/nvx_tools/performance.py |
Adds typed instability validation. |
scripts/test_performance.py |
Tests validation exit statuses. |
scripts/test_nvx_tools.py |
Checks workflow retry configuration. |
doc/usage.md |
Documents the validation command. |
doc/benchmarks.md |
Documents Windows retry behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ratio = p50 / p25 | ||
| if ratio > LIFECYCLE_SNAPSHOT_MAX_P50_OVER_P25: | ||
| raise PerformanceError( | ||
| raise UnstablePerformanceError( |
Pedro Henrique Penna (ppenna)
deleted the
copilot/fix-pooled-benchmark-retry
branch
September 18, 2026 20:34
8 tasks
1 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
Workflow run 35382585623, attempt 1, failed in
Platform / Windows / WHP / Virtual machineonazure-windows-2while publishing performance results.The snapshot-generation samples split into five normal 736-781 ms measurements and five slow 1233-1297 ms measurements. The split was isolated to
mapped_memory_flush, which alternated between 674-723 ms and 1172-1222 ms. That produced p50 1006.945 ms versus p25 753.063 ms, exceeding the deliberate 25% stability limit.This was not the 50% historical regression gate. Across the adjacent successful Windows runner artifacts inspected, stable cross-runner median differences peaked at 20.4%, while snapshot generation remained around 754-760 ms. The pooled scheduler exposed a transient WHP host-storage condition, but CI previously required a manual workflow rerun after detecting it.
Validation
python -m compileall -q scriptspython -m ruff check scriptspython -m ruff format --check scriptsgit diff --checkLocal Python 3.14 still fails the pre-existing Windows curl-shim resolution test identically on unchanged
dev; all other tests pass, and the full CI runner validation remains authoritative.