[Pedro Henrique Penna (ppenna)]

Fixes #119.

Summary

Windows/WHP lifecycle acceptance has split snapshot generation into fast and slow clusters, isolated to capture.mapped_memory_flush (runs 35382585623 and 35393916034). #118 made such a series remeasure once, and #203 moved benchmark scratch to a data disk, but the next split would still be hard to explain. This change makes every split self-describing:

• Attempt identification. performance validate-openvmm --attempt-log appends each bounded attempt's file, SHA-256 digest, outcome (valid, unstable, invalid), exit status, and error to acceptance-attempts.json. performance collect --lifecycle-attempt-log fails before writing a CSV unless acceptance.json is a byte-identical copy of exactly one valid attempt, prints the attempt that supplied the published metrics, and lists every attempt in the step summary. Both acceptance steps clear a stale log and stale attempt files first.
• Host provenance. A new nvx_tools.host_telemetry module records host in lifecycle JSON results and benchmark-metadata.json: runner and machine names, OS, CPU model, logical processors, memory, and the volume, file system, size, disk number, and SCSI address behind the workspace, temporary, and scratch directories. On a possible Azure VM it queries instance metadata once, without a proxy, and keeps only the VM size and managed-disk SKU, size, caching, write-accelerator, ephemeral, LUN, and Ultra Disk limit fields. It never records names, resource or VM IDs, the subscription, tags, addresses, or credentials. Summaries add a benchmark host table.
• Storage telemetry (Windows). With --snapshot-profile, each retained capture samples volume IOCTL_DISK_PERFORMANCE, GetSystemTimes, and SystemPerformanceInformation about every 50 ms, plus Defender (MsMpEng.exe, MpDefenderCoreService.exe) CPU and I/O from SystemProcessInformation about every 500 ms. These work for the Network Service account that runs CI jobs, which cannot read performance counters. Windows keeps disk counters on demand, so the sampler holds the disk-performance WMI data block (query access for every account) open while it samples. Each capture's capture.mapped_memory_flush and capture.snapshot_generation windows are mapped onto the observer clock and summarized: throughput, IOPS, latency, queue length, busy time, CPU busy time, lazy-writer and flush rates, dirty cache, and Defender activity. Each counter group spans its own samples nearest to the phase boundaries, so one failed read cannot blank a group; a group whose samples differ from the phase's reports its own interval. The job log prints each sample's flush with its host activity; the step summary tabulates every attempt, rejected ones included. Failed reads are recorded and never fail a benchmark. Linux captures flush no file-backed RAM image and record none.
• Policy. doc/benchmarks.md explains how to tell disk throttling, scanning, writeback, and unrelated host load apart, keeps runner identity diagnostic-only, and requires any runner storage change to apply to every pooled runner together with a history reset of the affected metrics.

Validation

Local (Windows 11, non-elevated):

• SystemProcessInformation offsets match GetProcessTimes and GetProcessIoCounters exactly; available pages match GlobalMemoryStatusEx; a child process's mapped-view flush moved the flush counters by the bytes the volume wrote.
• ruff check, ruff format --check, compileall, and Pyright (Linux and Windows) report no findings. The validate-nvx unit tests (932), adversarial (70) and host-inventory (4) tests, help smoke checks, and git diff --check pass.

Bare metal, with each CI step run verbatim from this revision's run-benchmark action using CI inputs (--cpus 0-7, reserve 0, timeout 40, 1 warmup + 10 runs):

Host	Result
Linux/MSHV (Azure Linux 3.0, Xeon Silver 4114)	554 unit tests pass, including the bash acceptance-step test. Acceptance recorded attempt 1 as valid; collect named it for all 29 metrics; the summary lists the attempt and host.
Linux/KVM (Ubuntu, Xeon Silver 4114)	Same: 554 unit tests pass; one valid attempt; collect names attempt 1 of 1.
Windows Server 2025/WHP (Xeon Silver 4114)	Acceptance rejected attempt 1 as unstable (a 2/8 split between 488 and 725 ms, 48.4% gap, on an otherwise idle host), recorded it, and remeasured; collect named attempt 2 of 2 and tabulated both attempts' telemetry.

The Windows run exposed that Windows Server keeps disk counters on demand: the volume IOCTL_DISK_PERFORMANCE failed (ERROR_FILE_NOT_FOUND) until something enabled collection. The sampler now holds the disk-performance WMI block open (its descriptor grants query access to Everyone and full access to Network Service), and the Windows counter test passes on that host.

A rerun overlapped another session's cargo build --release -p openvmm, and the telemetry captured it: during flushes, CPU busy time rose to 7–70% (1–2% idle), dirty cache to 205–450 MiB (0.1 MiB idle), lazy writes to 38 MiB/s, and write latency to 30–240 ms, as flushes grew from 477 to 1509 ms.

Overhead: 14 interleaved e2e runs (7 pairs, both orders) on the otherwise idle Windows host gave a snapshot-generation p50 median of 527 ms with telemetry and 522 ms without; cold start (476 ms) and restore (94 ms) matched.

CI Azure runners (Standard_D8s_v5, jobs run as Network Service): in runs 38067120152 (azure-windows-3) and 38068806476 (azure-windows-5), every counter group was recorded without errors, the IMDS provenance kept only the sanitized VM size and disk fields, and collect named attempt 1 of 1. In all 20 captures, the 512 GiB Premium_LRS (P20) scratch disk wrote 148–183 MiB/s during capture.mapped_memory_flush at 93–100% busy (average write latency 66–174 ms, queue 1.0–1.8), while CPU busy time stayed at 0–7%, the dirty cache stayed under 2 MiB, and Defender used at most 9.5% of one CPU. That rate is close to the P20 tier's documented 150 MB/s provisioned and 170 MB/s credit-based burst throughput, so the flush appears bound by the scratch disk rather than by host CPU, writeback, or scanning. A faster disk tier would be a storage change that falls under the history-reset policy above.

The storage migration in #203 has aged out of the current 10-point Windows windows (all ~0.95–1.11 s), so no history reset is needed for it now.

[Copilot]

🟡 Changes recommended

A transient counter-read failure can suppress otherwise available phase telemetry because endpoint samples are not selected per counter group.

1 open finding

• Select phase endpoints per counter group to preserve diagnostics · New

What changed in this PR

Adds benchmark-attempt tracking, safe host provenance, and Windows storage telemetry to diagnose lifecycle performance instability.

Changes:

• Preserves and identifies lifecycle acceptance attempts.
• Records host/storage provenance and capture telemetry.
• Expands CI validation, summaries, tests, and documentation.

File	Description
.github/​actions/​run-benchmark/​action.yml	Retains attempts and publishes telemetry.
.github/​actions/​validate-nvx/​action.yml	Runs telemetry tests.
.github/​skills/​nvx-benchmark/​SKILL.md	Adds provenance verification.
doc/​benchmarks.md	Documents telemetry and policy.
doc/​usage.md	Documents new CLI options.
scripts/​nvx_tools/​benchmark.py	Collects provenance and telemetry.
scripts/​nvx_tools/​host_telemetry.py	Implements host and storage monitoring.
scripts/​nvx_tools/​performance.py	Tracks attempts and produces summaries.
scripts/​setup/​README.md	Documents storage-change policy.
scripts/​test_host_telemetry.py	Tests telemetry implementation.
scripts/​test_nvx_tools.py	Tests benchmark and CI integration.
scripts/​test_performance.py	Tests attempt collection and reporting.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Low-level Windows counter parsing, timing correlation, and CI publication behavior warrant final human review despite extensive coverage.

0 open findings

1 resolved since last review

• Select phase endpoints per counter group to preserve diagnostics

🧠 Review effort: Balanced