From 9d2c04b3234d8c09aa87e21d2eadb6bea3dda399 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:01:41 -0700 Subject: [PATCH 001/188] doc: specify NVX time ABI v1 KVM, MSHV, and WHP present different time ABIs to the same Linux 6.18 guest, which causes the intermittent TSC warp, clocksource watchdog, and restore failures tracked in #211, #253, #265, and #292. Add the proposed specification of one backend-independent time ABI so that the OpenVMM, guest, and CI work on the time-abi-v1 branches implements a single contract. The specification defines the minimal Hyper-V frequency and invariant-TSC identity, the CPU time bits, the backend obligations, CPU profiles, the 250 ppm TSC rate policy, the 1 us cross-vCPU skew bound, downtime source selection, the ordered restore algorithm, manifest version 6, restore packet v4 with the portb time sample, the guest conformance, watcher, snapshot-agent, and wall-clock discipline duties, stable failure codes, host qualification, the performance gate, the test matrix, and the removed mechanisms. Items that depend on the Phase 1 backend and profile spikes are marked TBD. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design.md | 25 +- doc/design/time-abi.md | 1077 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1090 insertions(+), 12 deletions(-) create mode 100644 doc/design/time-abi.md diff --git a/doc/design.md b/doc/design.md index b46d705d5..d59f8f1b6 100644 --- a/doc/design.md +++ b/doc/design.md @@ -10,15 +10,16 @@ yet implemented; they are not part of the current machine contract. 2. [Configuration boundary](design/configuration-boundary.md) 3. [Cold boot](design/cold-boot.md) 4. [Machine and device ABI](design/machine-and-device-abi.md) -5. [Sandbox filesystem and agent architecture](design/sandbox-filesystem-and-agent-architecture.md) -6. [Ubuntu guest support](guests/ubuntu-guest.md) -7. [Snapshot and restore](design/snapshot-and-restore.md) -8. [Snapshot sharing and host storage](design/snapshot-sharing-and-host-storage.md) -9. [Host attachment model](design/host-attachment-model.md) -10. [Concurrency and trust boundaries](design/concurrency-and-trust-boundaries.md) -11. [Validation](design/validation.md) -12. [Current limits](design/current-limits.md) -13. [Remaining production work](design/remaining-production-work.md) -14. [Integrated implementation status](design/integrated-implementation-status.md) -15. [Component responsibility map](design/code-ownership-map.md) -16. [Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) +5. [Time ABI](design/time-abi.md) +6. [Sandbox filesystem and agent architecture](design/sandbox-filesystem-and-agent-architecture.md) +7. [Ubuntu guest support](guests/ubuntu-guest.md) +8. [Snapshot and restore](design/snapshot-and-restore.md) +9. [Snapshot sharing and host storage](design/snapshot-sharing-and-host-storage.md) +10. [Host attachment model](design/host-attachment-model.md) +11. [Concurrency and trust boundaries](design/concurrency-and-trust-boundaries.md) +12. [Validation](design/validation.md) +13. [Current limits](design/current-limits.md) +14. [Remaining production work](design/remaining-production-work.md) +15. [Integrated implementation status](design/integrated-implementation-status.md) +16. [Component responsibility map](design/code-ownership-map.md) +17. [Copilot-driven adversarial testing](design/copilot-adversarial-testing.md) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md new file mode 100644 index 000000000..76da731f4 --- /dev/null +++ b/doc/design/time-abi.md @@ -0,0 +1,1077 @@ +# Time ABI + +[Design index](../design.md) + +**Proposed.** This document specifies NVX time ABI v1, the single +guest-visible time, timer, and clock contract of the microVM profile on KVM, +MSHV, and WHP. It is authoritative for the `time-abi-v1` integration branches +of `microsoft/nvx` and `nanvix/openvmm`. When implemented, it replaces the +time rules in [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) +and the clock tokens in [Cold boot](cold-boot.md#effective-command-line). +Entries marked `TBD()` wait for that Phase 1 spike's report. + +Notation: + +| Symbol | Meaning | +| --- | --- | +| `F` | Declared guest TSC rate in Hz, returned by MSR `0x40000022` | +| `F_s` | Declared rate recorded in a snapshot | +| `F_d` | Native TSC rate of the destination partition in Hz | +| `L` | LAPIC timer rate in Hz, returned by MSR `0x40000023` | +| `C` | VP capacity of the machine (1, 2, 4, or 8) | +| `D` | Snapshot downtime in nanoseconds | +| `T_c` | Guest TSC of VP 0 at the capture anchor | +| `g` | Restore generation counter of a VM process | + +"Reject" means fail with the named error code and never fall back to another +behavior. Every check in this document is a hard check. + +## Scope, goals, and non-goals + +Goals: + +- One time ABI on every backend: the same CPUID identity, synthetic MSRs, + CPU time bits, clocksource, tick, timers, and restore semantics. +- No guest kernel patches and no clock command-line tokens. +- No implicit fallbacks. A host or snapshot that cannot meet the contract is + rejected with a stable error code. +- Restore within one backend across hosts of the same CPU generation and CPU + profile, including restores after a host reboot. +- Guest monotonic time advances by the snapshot downtime. Wall clock is set + on every restore and disciplined afterwards. +- RCU-stall, soft-lockup, hung-task, clocksource, and TSC-warp detectors never + trip across capture and restore. A detected violation fails fast. +- No performance regression beyond the gate in + [Performance expectations and acceptance gate](#performance-expectations-and-acceptance-gate). + +Non-goals: + +- Restore across backends or across CPU generations. Both are rejected. +- TSC scaling, and RDTSC trapping outside the WHP interim described in + [Backend obligations](#backend-obligations). +- Paravirtual clocks: kvmclock, the Hyper-V reference TSC page and reference + counter, synthetic timers, VMGenID, and VMClock. +- A virtual PMU. AMD profiles are deferred until an AMD host is registered. +- Live migration, standard-machine guests, and guests other than the NVX + Linux 6.18 kernel. +- Snapshots older than manifest version 6. They are rejected with a recapture + error. + +The time ABI is versioned independently of the microVM machine ABI: +`time_abi_version` is 1 in the manifest and in CPUID leaf `0x40000002`, while +the microVM ABI version stays 2. Performance history keys its baselines by +microVM ABI version, so the time ABI does not create a new baseline series. + +## Guest-visible contract + +### Hypervisor identity + +Every VP sees a minimal Hyper-V frequency and invariant-TSC identity. OpenVMM +configures exactly these six leaves on every backend: + +| Leaf | EAX | EBX | ECX | EDX | +| --- | --- | --- | --- | --- | +| `0x40000000` | `0x40000005` (maximum leaf) | `0x7263694d` (`"Micr"`) | `0x666f736f` (`"osof"`) | `0x76482074` (`"t Hv"`) | +| `0x40000001` | `0x31237648` (`"Hv#1"`) | 0 | 0 | 0 | +| `0x40000002` | `0x0058564e` (`"NVX"`, build) | `0x00010000` (time ABI 1.0) | 0 | 0 | +| `0x40000003` | `0x00008860` | 0 | 0 | `0x00000100` | +| `0x40000004` | 0 | `0xffffffff` | 0 | 0 | +| `0x40000005` | `C` | `C` | 0 | 0 | + +`0x40000003` EAX grants exactly `AccessHypercallMsrs` (bit 5) and +`AccessVpIndex` (bit 6), which Linux requires to detect the platform, plus +`AccessFrequencyRegs` (bit 11) and `AccessTscInvariantControls` (bit 15). EDX +sets only `FrequencyRegsAvailable` (bit 8). `0x40000004` recommends nothing +and sets the spinlock retry count to "never notify". + +Every other leaf in `0x40000006..=0x400000ff` returns either all zeros or the +vendor's architectural out-of-range result (on Intel, the result of the +highest basic leaf for the same subleaf). It never returns Hyper-V feature +data, a `VS#1` interface signature at `0x40000081`, or any hypervisor +signature. No base `0x40000100..=0x4000ff00` (step `0x100`) carries a KVM, +Xen, VMware, or other hypervisor signature, so Linux selects only the Hyper-V +platform. + +### Synthetic MSRs + +OpenVMM serves the identity MSR range `0x40000000..=0x400001ff` identically +on every backend: + +| MSR | Name | Read | Write | +| --- | --- | --- | --- | +| `0x40000002` | `HV_X64_MSR_VP_INDEX` | VP index | #GP | +| `0x40000022` | `HV_X64_MSR_TSC_FREQUENCY` | `F` | #GP | +| `0x40000023` | `HV_X64_MSR_APIC_FREQUENCY` | `L` | #GP | +| `0x40000118` | `HV_X64_MSR_TSC_INVARIANT_CONTROL` | Last value written, 0 after reset | 0 or 1 accepted; any other value #GP | +| Any other MSR in the range | | #GP | #GP | + +`HV_X64_MSR_TSC_INVARIANT_CONTROL` is one partition-wide value. It is guest +state: it is saved with the VM and restored with it. Linux writes 1 at boot +with an unchecked `wrmsrq`, so this write must never fault on any backend. + +### CPU time bits + +Every CPU profile fixes these bits. A profile that violates them is rejected. + +| Leaf | Register and bits | Value | +| --- | --- | --- | +| `0x1` | ECX[31] hypervisor present | 1 | +| `0x1` | ECX[24] TSC-deadline timer | 0 | +| `0x1` | ECX[15] PDCM | 0 | +| `0x1` | EDX[4] TSC | 1 | +| `0x6` | EAX, EBX, ECX, EDX | `0x00000004` (ARAT only), 0, 0, 0 | +| `0x7.0` | EBX[1] `IA32_TSC_ADJUST` | 0 | +| `0xa` | EAX, EBX, ECX, EDX | 0 (no PMU) | +| `0x15`, `0x16` | EAX, EBX, ECX, EDX | 0, when within the maximum basic leaf | +| `0x80000001` | EDX[27] RDTSCP | 1 | +| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only) | + +Leaves `0x15` and `0x16` are zeroed rather than synthesized. Linux takes both +rates from the frequency MSRs, and zero leaves keep the profile's CPUID +independent of any host's rate. + +### Rates + +- **TSC.** The guest TSC runs at the destination's native rate `F_d` and is + never scaled. The declared rate `F` is fixed for the life of a snapshot + lineage: a cold boot declares its native rate, and every restored process + keeps `F = F_s`, including when it is captured again. The kernel computes + `tsc_khz = floor(F / 1000)` once and never recalibrates. The + [rate policy](#tsc-rate-policy-and-lapic-rate-rule) bounds + `|F_d - F| / F`. +- **LAPIC.** `L` is a per-backend constant: 1,000,000,000 Hz on KVM and + 200,000,000 Hz on MSHV and WHP. Linux sets `lapic_timer_period = L / HZ` + (`0x989680` on KVM, `0x1e8480` on MSHV and WHP, with HZ=100) and never + calibrates the LAPIC. + +### Clocksource, tick, and PIT + +With this identity, Linux 6.18 built without `CONFIG_HYPERV`: + +- reads `F` from MSR `0x40000022`, sets `X86_FEATURE_TSC_KNOWN_FREQ`, and + registers clocksource `tsc` at `device_initcall` with no refinement; +- writes 1 to MSR `0x40000118` and sets `X86_FEATURE_TSC_RELIABLE`, which + disables the clocksource watchdog for `tsc` and the CPU-online TSC warp + test; +- presets `lapic_timer_period` from MSR `0x40000023`, sets `no_timer_check`, + and disables the hard-lockup detector; and +- disables the PIT at boot because the TSC rate is known, ARAT is present, + and the LAPIC period is preset. + +The guest must then be in this state: + +- `/proc/cpuinfo` flags on every CPU include `tsc`, `constant_tsc`, + `nonstop_tsc`, `tsc_known_freq`, `tsc_reliable`, `rdtscp`, `hypervisor`, + and `arat`, and exclude `tsc_deadline_timer` and `tsc_adjust`. +- `current_clocksource` is `tsc`. `available_clocksource` contains `tsc` and + nothing other than `tsc`, `refined-jiffies`, and `jiffies`. +- Ticks are tickless-idle with high-resolution timers. Every online CPU's + tick device is `lapic` in one-shot mode from its first tick onward. No + `pit` or `hpet` clock event device and no broadcast device exist. +- The LAPIC timer is never periodic after boot and never in TSC-deadline mode. +- PIT channel 0 never counts after boot, and IRQ 0 has no timer handler. + +Because the kernel no longer checks cross-vCPU TSC consistency, the +[skew bound](#cross-vcpu-skew-bound) is enforced outside the guest. + +### Deviations from the Hyper-V TLFS + +- `0x40000003` advertises `AccessHypercallMsrs` only for platform detection. + `HV_X64_MSR_GUEST_OS_ID` (`0x40000000`) and `HV_X64_MSR_HYPERCALL` + (`0x40000001`) raise #GP, and there is no hypercall page. The NVX kernel is + built without `CONFIG_HYPERV` and never touches them. `VMCALL` behavior is + outside the contract; the guest never executes it. +- `HV_X64_MSR_TSC_FREQUENCY` returns the declared rate `F`, which differs + from the physical rate `F_d` by up to the rate tolerance after a restore. +- CPUID `0x80000007` EDX[8] is set by the CPU profile from boot. It does not + depend on writes to `HV_X64_MSR_TSC_INVARIANT_CONTROL`. +- `0x40000002` carries an NVX signature and the time ABI version, not a + Hyper-V build number. +- `0x40000005` reports the VP capacity in both EAX and EBX. + +Side effects on Linux are limited to selecting the Hyper-V platform: an +unknown-NMI handler is registered, `pv_info.name` is `Hyper-V`, the +hard-lockup detector is off, and `no_timer_check` is set. No KVM +paravirtual feature (kvmclock, PV EOI, steal time, async page faults, PV TLB +flush, or PV spinlocks) exists on any backend; `sched_clock` and the vDSO use +the raw TSC. + +## Backend obligations + +Each backend implements these mechanisms behind one OpenVMM interface. The +guest-visible result is identical; only the mechanism differs. + +| Obligation | KVM | MSHV | WHP | +| --- | --- | --- | --- | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity leaves; every KVM `0x4xxxxxxx` entry removed | CPUID intercept results (`always_override`) for `0x40000000..=0x400000ff` — TBD(mshv) | CPUID result list or CPUID exits for `0x40000000..=0x400000ff` — TBD(whp) | +| Profile CPUID and time bits | `KVM_SET_CPUID2` | Processor feature banks plus CPUID intercept results | Processor feature banks plus CPUID result list or exits — TBD(whp) | +| Identity MSRs routed to OpenVMM | `KVM_X86_SET_MSR_FILTER` denying `0x40000000..=0x400001ff`, with user-space MSR exits for filtered and unknown MSRs — TBD(kvm) confirm precedence over in-kernel Hyper-V MSRs | MSR intercept for the range, or native frequency MSRs plus an intercept for `0x40000118` only if native values match the table exactly — TBD(mshv) | Unhandled-MSR exits enabled for the offloaded APIC; no `hv1_emulator` — TBD(whp) | +| Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | +| No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | +| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | `ApicFrequency` property, 200 MHz — TBD(mshv) | Offloaded APIC only, 200 MHz; the emulated APIC is rejected — TBD(whp) | +| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; `IA32_TSC_ADJUST` raises #GP | Feature-bank bits cleared and CPUID bits cleared | Feature-bank bits cleared and CPUID bits cleared — TBD(whp) | +| Invariant TSC exposed | CPUID bit | `tsc_invariant_support` plus CPUID; hosts without it are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition — TBD(whp): CPUID override and qualification evidence | +| No paravirtual or synthetic features | No KVM leaves; no KVM Hyper-V enlightenment enabled beyond MSR routing | Synthetic processor features limited to what MSR routing needs — TBD(mshv) | `--hv` stays rejected for the microVM | +| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | VP 0 `IA32_TSC` read and host sample back to back — TBD(kvm) | TBD(mshv) | TBD(whp) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, computed from one host TSC read (Linux 5.16 or newer); no `IA32_TSC` writes | Freeze partition time, write the target to every created VP, thaw at release — TBD(mshv) | TBD(whp): suspend partition time and write the target to every VP | +| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value | Every created VP's TSC equals the target while time is frozen | TBD(whp) | +| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification | TBD(mshv) | TBD(whp): decision gate for RDTSC emulation | +| VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | +| Partition capabilities | Derived from CPUID with the identity range masked: `hv1` and `kvm_clock` are false | Same | Same | +| Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor — TBD(mshv) | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | +| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ` | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` (unless the interim applies), leaf `0x15` synthesis | + +**WHP interim.** If no WHP primitive keeps live cross-VP skew within 1 µs +after release, WHP keeps its partition-reference-time TSC model for restored +SMP partitions (RDTSC, RDTSCP, and `IA32_TSC` read exits) as a documented +interim, and the case is escalated. The guest-visible ABI is unchanged. +TBD(whp). + +## CPU profiles + +A CPU profile is the complete guest-visible CPU surface for one CPU +generation. Profiles are pinned in the OpenVMM source tree, derived by +intersecting host fingerprints (the Firecracker `cpu-template-helper` +workflow), and immutable once released: any change creates a new revision ID. + +**Format.** A profile is a canonical protobuf message: + +| Field | Content | +| --- | --- | +| `id` | `--v`, lowercase | +| `vendor` | The 12-byte CPUID vendor string | +| `generation` | Allowed `(family, model)` display signatures; stepping is unconstrained | +| `cpuid` | Every basic and extended leaf and subleaf the guest can observe, with exact register values and a mask of topology fields that OpenVMM synthesizes from the machine topology | +| `xcr0`, `xss` | Exposed XSAVE feature masks | +| `xsave_components` | Exact offset, size, and alignment of every enabled component | +| `physical_address_width` | Guest physical address width | +| `msrs` | Pinned MSR values the guest can read, such as `IA32_ARCH_CAPABILITIES` | + +The profile digest is the SHA-256 of the canonical encoding. Profile content +outside the topology mask is host-independent, so the effective guest CPUID +is a pure function of the profile, the topology, and the time ABI. + +**Selection.** A cold boot uses `--cpu-profile `, or `auto`, which maps +the host's `(vendor, family, model)` to exactly one pinned profile. An +unmapped host is rejected (`E_PROFILE_HOST_UNKNOWN`); host CPUID passthrough +does not exist. A restore always uses the profile embedded in the snapshot; +an explicit `--cpu-profile` must name the same profile. + +**Verification.** At partition creation, for cold boot and restore: + +1. The profile satisfies the [CPU time bits](#cpu-time-bits) + (`E_PROFILE_TIME_BITS`). +2. The host's `(vendor, family, model)` is in the profile's generation + (`E_CPU_GENERATION`). +3. The backend supports every feature bit the profile sets, every numeric + limit is within the backend's, the XSAVE layout matches exactly, and every + pinned MSR value can be presented (`E_PROFILE_UNSUPPORTED`, naming the + first missing leaf, register, and bit). +4. On restore only: this OpenVMM pins a profile with the same ID and digest + (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective + CPUID equals the recorded one (`E_CPU_SURFACE`). + +Initial generations are Skylake-SP (family 6, model 85: the bare-metal hosts), +Ice Lake-SP (6/106: the Xeon Platinum 8370C runners), and Emerald Rapids +(6/207: the 8573C runners). Profile IDs, contents, whether one profile serves +all backends of a generation, and the fate of the `MYSTERY_MSRS` stubs are +TBD(profiles). + +## TSC rate policy and LAPIC rate rule + +**TSC rate.** At restore, with integer arithmetic in at least 128 bits: + +```text +accept iff |F_d - F_s| * 1_000_000 <= 250 * F_s +``` + +The boundary is accepted. Beyond it, restore is rejected +(`E_TSC_RATE_TOLERANCE`), reporting both rates and the deviation. The +tolerance is recorded in the manifest as `tsc_tolerance_ppm = 250`; any other +recorded value is rejected. Within the tolerance: + +- the TSC is never scaled, and RDTSC is never trapped (except the + [WHP interim](#backend-obligations)); +- MSR `0x40000022` keeps returning `F_s`, so the guest's `tsc_khz` is + unchanged; +- guest clocks run fast or slow by at most the deviation (measured hosts + differ by at most 1 ppm), and the restore packet reports the deviation so + the guest pre-compensates it (see + [Wall-clock discipline](#wall-clock-discipline)). + +Every rate is also checked for plausibility, `500 MHz <= F <= 10 GHz` +(`E_TSC_RATE_IMPLAUSIBLE`). A backend that cannot report its native rate is +rejected at cold boot, capture, and restore alike (`E_TSC_RATE_UNAVAILABLE`); +there is no command-line or calibration fallback. + +**Rate deviation.** The restore packet carries the deviation in the unit of +`adjtimex` frequency, ppm scaled by 2^16, rounded to nearest: + +```text +rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s) +``` + +Within the tolerance its magnitude is at most 16,384,000. + +**LAPIC rate.** `L` must equal the backend constant (1 GHz on KVM, 200 MHz on +MSHV and WHP), and on restore `L_d` must equal `L_s` exactly +(`E_LAPIC_RATE_MISMATCH`). A counting-mode one-shot timer advances over a +downtime `D` by exactly + +```text +ticks = floor(D * L / 1_000_000_000 / divide) +``` + +where `divide` is the divide-configuration value (1 to 128). If `ticks` is at +least the current count, the count becomes 0 and the timer interrupt is queued +in the restored LAPIC state unless the LVT is masked; otherwise the current +count decreases by `ticks`. A periodic or TSC-deadline LAPIC timer is never +valid: capture and restore reject one that is armed (`E_LAPIC_PERIODIC`, +`E_LAPIC_TSC_DEADLINE`). + +## Cross-vCPU skew bound + +At every guest-observable instant, the TSCs of any two online VPs differ by +at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this +(`TSC_RELIABLE`), so it is enforced at four points: + +1. **VMM synchronized set.** Restore writes one target to every instantiated + VP at one host instant and verifies it by read-back before any VP runs + (`E_TSC_SYNC_READBACK`). The VMM introduces no skew of its own. +2. **Host qualification.** `nvx.py doctor` and `validate-runner` run a host + TSC skew probe and a guest warp probe; a host above 1 µs is not qualified. +3. **CI warp probe.** Conformance, restore-matrix, and soak runs execute the + guest warp probe after boot and after every restore and fail above 1 µs. +4. **Backend live skew.** Each backend shows that live skew stays within the + bound after release (TBD per backend; WHP is a decision gate). + +The guest warp probe reports, for every ordered pair of online CPUs: + +- `max_warp_ns`: the largest backward step observed when one CPU reads its + TSC after observing the other CPU's latest TSC through a shared cache line, + over at least 1,000,000 handoffs per pair; and +- `max_offset_ns`: the pairwise offset estimated by ping-pong exchanges, + taking each pair's minimum round trip. + +Both must be at most 1,000 ns. Cycle values convert to nanoseconds with `F`. + +## Downtime semantics and source selection + +Guest monotonic time advances by the downtime `D`, the host time elapsed +between the capture anchor and the restore anchor, plus at most the restore +latency `R` defined below. The guest observes no other discontinuity between +the snapshot `out` and the first restored instruction. + +Capture records, at the capture anchor: + +- `T_c`, the guest TSC of VP 0; +- host UTC in nanoseconds since the Unix epoch; +- host monotonic time in nanoseconds: `CLOCK_BOOTTIME` on Linux, or + `QueryInterruptTimePrecise` × 100 on Windows (both include host suspend); +- the host identity: `/etc/machine-id` on Linux, or the `MachineGuid` + registry value on Windows, as 16 bytes; and +- the host boot identity: `/proc/sys/kernel/random/boot_id` on Linux, or the + `PrefetchParameters\BootId` registry value on Windows (zero-extended), as + 16 bytes. + +A host that cannot provide all of these rejects capture and restore +(`E_HOST_IDENTITY`). Restore takes the same sample at the restore anchor and +selects exactly one source: + +| Condition | Source | `D` | +| --- | --- | --- | +| Same host identity, same boot identity, and same host clock kind | Host monotonic | `monotonic_now - monotonic_capture` | +| Anything else | UTC | `utc_now - utc_capture` | + +`D` must satisfy `0 <= D <= 30 days` (2,592,000 s); otherwise restore is +rejected (`E_DOWNTIME_NEGATIVE`, `E_DOWNTIME_EXCESSIVE`). On the monotonic +path, a UTC delta that differs from `D` by more than 1 s is logged as a host +wall-clock step and does not change `D`. The packet reports the source, so +the guest and tests can tell the two paths apart. + +The restore anchor is the instant of the synchronized TSC set. On KVM the +guest TSC runs from that instant; on MSHV and WHP it is frozen until the +first VP runs. Either way, the guest monotonic advance across a restore lies +in `[D, D + R]`, where `R` is the time from the restore anchor to the first +restored instruction. Wall-clock repair absorbs `R` for `CLOCK_REALTIME`. + +## Restore algorithm + +Restore runs these steps in order. Any failure rejects the restore before a +restored VP runs, with the named code. + +Before worker construction: + +1. Read the bounded manifest. Its version must be 6 with format magic + `OPENVMM_SNAPSHOT_V6\0` (`E_SNAPSHOT_VERSION`, which asks for a + recapture). +2. Validate the time contract and the CPU profile record: `time_abi_version` + is 1, `tsc_tolerance_ppm` is 250, `F_s` is plausible, `L_s` is a backend + constant, identities are 16 bytes, `capture_generation < 2^32 - 1`, and + the embedded profile and effective-CPUID digests verify + (`E_MANIFEST_TIME`, `E_PROFILE_DIGEST`). +3. Require the destination backend to equal `source_hypervisor` + (`E_BACKEND_MISMATCH`). +4. Require a pinned profile with the recorded ID and digest + (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the host's CPU generation + to be in it (`E_CPU_GENERATION`). +5. Preflight the downtime: sample the host clocks, select the source, and + check the bounds (`E_DOWNTIME_*`, `E_HOST_IDENTITY`). The authoritative + `D` is measured again in step 12. +6. Validate the rest of the machine contract as today (topology, devices, + attachments, command line, blocks). The command line must not contain + `tsc_early_khz=` or `lapic_timer_hz=` (`E_CMDLINE_CLOCK_TOKEN`). +7. Generate the restore entropy and generation ID, and set + `g = capture_generation + 1`. + +In the worker, with every VP stopped: + +8. Create the partition with the profile CPUID, the identity leaves, and the + time platform declaring `F = F_s` and `L = L_s`. Preflight the backend: + profile support and effective CPUID (`E_PROFILE_UNSUPPORTED`, + `E_CPU_SURFACE`), identity routing (`E_IDENTITY_ROUTING`), the + synchronized-set primitive (`E_TSC_SYNC_UNSUPPORTED`), and no scaling + (`E_TSC_SCALING_ACTIVE`). +9. Read `F_d` and `L_d` and apply the + [rate policy](#tsc-rate-policy-and-lapic-rate-rule) + (`E_TSC_RATE_UNAVAILABLE`, `E_TSC_RATE_IMPLAUSIBLE`, + `E_TSC_RATE_TOLERANCE`, `E_LAPIC_RATE_UNAVAILABLE`, + `E_LAPIC_RATE_MISMATCH`). +10. Restore every state unit while stopped: VM time, chipset and virtio + devices, the time platform (`HV_X64_MSR_TSC_INVARIANT_CONTROL`), the + partition, and every VP. +11. Assert the saved timers: no armed periodic or TSC-deadline LAPIC timer on + any VP (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`), and PIT channel 0 not + counting in a periodic mode (`E_PIT_ACTIVE`). Freeze the instantiated VP + set; creating a VP after this point is an internal error + (`E_VP_LATE_CREATION`). +12. Synchronized TSC set. The backend takes the restore anchor (one host + instant) and passes its host sample to the orchestrator, which selects + the downtime source, computes `D` and checks its bounds, and returns + + ```text + TSC_target = T_c + floor(D * F_s / 1_000_000_000) + ``` + + (`E_TSC_TARGET_OVERFLOW` if it exceeds 64 bits). The backend writes + `TSC_target` to every instantiated VP at that instant. Per-VP TSC values + from saved VP state are superseded and never applied afterwards. +13. Read back: every instantiated VP holds the synchronized value as defined + by its backend (`E_TSC_SYNC_READBACK`). +14. Advance every VP's counting-mode LAPIC timer by `D` at `L`. +15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT + catches up from its saved VM-time cursor; it is idle. +16. Seal the time fields of the restore packet: `D`, its source, the rate + deviation, `g`, and the test-hook flag. +17. Start the state units, with host input gated when the restore requires an + acknowledgement; publish the readiness event; release the VPs. On MSHV + and WHP, partition time thaws when the first VP runs. + +The guest then repairs its clocks (see +[Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. + +## Snapshot manifest + +New snapshots use manifest version 6 with format magic +`OPENVMM_SNAPSHOT_V6\0`. Restore accepts exactly version 6; versions 2 +through 5 and every other value are rejected with `E_SNAPSHOT_VERSION`, whose +message tells the operator to recapture the snapshot. The legacy version +branches and their format constants are deleted. + +**Capture.** After every VP has stopped at the snapshot boundary and the +state units are quiesced, capture: + +1. asserts that no LAPIC timer is periodic or in TSC-deadline mode and that + PIT channel 0 is not counting periodically (`E_LAPIC_PERIODIC`, + `E_LAPIC_TSC_DEADLINE`, `E_PIT_ACTIVE`); +2. takes the capture anchor: VP 0's TSC paired with a host time sample taken + within 10 µs of it (`E_TSC_ANCHOR`), plus the host identities + (`E_HOST_IDENTITY`); and +3. records the declared rates, the CPU profile, the effective CPUID, and the + process's generation counter. + +Each of these failures is a rollback-safe capture failure. + +**Machine contract changes.** `SnapshotMachineContract` retires protobuf +fields 11 (`capture_wall_clock`), 12 (`tsc_frequency_hz`), 13 +(`tsc_tolerance_ppm`), 14 (`cpu_contract`), 15 (`cpu_contract_sha256`), 17 +(`clock_policy`), and 20 (`apic_frequency_hz`); their numbers are never +reused. It adds two required fields: + +| Field | Number | Type | +| --- | --- | --- | +| `time` | 31 | `SnapshotTimeContract` | +| `cpu_profile` | 32 | `SnapshotCpuProfile` | + +`SnapshotTimeContract` (package `openvmm.snapshot`): + +| Number | Field | Type | Content | +| --- | --- | --- | --- | +| 1 | `time_abi_version` | `u32` | 1 | +| 2 | `tsc_frequency_hz` | `u64` | `F_s`, the declared rate | +| 3 | `tsc_tolerance_ppm` | `u32` | 250 | +| 4 | `apic_frequency_hz` | `u64` | `L_s` | +| 5 | `capture_tsc` | `u64` | `T_c` | +| 6 | `capture_utc_ns` | `u64` | Host UTC at the capture anchor, nanoseconds since the Unix epoch | +| 7 | `capture_monotonic_ns` | `u64` | Host monotonic time at the capture anchor | +| 8 | `host_clock` | `string` | `linux-boottime` or `windows-interrupt-time` | +| 9 | `host_id` | `bytes` | 16-byte host identity | +| 10 | `host_boot_id` | `bytes` | 16-byte host boot identity | +| 11 | `capture_generation` | `u32` | `g` of the captured process | + +`SnapshotCpuProfile` (package `openvmm.snapshot`): + +| Number | Field | Type | Content | +| --- | --- | --- | --- | +| 1 | `id` | `string` | Profile ID | +| 2 | `sha256` | `bytes` | Profile digest, 32 bytes | +| 3 | `profile` | `bytes` | Canonical profile encoding, at most 1 MiB | +| 4 | `effective_cpuid` | `bytes` | Canonical encoding of the effective guest CPUID: profile, topology, and identity leaves | +| 5 | `effective_cpuid_sha256` | `bytes` | Digest of `effective_cpuid`, 32 bytes | +| 6 | `capture_cpu_signature` | `u32` | CPUID.1:EAX of the capture host, for diagnostics | + +Other rules: + +- The effective command line contains no `tsc_early_khz=` or + `lapic_timer_hz=` token, and platform-tier validation no longer requires + one (`E_CMDLINE_CLOCK_TOKEN`). +- The state-unit inventory gains the time platform unit, `time-abi`, which + saves `HV_X64_MSR_TSC_INVARIANT_CONTROL`. +- `source_hypervisor` keeps its meaning and must match exactly. +- Nothing in the manifest depends on the capture host's native TSC rate + except `F_s`, which is the lineage's declared rate. + +## Restore packet v4 and the time-sample selector + +Every byte the guest reads from portb is a port exit, so the time ABI keeps +records small and lets the guest read four bytes per exit. + +### Portb registers + +| Port | Access | Time ABI v1 behavior | +| ---: | --- | --- | +| `0xe9` | Read | With the restore packet or generation ID selected, a 1-, 2-, or 4-byte read returns that many bytes of the selected record, little-endian, zero-filled past its end. Console input reads are unchanged: one byte per read. | +| `0xea` | Read | Status bits 0 to 5 are unchanged. Bit 6 (`0x40`) is always set: the time-sample window exists. Bit 7 is zero. | +| `0xea` | Write | `0xa5` selects the restore packet and `0xa6` the generation ID, as today. `0xa7` latches a fresh time sample into the window at `0xeb` and does not change the `0xe9` selection. | +| `0xeb` | Read | A 1-, 2-, or 4-byte read returns the next bytes of the latched time sample, zero-filled past its end and when no sample is latched. | + +Port `0xeb` belongs to the portb device, whose range becomes +`0xe9..=0xeb`. The time window is separate from `0xe9` because the console +driver polls `0xea` and reads `0xe9` from a timer thread; a sample on the +console stream could interleave with input bytes. The window is not saved +state: restore empties it. Guest users of selectors `0xa5`, `0xa6`, and +`0xa7` serialize their transactions with an exclusive `flock` on +`/run/nvx/portb.lock`. + +### Restore packet + +OpenVMM exposes a restore packet on every microVM restore, for every tier and +for untiered snapshots; status bit 1 advertises it. Version 4 replaces +versions 1 through 3, which are no longer produced or accepted. + +```text +offset size field + 0 3 magic "OVR" + 3 1 version = 4 + 4 1 flags + 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 + 6 1 memory_range_count 0 unless MEMORY_TARGET + 7 1 reserved = 0 + 8 4 generation u32, little-endian, g >= 1 + 12 4 rate_deviation i32, little-endian, ppm scaled by 2^16 + 16 8 downtime_ns u64, little-endian, D + 24 8 utc_ns u64, little-endian, host UTC at selection + 32 16*n ranges n x { u64 gpa_start, u64 length }, LE + 32+16n 64 entropy +``` + +Flags: + +| Bit | Name | Meaning | +| ---: | --- | --- | +| 0 | `DOWNTIME_UTC` | `D` came from the UTC delta; clear means same-boot host monotonic time | +| 1 | `MEMORY_TARGET` | An explicit RAM target was requested; `memory_range_count` is valid and may be 0 | +| 2 | `ACK_REQUIRED` | Host input is gated; the guest must acknowledge through `0x605` after repair | +| 3 | `TEST_HOOKS` | A test hook altered the downtime source, `D`, UTC, or the rate deviation | +| 4–7 | | Zero; the guest rejects a packet with any of them set | + +`utc_ns` is latched when the guest first writes selector `0xa5` after the +restore; every other field is sealed before the first restored VP runs (step +16 of the [restore algorithm](#restore-algorithm)). A packet without ranges is +96 bytes, 24 four-byte reads. Status bits 2 to 4 remain and agree with the +packet; the packet is authoritative. `ACK_REQUIRED` replaces the guest's +inference of gating from the tier and targets, so an untiered guest never +writes `0x605` after an ungated restore. + +### Time sample + +Writing `0xa7` to `0xea` latches a 16-byte sample into the window at `0xeb`: + +```text +offset size field + 0 1 version = 1 + 1 1 flags bit 3 TEST_HOOKS; other bits zero + 2 2 reserved = 0 + 4 4 generation u32, little-endian, g of this VM process + 8 8 utc_ns u64, little-endian, host UTC at the latch +``` + +The guest pairs `utc_ns` with its own clock: it reads `CLOCK_REALTIME` as +`t0` immediately before the selector write and as `t1` immediately after it. +The host instant lies in `[t0, t1]`, so the offset is + +```text +theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 +``` + +The same bracket applies to `utc_ns` in the restore packet, around the first +`0xa5` write. A sample costs one write and four reads. OpenVMM cannot observe +a running VP's TSC on every backend, and frozen time on MSHV and WHP thaws at +an unobservable instant, so the guest forms the UTC and TSC pair itself +through this bracket; its `CLOCK_REALTIME` is derived from the TSC. + +### Generation counter and generation ID + +- The generation counter `g` is 0 in a cold-booted process and + `capture_generation + 1` in a restored one. It orders restores within a + snapshot lineage; clones of one snapshot share it. The packet and every + time sample carry it, so a guest observes a restore between two samples as + a change of `g`. +- The generation ID is unchanged: 16 random bytes, unique to each VM process, + selected with `0xa6`, and equal to the first 16 entropy bytes when a + restore packet exists. It distinguishes clones. +- After a restore, the guest requires `g` in the packet to equal its + recorded `g` plus one, and the generation ID to differ from the recorded + one. Either mismatch fails repair. + +## Guest obligations + +The guest's time component (`nvx-time` in `guest/common`) performs the +conformance checks, runs the violation watcher and the wall-clock discipline +as one daemon, and executes the snapshot agent's time steps. Every guest mode +starts it: init's interactive and test paths, `nvx_exec`, the sandbox agent, +and the managed agent. + +### Conformance checks and the `NVX-TIME-ABI` marker + +Init runs the boot check immediately after mounting `/proc`, `/sys`, and +`/dev`, before any other guest work, and starts the daemon only if it passes. +CPUID and MSRs are read on every online CPU through `/dev/cpu//cpuid` and +`/dev/cpu//msr`. + +| ID | Check | Phases | +| --- | --- | --- | +| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; `0x40000006` and `0x40000081` follow the out-of-range rule; no base `0x40000100..=0x4000ff00` carries `KVMKVMKVM` | boot, restore (new CPUs) | +| `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | +| `C3` | MSR `0x40000002` equals the CPU's VP index; `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz; `0x40000023` equals 1,000,000,000 or 200,000,000; `0x40000118` equals 1; reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | +| `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | +| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU | boot, restore (new CPUs) | +| `C6` | `current_clocksource` is `tsc`; `available_clocksource` contains only `tsc`, `refined-jiffies`, and `jiffies` | boot, capture, restore | +| `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | +| `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | +| `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | +| `C10` | The time daemon is running and has recorded no violation | capture, restore | +| `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | + +Capture and restore checks never wait. The boot `C7` check and the deferred +restore `C7` check may poll for at most 200 ms, because a CPU switches to +one-shot mode at its first tick. The CI conformance suite adds an exhaustive +check: every leaf `0x40000006..=0x400000ff`, writes of 0 and 1 to +`0x40000118` and of 2 (which must fail), and writes to every read-only +identity MSR (which must fail). + +On success the check prints one line to the console: + +```text +NVX-TIME-ABI: v=1 phase= status=ok cpus= tsc_hz= lapic_hz= generation= elapsed_us= +``` + +On failure it prints `status=fail check= detail=""` in the same +format, emits a violation event with code `G_CONFORMANCE_`, and powers +off with status 193. After the boot check passes, init takes one time sample +and steps the clock to host UTC before any workload starts; this initial step +is not counted as a discontinuity. + +### Violation watcher + +The daemon reads `/dev/kmsg` from the start of the buffer and then follows +it; it sets `oom_score_adj` to -1000. A record matches when its message text +contains every substring of a row: + +| Code | Substrings | +| --- | --- | +| `G_TSC_UNSTABLE` | `Marking TSC unstable due to ` | +| `G_CLOCKSOURCE_UNSTABLE` | `timekeeping watchdog on CPU` and `as unstable because the skew is too large` | +| `G_CLOCKSOURCE_SWITCH` | `clocksource: Switched to clocksource ` (any switch after the boot check) | +| `G_CLOCKSOURCE_SKEW` | `clocksource: ` and ` ahead of CPU `, or `clocksource: ` and ` behind CPU ` | +| `G_TSC_WARP` | `TSC synchronization [CPU#`, `cycles TSC warp between CPUs`, or `TSC warped randomly between CPUs` | +| `G_TSC_ADJUST` | `TSC ADJUST` | +| `G_RCU_STALL` | `rcu: INFO: ` and one of ` detected stalls on CPUs/tasks`, ` self-detected stall on CPU`, or ` detected expedited stalls` | +| `G_RCU_STARVED` | `rcu: ` and one of ` kthread starved for ` or ` kthread timer wakeup didn't happen for ` | +| `G_SOFT_LOCKUP` | `watchdog: BUG: soft lockup - CPU#` | +| `G_HARD_LOCKUP` | `Watchdog detected hard LOCKUP` | +| `G_HUNG_TASK` | `INFO: task ` and ` blocked for more than ` | +| `G_UNCHECKED_MSR` | `unchecked MSR access error` | +| `G_KMSG_OVERRUN` | A read of `/dev/kmsg` fails with `EPIPE` (records were lost) | + +Before the boot check passes, a match is reported by `C4`. Afterwards it is +a runtime violation: the daemon emits the event and powers off with status +194. + +A violation event is one line of at most 512 bytes: + +```text +NVX-TIME-ABI-VIOLATION: v=1 code= source= phase= generation= boottime_ns= detail="" +``` + +The guest writes it to `/dev/kmsg` at priority 2, to the portb data port +`0xe9` through `/dev/port`, and to the console. Consumers treat identical +lines as one event. The guest then writes the status to the shutdown port +`0x604` through `nvx-exit`, whose first byte becomes the OpenVMM process exit +status. + +### Snapshot agent + +`nvx-snapshot` performs these steps for every tier and for untiered +snapshots. Steps marked "debug" apply only to the CI debug kernel, which +builds the soft-lockup and hung-task detectors. + +Before the capture request: + +1. Run the capture checks (`C6`, `C7`, `C10`). They do not wait. +2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. +3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and + `/proc/sys/kernel/hung_task_timeout_secs`. +4. Apply the existing freezer and scratch barriers. +5. Write the capture request to `0x605`. + +If the write returns without a restore (no destination or a rejected +capture), the agent removes the barriers and then restores the values saved +in steps 2 and 3. Otherwise, in the restored process: + +6. Read the status from `0xea`. Bit 1 is always set after a restore. +7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, + and read the packet with four-byte reads. Validate the magic, version, + reserved bits, counts, `g`, and the generation ID (`G_REPAIR_PACKET`, + `G_REPAIR_GENERATION`). +8. Set the wall clock. Compute `theta` and `epsilon` from the bracket. If + `epsilon` exceeds 1 ms, take up to two time samples and keep the one with + the smallest uncertainty; if none is within 1 ms, fail + (`G_REPAIR_SAMPLE`). Apply `adjtimex` with `ADJ_SETOFFSET | ADJ_NANO` and + `time = theta`, then `ADJ_FREQUENCY | ADJ_STATUS` with + `freq = -rate_deviation` (clamped to ±500 ppm) and + `status = STA_PLL | STA_NANO` (`G_REPAIR_CLOCK`). Record the restore + discontinuity and the new `g`. +9. Run the existing processor and memory activation. +10. Run the existing entropy and identity repair for the tier. The RTC-based + wall-clock refresh is removed; `instance-checkpoint` restores also get + step 8. +11. Run the restore checks (`C1`, `C2`, and `C5` on newly onlined CPUs; `C3` + for CPU 0; `C6`; `C10`). They do not wait. +12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. +13. Signal the daemon to finish the restore off the latency path: + `membarrier(MEMBARRIER_CMD_GLOBAL, 0, 0)` (which waits for an RCU grace + period, so the grace period captured in flight has ended); restore the + values saved in steps 2 and 3 (`G_REPAIR_SUPPRESSION`); run the deferred + `C7` check; print `NVX-TIME-ABI: v=1 phase=restore status=ok`; and restart + the discipline at the fast cadence. + +Steps 6, 7, and 8 run in one helper process. Repair failures emit a violation +event and power off with status 195. + +### Wall-clock discipline + +The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: + +- **Cadence.** A poll every 16 s for the first four polls after the boot + check or a restore, then every 64 s. +- **Sample.** Up to three time samples per poll; keep the first with + `epsilon <= 50 µs`. Otherwise skip the poll and count a rejected sample. + A sample whose `g` differs from the recorded `g` is discarded: a restore + happened, and restore repair resets the discipline. +- **Step.** If `|theta| >= 128 ms`, apply `ADJ_SETOFFSET | ADJ_NANO` with + `time = theta`, count a discontinuity, and reapply the frequency and + status. +- **Slew.** Otherwise apply `ADJ_OFFSET | ADJ_STATUS | ADJ_NANO | + ADJ_TIMECONST | ADJ_MAXERROR | ADJ_ESTERROR` with `offset = theta` + nanoseconds, `status = STA_PLL | STA_NANO` (clearing `STA_UNSYNC`), + `constant = 4` at the 16 s cadence or 6 at the 64 s cadence, + `maxerror = ceil((|theta| + epsilon) / 1000)` µs, and + `esterror = ceil(epsilon / 1000)` µs. +- **Bounds.** The kernel limits the frequency to ±500 ppm; the declared rate + tolerance uses at most 250 ppm of it. + +The discipline never powers off the guest: a host wall-clock step is +followed, not reported as a violation. + +**Discontinuity state.** The daemon publishes `/run/nvx/time/state`, replaced +atomically with `rename(2)` after every change. The sandbox agent bind-mounts +`/run/nvx/time` read-only into the container at the same path. The file holds +`key=value` lines: + +| Key | Content | +| --- | --- | +| `version` | 1 | +| `generation` | `g` | +| `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | +| `last_discontinuity` | `none`, `restore`, or `step` | +| `last_step_ns` | Signed size of the last step | +| `last_step_realtime_ns` | `CLOCK_REALTIME` right after the last step | +| `last_downtime_ns` | `D` of the last restore | +| `last_downtime_source` | `monotonic` or `utc` | +| `synchronized` | 1 when the last accepted sample is at most 128 s old | +| `offset_ns` | Last measured `theta` | +| `uncertainty_ns` | Last accepted `epsilon` | +| `frequency_ppb` | Current kernel frequency correction | +| `samples`, `rejected_samples` | Sample counters | +| `violations` | 0; a violation powers the guest off | + +Workloads that need prompt notice of a step can also arm a `timerfd` with +`TFD_TIMER_CANCEL_ON_SET`, which the kernel cancels on every step. + +## Failure codes + +Codes are stable. OpenVMM errors put the code in brackets at the start of the +message, for example `[E_TSC_RATE_TOLERANCE] destination TSC rate ...`, and +the code survives error wrapping across the worker boundary. A rejected cold +boot or restore exits the OpenVMM process with status 1, the existing +fatal-error status; tests and orchestrators match the bracketed code. A +rejected capture is rollback-safe: the guest continues, and OpenVMM logs the +code. + +| VMM code | Condition | Detected at | +| --- | --- | --- | +| `E_SNAPSHOT_VERSION` | Manifest version is not 6; the snapshot must be recaptured | Restore | +| `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | +| `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | +| `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM | Restore | +| `E_PROFILE_DIGEST` | Embedded, recorded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | +| `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile | Cold boot | +| `E_PROFILE_TIME_BITS` | The profile violates the CPU time bits | Cold boot, restore | +| `E_CPU_GENERATION` | Host CPU generation not in the profile | Cold boot, restore | +| `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, or MSR value of the profile | Cold boot, restore | +| `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one | Restore | +| `E_IDENTITY_ROUTING` | Backend cannot deliver the identity CPUID or MSRs | Cold boot, restore | +| `E_TSC_SYNC_UNSUPPORTED` | Backend lacks the synchronized TSC set | Cold boot, restore | +| `E_TSC_SCALING_ACTIVE` | The guest TSC would be scaled | Cold boot, restore | +| `E_TSC_RATE_UNAVAILABLE` | Backend cannot report its native TSC rate | Cold boot, capture, restore | +| `E_TSC_RATE_IMPLAUSIBLE` | A rate is outside 500 MHz to 10 GHz | Cold boot, capture, restore | +| `E_TSC_RATE_TOLERANCE` | `abs(F_d - F_s)` exceeds 250 ppm of `F_s` | Restore | +| `E_LAPIC_RATE_UNAVAILABLE` | Backend cannot report its LAPIC rate | Cold boot, capture, restore | +| `E_LAPIC_RATE_MISMATCH` | `L` is not the backend constant, or `L_d` differs from `L_s` | Cold boot, restore | +| `E_HOST_IDENTITY` | Host identity, boot identity, or clocks unavailable | Capture, restore | +| `E_DOWNTIME_NEGATIVE` | Downtime below zero | Restore | +| `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days | Restore | +| `E_TSC_ANCHOR` | Capture anchor unavailable or not paired within 10 µs | Capture | +| `E_TSC_TARGET_OVERFLOW` | `TSC_target` exceeds 64 bits | Restore | +| `E_TSC_SYNC_READBACK` | A VP does not hold the synchronized value | Restore | +| `E_VP_LATE_CREATION` | A VP was instantiated after the synchronized set | Restore | +| `E_LAPIC_PERIODIC` | A periodic LAPIC timer is armed | Capture, restore | +| `E_LAPIC_TSC_DEADLINE` | TSC-deadline mode or state is present | Capture, restore | +| `E_PIT_ACTIVE` | PIT channel 0 is counting in a periodic mode | Capture, restore | +| `E_GENERATION_EXHAUSTED` | `capture_generation + 1` overflows `u32` | Restore | +| `E_CMDLINE_CLOCK_TOKEN` | `tsc_early_khz=` or `lapic_timer_hz=` in a supplied or saved command line | Cold boot, restore | +| `E_TEST_HOOK` | A malformed or unknown test hook | Cold boot, restore | + +Guest failures power off through `0x604` with a status distinct from the +statuses the guest already uses (0, 1, 37, 125, 126, 127, 128 to 192 for +signals, and 255): + +| Status | Class | Guest codes | +| ---: | --- | --- | +| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C11` | +| 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_KMSG_OVERRUN` | +| 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | + +A workload can exit with any 8-bit status, so a harness classifies a time +failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. + +## Host qualification + +`nvx.py doctor --backend ` qualifies a host interactively, and the +`validate-runner` action runs the same checks before every CI job. Both print +one `NVX-DOCTOR: check= status= detail=...` line per check and +fail if any check fails. The time checks replace the `nonstop_tsc` check. + +| ID | Check | +| --- | --- | +| `H1` | The backend device or API is present and usable | +| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, and the profile that `auto` selects (`E_PROFILE_HOST_UNKNOWN` fails) | +| `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | +| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d` | +| `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_offset_ns <= 1000` | +| `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_warp_ns` and `max_offset_ns` at most 1,000 | +| `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | +| `H8` | Runner only: the runner's `cpu-` label equals the fingerprinted generation | + +Runners carry exactly one generation label, `cpu-`, whose value is +the profile's generation key: Skylake-SP, Ice Lake-SP (8370C), or Emerald +Rapids (8573C); exact label values are TBD(profiles). Same-generation restore +tests select runners by label. Labels are assigned when a runner is +registered; agents never change runner registration. + +## Performance expectations and acceptance gate + +**Gate.** Per backend and vCPU count, no p50 is worse than the base-branch +median by more than `max(5%, 2 ms)`. The gate covers all 36 one-vCPU metrics +and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. + +Expected effects: + +| Change | Expected effect | +| --- | --- | +| WHP restored SMP without RDTSC emulation (if the WHP spike meets 1 µs) | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50) | +| Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits, most visible on WHP | +| Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | +| No capture-time clocksource waits | Removes up to 5 s of capture latency on MSHV and WHP; outside the restore metrics | +| No LAPIC calibration and no `tsc-early` window at cold boot | Unchanged or faster cold boot | +| Boot check and daemon start | Added cold-boot cost; budget 2 ms on KVM and MSHV and 5 ms on WHP, reported as `elapsed_us` | +| Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | +| Restore repair and checks before the acknowledgement | Run in one helper process; the RCU release and deferred checks run after the acknowledgement | + +Expected wins are tracked separately and do not relax the gate. + +## Test matrix + +**Unit tests (OpenVMM).** Rate boundary (250 ppm accepted, one hertz beyond +rejected) in both directions; downtime source selection and bounds, including +equal identities with a different clock kind; `TSC_target` arithmetic and +overflow; LAPIC tick arithmetic for every divide value, expiry, masking, and +periodic rejection; identity CPUID and MSR tables, including every #GP case +and `0x40000118` save and restore; capabilities derivation (`hv1` and +`kvm_clock` false); manifest version 6 validation and rejection of versions +2 through 5; packet v4 and time-sample golden vectors and their guest-side +parser; four-byte portb reads; and the generation counter. + +**Conformance.** The boot and restore checks pass on all 18 registered hosts +at 1, 2, 4, and 8 vCPUs, plus the exhaustive CI check and the warp probe on +every backend. + +**Restore matrix.** Every case runs with zero violations; rejected cases must +fail with the listed code. + +| Case | Hosts | Expected | +| --- | --- | --- | +| Same host, immediate | All | Restored; `DOWNTIME_UTC` clear | +| Same host, downtime of 30 s (over the 21 s RCU stall timeout) | All | Restored; no RCU stall | +| Same host under DVFS load | prometheus32 | Restored | +| Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | +| Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | +| Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | +| Across VMs of one generation | `azure-kvm-1` to `-2`; `azure-azlinux-3` to `-4`; `azure-windows-1` to `-2`; `azure-windows-3` to `-4` | Restored | +| Across generations | `azure-windows-1` (8370C) to `-3` (8573C) | `E_CPU_GENERATION` | +| Host without invariant TSC | `azure-azlinux-2` | Cold boot and restore rejected: `E_PROFILE_UNSUPPORTED` | +| Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | +| Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | +| Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | +| Every tier and an untiered snapshot; capture-restore chains | All backends | Restored; `g` increments by one per restore | + +No test reboots a host. The orchestrator asks the user before any real +reboot. + +**Reliability.** Repeated snapshot scenarios on every registered host at 1, +2, 4, and 8 vCPUs, with the warp probe and the watcher active, report no RCU, +soft-lockup, hung-task, clocksource, or warp message. The CI debug-kernel +variant runs the same-host cases with the soft-lockup and hung-task detectors +enabled. + +**Performance.** The gate above, computed by the performance agent from the +CI benchmark matrix. + +Test hooks are hidden OpenVMM options (`--x-time-abi-test-hook `, +repeatable): + +| Hook | Effect | +| --- | --- | +| `force-utc-downtime` | Select the UTC source even on the same host boot | +| `boot-id-mismatch` | Treat the destination boot identity as different | +| `dest-rate-offset-ppm=` | Perturb the measured `F_d` used by the rate policy and the reported deviation; the TSC is never scaled | +| `downtime-add-s=` | Add `n` seconds to the measured `D` before the bounds check | +| `utc-offset-ms=` | Add `n` milliseconds to every destination UTC reading: the downtime sample, the packet, and time samples | + +Every active hook is logged at warning level and sets `TEST_HOOKS` in the +packet and in every time sample. + +## Removed mechanisms and migration + +Removed from OpenVMM: + +- Cold-boot clock tokens: `prepare_cold_boot_command_line` no longer injects + `tsc_early_khz=` or `lapic_timer_hz=`, and their propagation, parsing, and + platform-tier validation are deleted. +- The exact-equality CPU contract (`CpuCompatibilityContract`, including its + `tsc_deadline` and `kvm_clock` fields) and host-derived CPUID, replaced by + CPU profiles. +- Leaf `0x15` synthesis (`tsc_frequency_cpuid_leaves`) on every backend. +- Per-backend downtime paths (`advance_snapshot_time`), per-VP TSC + advancement (`advance_tsc`), TSC-deadline advancement, periodic LAPIC + advancement, and wall-clock-only downtime (`calculate_snapshot_downtime`). +- KVM: `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, KVM CPUID leaves, + and `KVM_SET_TSC_KHZ`. +- MSHV: BSP-copy TSC alignment and exact rate equality. +- WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` + unless the WHP interim applies. +- Restore packets v1 to v3 and manifest versions 2 to 5. + +Removed from NVX: + +- Kernel: patch 0004 (`lapic_timer_hz`), `CONFIG_CPU_FREQ`, and + `CONFIG_X86_INTEL_PSTATE`. `CONFIG_HYPERVISOR_GUEST` stays on and + `CONFIG_HYPERV` stays off. +- Guest: RTC polling in `nvx-reseed`, and the restore packet v1 to v3 parser + in `nvx-port-io`. +- Harness: `tsc=reliable` and `no_timer_check` in `BASE_TUNING`; per-backend + `clocksource=` tokens; `stable_clocksource_wait_script` and the + capture-time clocksource waits; the KVM and WHP branches of + `snapshot-core`; the `smp-lapic` scenarios (`lapic=notscdeadline`), which + duplicate `smp`; `restore-tsc-sync`, `clearcpuid=tsc_adjust`, and the + fresh-boot TSC control, which the warp probe replaces; and the dmesg greps + in `restore-processors.sh`, which the watcher replaces. The + `cold_start_clocksource` benchmark scenario passes `clocksource=tsc` on + every backend. +- CI: the `nonstop_tsc` runner check, replaced by host qualification. + +Migration impact: + +- Every existing snapshot is rejected with `E_SNAPSHOT_VERSION`; templates + must be recaptured. +- OpenVMM, the kernel, and the initramfs must be upgraded together. A new + guest on an old OpenVMM fails the boot check (status 193); an old guest's + snapshot agent cannot parse packet v4 and fails closed. +- The guest identifies as Hyper-V on every backend, including KVM. +- CPUID becomes profile-defined, so guests can lose host features that no + profile of their generation pins. +- Hosts that fail qualification, such as `azure-azlinux-2`, cannot run + microVMs until replaced. +- Same-generation restore tests need `cpu-` runner labels. +- The `cold_start_clocksource` metric on KVM changes meaning (from + `kvm-clock` to `tsc`). +- Documentation updates: this document, [Snapshot and + restore](snapshot-and-restore.md), [Machine and device + ABI](machine-and-device-abi.md), [Cold boot](cold-boot.md), the + benchmarks and CI guides, and the OpenVMM Guide (see the appendix). + +## Appendix: OpenVMM Guide update plan + +`Guide/src/user_guide/openvmm/snapshots.md` changes as follows when the +implementation lands: + +1. **Overview.** Note that microVM snapshots carry a time contract and a CPU + profile, and that only manifest version 6 is accepted. +2. **Restoring a snapshot.** Replace the paragraphs on MSHV + `IA32_TSC_ADJUST`, KVM TSC offset advancement, MSHV and WHP time freezing + and BSP alignment, and the WHP partition-reference-time TSC with one + paragraph on the synchronized TSC set and its read-back. Keep the WHP + interim paragraph only if it applies. +3. **Generation ID.** Extend the portb paragraph with selector `0xa7`, window + port `0xeb`, status bit 6, four-byte reads, and packet v4. +4. **Device configuration on restore.** Replace the CPU contract, TSC + frequency, and `lapic_timer_hz` paragraphs with a new **Time and CPU + compatibility** section: the identity, the CPU profile and + `--cpu-profile`, the 250 ppm rate rule, the exact LAPIC rule, the + downtime sources and bounds, the restore steps, and the error codes. +5. **Limitations.** State that restore requires the same backend, CPU + generation, and profile, and that older snapshots must be recaptured. +6. **CLI reference.** Document `--cpu-profile` and the doctor verification + mode. The test hooks stay undocumented in the Guide; this document + describes them. From 88986ea00992c90570d277abb528f8dc8a89acfd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 00:49:35 -0700 Subject: [PATCH 002/188] kernel: drop cpufreq and intel_pstate from the microVM config intel_pstate reads MSR_TURBO_RATIO_LIMIT and writes MSR_IA32_PERF_CTL through rdmsrq_on_cpu() and wrmsrq_on_cpu(), which run with interrupts disabled. The microVM implements neither MSR, so both accesses fault and Linux prints full stack traces from that context. On the Azure WHP runners the two traces masked interrupts for about 200 ms of a console-logging boot, enough lost ticks to push tsc-early past the clocksource watchdog margin (#292). Under time ABI v1 the hypervisor identity gives Linux the TSC and LAPIC rates directly, so frequency scaling has no role in the guest. Turn off CONFIG_CPU_FREQ, which removes the governors and every x86 cpufreq driver, including CONFIG_X86_INTEL_PSTATE. CONFIG_SCHED_MC_PRIO goes too: it selects CPU_FREQ and X86_INTEL_PSTATE, so olddefconfig would otherwise turn both back on. It orders cores for Intel Turbo Boost Max 3.0 from ACPI CPPC data, which this ACPI-free guest never has. The input config also drops the lines olddefconfig no longer emits (ENERGY_MODEL, UCLAMP_TASK, and INTEL_TURBO_MAX_3). Running olddefconfig on the old and new inputs yields configs that differ only in these symbols. The kernel build, release packaging, and source verification now require the time-ABI kernel settings: HYPERVISOR_GUEST and PARAVIRT on, HYPERV and CPU_FREQ off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 7 +++++- kernel/config-microvm | 31 ++---------------------- scripts/nvx_tools/build.py | 9 +++++++ scripts/nvx_tools/build_constants.py | 6 +++++ scripts/nvx_tools/release.py | 1 + scripts/test_nvx_tools.py | 35 ++++++++++++++++++++++++++++ 6 files changed, 59 insertions(+), 30 deletions(-) diff --git a/doc/build.md b/doc/build.md index d8401c160..6cbfb1762 100644 --- a/doc/build.md +++ b/doc/build.md @@ -161,7 +161,12 @@ The native kernel build caches the verified and patched source under `.cache/linux`, uses `O=build/linux`, runs `olddefconfig`, exports the exact generated config as `build/vmlinux.config`, and fails if ACPI is enabled, PVH remains enabled, or the MP-table, APIC, IOAPIC, and command-line -virtio-mmio requirements are missing. Changing an archive hash or patch +virtio-mmio requirements are missing. It also enforces the time-ABI settings: +`CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` stay on, while `CONFIG_HYPERV` +and `CONFIG_CPU_FREQ` stay off. Without cpufreq, `intel_pstate` cannot probe +MSRs the microVM does not implement, so no `#GP` traces are printed with +interrupts disabled during boot. `CONFIG_SCHED_MC_PRIO` is off as well because +it selects both cpufreq and `intel_pstate`. Changing an archive hash or patch invalidates both source and object caches; changing the input configuration invalidates the object cache. diff --git a/kernel/config-microvm b/kernel/config-microvm index 2f9212728..9c5debed6 100644 --- a/kernel/config-microvm +++ b/kernel/config-microvm @@ -164,7 +164,6 @@ CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y # # Scheduler features # -# CONFIG_UCLAMP_TASK is not set # end of Scheduler features CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y @@ -353,7 +352,7 @@ CONFIG_NR_CPUS_RANGE_BEGIN=2 CONFIG_NR_CPUS_RANGE_END=512 CONFIG_NR_CPUS_DEFAULT=64 CONFIG_NR_CPUS=64 -CONFIG_SCHED_MC_PRIO=y +# CONFIG_SCHED_MC_PRIO is not set CONFIG_X86_LOCAL_APIC=y CONFIG_X86_IO_APIC=y CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y @@ -483,38 +482,13 @@ CONFIG_PM=y CONFIG_PM_DEBUG=y # CONFIG_PM_ADVANCED_DEBUG is not set # CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set -# CONFIG_ENERGY_MODEL is not set CONFIG_ARCH_SUPPORTS_ACPI=y # CONFIG_ACPI is not set # # CPU Frequency scaling # -CONFIG_CPU_FREQ=y -CONFIG_CPU_FREQ_GOV_ATTR_SET=y -CONFIG_CPU_FREQ_GOV_COMMON=y -# CONFIG_CPU_FREQ_STAT is not set -# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set -# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set -CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y -# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set -CONFIG_CPU_FREQ_GOV_PERFORMANCE=y -# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set -CONFIG_CPU_FREQ_GOV_USERSPACE=y -CONFIG_CPU_FREQ_GOV_ONDEMAND=y -# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set -CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y - -# -# CPU frequency scaling drivers -# -CONFIG_X86_INTEL_PSTATE=y -# CONFIG_X86_P4_CLOCKMOD is not set - -# -# shared options -# -CONFIG_CPUFREQ_ARCH_CUR_FREQ=y +# CONFIG_CPU_FREQ is not set # end of CPU Frequency scaling # @@ -1467,7 +1441,6 @@ CONFIG_X86_PLATFORM_DEVICES=y # end of Intel Uncore Frequency Control # CONFIG_INTEL_PUNIT_IPC is not set -# CONFIG_INTEL_TURBO_MAX_3 is not set # CONFIG_SIEMENS_SIMATIC_IPC is not set # CONFIG_WINMATE_FM07_KEYS is not set # CONFIG_COMMON_CLK is not set diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 7674f4c12..404ed981a 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -118,12 +118,21 @@ def _assert_shared_status_kernel_config(path: Path) -> None: ) +def _assert_time_abi_kernel_config(path: Path) -> None: + _assert_kernel_config( + path, + KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + "kernel configuration does not meet the NVX time ABI: ", + ) + + def assert_required_kernel_config(path: Path) -> None: """Validate the generated configuration required by the NVX platform.""" _assert_direct_boot_kernel_config(path) _assert_virtio_console_kernel_config(path) _assert_sandbox_kernel_config(path) _assert_shared_status_kernel_config(path) + _assert_time_abi_kernel_config(path) def _require_linux(workflow: str) -> None: diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 342f4be09..a7279ce72 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -74,6 +74,12 @@ class KernelBuildConstants: "CONFIG_VETH=y", "CONFIG_VIRTIO_BLK=y", ) + REQUIRED_TIME_ABI_CONFIG: Final = ( + "CONFIG_HYPERVISOR_GUEST=y", + "CONFIG_PARAVIRT=y", + "# CONFIG_HYPERV is not set", + "# CONFIG_CPU_FREQ is not set", + ) class OpenVMMBuildConstants: diff --git a/scripts/nvx_tools/release.py b/scripts/nvx_tools/release.py index 5fdc927ee..b26c8e3e4 100644 --- a/scripts/nvx_tools/release.py +++ b/scripts/nvx_tools/release.py @@ -1649,6 +1649,7 @@ def verify_source_tree() -> None: "CONFIG_VIRTIO_FS=y", "CONFIG_FUSE_FS=y", *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, + *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, ): if setting not in config.splitlines(): raise ScriptError(f"{config_path} is missing {setting}") diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 1772a1b3d..381279b3f 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -278,6 +278,7 @@ def _write_release_fixture( *KernelBuildConstants.REQUIRED_VIRTIO_CONSOLE_CONFIG, *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, + *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, ) ) + "\n", @@ -4620,6 +4621,7 @@ def test_kernel_build_rejects_input_config_mutation_during_build(self): *KernelBuildConstants.REQUIRED_VIRTIO_CONSOLE_CONFIG, *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, + *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, ) ) + "\n", @@ -5976,6 +5978,39 @@ def test_shared_status_kernel_config_is_required(self): ): build._assert_shared_status_kernel_config(config) + def test_checked_in_config_meets_time_abi_kernel_contract(self): + config = BuildConstants.REPO_ROOT / KernelBuildConstants.INPUT_CONFIG + build._assert_time_abi_kernel_config(config) + configured = config.read_text(encoding="utf-8").splitlines() + for removed in ( + "CONFIG_CPU_FREQ=y", + "CONFIG_X86_INTEL_PSTATE=y", + "CONFIG_SCHED_MC_PRIO=y", + "CONFIG_HYPERV=y", + ): + with self.subTest(removed=removed): + self.assertNotIn(removed, configured) + self.assertFalse( + any(line.startswith("CONFIG_CPU_FREQ_") for line in configured) + ) + + def test_time_abi_kernel_config_requires_every_setting(self): + with tempfile.TemporaryDirectory() as temporary: + config = Path(temporary) / ".config" + for missing in KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG: + with self.subTest(missing=missing): + config.write_text( + "\n".join( + setting + for setting in KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG + if setting != missing + ) + + "\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(common.ScriptError, missing): + build._assert_time_abi_kernel_config(config) + def _posix_shell() -> str | None: shell = shutil.which("sh") From b267c47265fce2b379f3b4830875295581bcfcf3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 00:54:11 -0700 Subject: [PATCH 003/188] kernel: add a CI debug kernel variant with the watchdogs enabled Production kernels keep the soft-lockup and hung-task detectors off, so nothing proves that snapshot restores never trip them. Time ABI v1 requires that RCU stall, soft-lockup, and hung-task reports never appear across a restore, so CI needs a kernel that has the detectors and is otherwise the production kernel. kernel/config-microvm-debug is a Kconfig fragment applied on top of kernel/config-microvm. Each assignment replaces the matching base line before olddefconfig, as merge_config.sh does, so Kconfig sees one value per symbol. The fragment enables DEBUG_KERNEL (both detectors depend on it), SOFTLOCKUP_DETECTOR, DETECT_HUNG_TASK, and RCU_CPU_STALL_CPUTIME. It pins the production RCU stall timeouts and turns off DEBUG_MISC, RCU_TRACE, and X86_DEBUG_FPU, which DEBUG_KERNEL would otherwise enable by default. Running olddefconfig with and without the fragment yields configs whose enabled options differ only in those detectors. `build-kernel --debug` and `build-guest --debug-kernel` build the variant into build/vmlinux-debug, with its own object directory and provenance. The provenance adds the fragment path and SHA-256 to the production fields. Production provenance and packaging are unchanged, and packaging rejects a kernel that enables the debug-only detectors. The Docker kernel-debug-artifacts target and the opt-in debug-kernel input of the build-guest-artifacts action let CI cache and build it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .gitattributes | 1 + .../actions/build-guest-artifacts/action.yml | 48 +++ doc/build.md | 38 +- doc/project-structure.md | 6 +- doc/usage.md | 6 +- docker/Dockerfile | 8 + kernel/config-microvm-debug | 22 ++ scripts/nvx.py | 25 +- scripts/nvx_tools/build.py | 150 ++++++- scripts/nvx_tools/build_config.py | 20 + scripts/nvx_tools/build_constants.py | 20 + scripts/test_nvx_tools.py | 369 +++++++++++++++++- 12 files changed, 680 insertions(+), 33 deletions(-) create mode 100644 kernel/config-microvm-debug diff --git a/.gitattributes b/.gitattributes index 2bee533a2..ad5ad7e6f 100644 --- a/.gitattributes +++ b/.gitattributes @@ -13,5 +13,6 @@ guest/* text eol=lf guest/**/* text eol=lf Dockerfile text eol=lf kernel/config-microvm text eol=lf +kernel/config-microvm-debug text eol=lf .github/workflows/*.lock.yml linguist-generated=true \ No newline at end of file diff --git a/.github/actions/build-guest-artifacts/action.yml b/.github/actions/build-guest-artifacts/action.yml index 138ac3186..1897c93f7 100644 --- a/.github/actions/build-guest-artifacts/action.yml +++ b/.github/actions/build-guest-artifacts/action.yml @@ -8,11 +8,18 @@ inputs: and cache saves for artifacts that miss the cache required: false default: "false" + debug-kernel: + description: Also cache or build the CI debug kernel (build/vmlinux-debug) + required: false + default: "false" outputs: kernel-cache-key: description: Exact cache key for the Linux kernel artifacts value: ${{ steps.guest-cache-keys.outputs.kernel }} + debug-kernel-cache-key: + description: Exact cache key for the Linux debug kernel artifacts + value: ${{ steps.guest-cache-keys.outputs.kernel-debug }} alpine-cache-key: description: Exact cache key for the Alpine disk image artifacts value: ${{ steps.guest-cache-keys.outputs.alpine }} @@ -59,6 +66,7 @@ runs: UBUNTU_VERSION: ${{ steps.guest-versions.outputs.ubuntu-version }} AZURELINUX_VERSION: ${{ steps.guest-versions.outputs.azurelinux-version }} KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/patches/**', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} + DEBUG_KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/config-microvm-debug', 'kernel/patches/**', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} ALPINE_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'guest/alpine/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py') }} UBUNTU_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'guest/ubuntu/**', 'ubuntu/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py', 'scripts/nvx_tools/ubuntu.py') }} AZURELINUX_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'azurelinux/**', 'docker/Dockerfile', 'scripts/nvx_tools/azurelinux.py', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py') }} @@ -66,6 +74,7 @@ runs: set -euo pipefail { echo "kernel=linux-kernel-v1-${KERNEL_VERSION}-${KERNEL_INPUT_HASH}" + echo "kernel-debug=linux-kernel-debug-v1-${KERNEL_VERSION}-${DEBUG_KERNEL_INPUT_HASH}" echo "alpine=alpine-image-v1-${ALPINE_VERSION}-${ALPINE_INPUT_HASH}" echo "ubuntu=ubuntu-initramfs-v1-${UBUNTU_VERSION}-${UBUNTU_INPUT_HASH}" echo "ubuntu-distro=ubuntu-distro-v1-${UBUNTU_VERSION}-${UBUNTU_INPUT_HASH}" @@ -145,6 +154,45 @@ runs: key: ${{ steps.guest-cache-keys.outputs.kernel }} enableCrossOsArchive: true + - name: Restore Linux debug kernel (zstd) + id: cache-linux-debug-kernel-zstd + if: inputs.debug-kernel == 'true' + uses: actions/cache/restore@v5 + with: + path: | + build/vmlinux-debug + build/vmlinux-debug.config + build/vmlinux-debug.provenance.json + key: ${{ steps.guest-cache-keys.outputs.kernel-debug }} + enableCrossOsArchive: true + + - name: Build Linux debug kernel (Docker) + if: >- + inputs.restore-only != 'true' && + inputs.debug-kernel == 'true' && + steps.cache-linux-debug-kernel-zstd.outputs.cache-hit != 'true' + shell: bash + run: >- + docker build + --file docker/Dockerfile + --target kernel-debug-artifacts + --output type=local,dest=build + . + + - name: Save Linux debug kernel (zstd) + if: >- + inputs.restore-only != 'true' && + inputs.debug-kernel == 'true' && + steps.cache-linux-debug-kernel-zstd.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + with: + path: | + build/vmlinux-debug + build/vmlinux-debug.config + build/vmlinux-debug.provenance.json + key: ${{ steps.guest-cache-keys.outputs.kernel-debug }} + enableCrossOsArchive: true + - name: Build Alpine disk image (Docker) if: inputs.restore-only != 'true' && steps.cache-alpine-image-zstd.outputs.cache-hit != 'true' shell: bash diff --git a/doc/build.md b/doc/build.md index 6cbfb1762..61297f0fa 100644 --- a/doc/build.md +++ b/doc/build.md @@ -166,10 +166,46 @@ virtio-mmio requirements are missing. It also enforces the time-ABI settings: and `CONFIG_CPU_FREQ` stay off. Without cpufreq, `intel_pstate` cannot probe MSRs the microVM does not implement, so no `#GP` traces are printed with interrupts disabled during boot. `CONFIG_SCHED_MC_PRIO` is off as well because -it selects both cpufreq and `intel_pstate`. Changing an archive hash or patch +it selects both cpufreq and `intel_pstate`. Production kernels must not enable +the soft-lockup or hung-task detectors. Changing an archive hash or patch invalidates both source and object caches; changing the input configuration invalidates the object cache. +## CI debug kernel + +CI proves that snapshot restores never trip the kernel watchdogs with a debug +variant of the same kernel: + +```bash +python3 scripts/nvx.py build-guest --debug-kernel +python3 scripts/nvx.py build-kernel --debug # native, Linux only +``` + +The variant applies the +[`kernel/config-microvm-debug`](../kernel/config-microvm-debug) fragment on top +of `kernel/config-microvm`. Each assignment in the fragment replaces the +matching base assignment before `olddefconfig`. It enables `DEBUG_KERNEL`, the +soft-lockup and hung-task detectors, and extra RCU stall diagnostics. It pins +the production RCU stall timeouts and turns off the debug options that +`DEBUG_KERNEL` would otherwise enable by default. The build fails if the +detectors are missing from the generated config. CI can shorten the thresholds +for one boot with `watchdog_thresh=`, `rcupdate.rcu_cpu_stall_timeout=`, and +`sysctl.kernel.hung_task_timeout_secs=`. + +The variant builds in `build/linux-debug`, so it never invalidates the +production object cache, and it produces: + +```text +build/vmlinux-debug +build/vmlinux-debug.config +build/vmlinux-debug.provenance.json +``` + +Its provenance has the production fields plus a `debug_config_fragment` path +and SHA-256. The debug kernel is a CI artifact and is never packaged. The +`build-guest-artifacts` action caches and builds it only when its +`debug-kernel` input is `true`. + Release packaging stages and verifies a complete output before replacing an existing `dist/` version. Its `SOURCE-MANIFEST.json` records the package version and exact hashes for OpenVMM, Linux, the generated kernel config, and diff --git a/doc/project-structure.md b/doc/project-structure.md index e67ded4c9..92e817731 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -65,7 +65,8 @@ nvx/ |-- kernel/ Linux configuration and NVX patch set | |-- patches/ Ordered patches applied to Linux | |-- COPYING-LINUX Linux copyright and license notice -| `-- config-microvm MicroVM kernel configuration +| |-- config-microvm MicroVM kernel configuration +| `-- config-microvm-debug CI debug-kernel fragment (watchdogs on) |-- aci_edge_sandboxes/ Rust crate `aci_edge_sandboxes` for the state-aware sandbox API | |-- src/ Facade, contract model, and backends | | |-- openvmm/ Default backend that drives the openvmm binary @@ -162,7 +163,8 @@ reproducible Linux environment. ### `kernel/` Inputs owned by NVX for producing the guest kernel. `config-microvm` defines the -kernel build. Files in `patches/` are applied in name order to the pinned Linux +kernel build. `config-microvm-debug` is a fragment applied on top of it for the +CI debug kernel. Files in `patches/` are applied in name order to the pinned Linux source. `COPYING-LINUX` records the upstream Linux copyright and license notice. See the [build guide](build.md#building-the-packaged-linux-source) for kernel-specific details. diff --git a/doc/usage.md b/doc/usage.md index 9b5e67d17..9eeeee0fc 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -115,20 +115,24 @@ revision and is used before packaging kernel provenance inputs. python3 scripts/nvx.py build-guest [--guest {alpine,ubuntu,azurelinux,all}] [--native] + [--debug-kernel] ``` By default, builds the guest kernel and initramfs with Docker. `--native` builds the selected artifacts directly on Linux instead. Alpine is the default. Azure Linux does not support `--native` and always builds through Docker. `--guest all` also builds the Ubuntu EROFS distro layer. +`--debug-kernel` also builds the CI debug kernel, `build/vmlinux-debug`; see +[Build](build.md#ci-debug-kernel). ### `build-kernel` ```console -python3 scripts/nvx.py build-kernel +python3 scripts/nvx.py build-kernel [--debug] ``` Fetches, verifies, patches, and builds the pinned kernel directly on Linux. +`--debug` builds the CI debug variant instead of the production kernel. ### `build-initramfs` diff --git a/docker/Dockerfile b/docker/Dockerfile index 6c046674d..1047a7e6d 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -39,6 +39,9 @@ COPY LICENSE README.md SOURCE-MANIFEST.json THIRD_PARTY_NOTICES.md /repo/ FROM base AS kernel RUN python3 scripts/nvx.py build-kernel +FROM base AS kernel-debug +RUN python3 scripts/nvx.py build-kernel --debug + FROM base AS initramfs RUN python3 scripts/nvx.py build-initramfs --guest alpine @@ -201,6 +204,11 @@ COPY --from=kernel /repo/build/vmlinux /vmlinux COPY --from=kernel /repo/build/vmlinux.config /vmlinux.config COPY --from=kernel /repo/build/vmlinux.provenance.json /vmlinux.provenance.json +FROM scratch AS kernel-debug-artifacts +COPY --from=kernel-debug /repo/build/vmlinux-debug /vmlinux-debug +COPY --from=kernel-debug /repo/build/vmlinux-debug.config /vmlinux-debug.config +COPY --from=kernel-debug /repo/build/vmlinux-debug.provenance.json /vmlinux-debug.provenance.json + FROM scratch AS initramfs-artifacts COPY --from=initramfs /repo/build/initramfs.cpio.gz /initramfs.cpio.gz COPY --from=initramfs /repo/build/initramfs.cpio.gz.packages.json /initramfs.cpio.gz.packages.json diff --git a/kernel/config-microvm-debug b/kernel/config-microvm-debug new file mode 100644 index 000000000..68994a1f5 --- /dev/null +++ b/kernel/config-microvm-debug @@ -0,0 +1,22 @@ +# NVX CI debug-kernel fragment. +# +# `nvx.py build-kernel --debug` appends this fragment to kernel/config-microvm +# before `olddefconfig`. It enables the watchdogs that production keeps off so +# CI can prove that snapshot restores never trip them. Nothing else may differ +# from production: DEBUG_KERNEL would otherwise default DEBUG_MISC, RCU_TRACE, +# and X86_DEBUG_FPU on, and the RCU stall settings are pinned to the production +# values. CI can shorten the thresholds per boot with `watchdog_thresh=`, +# `rcupdate.rcu_cpu_stall_timeout=`, and +# `sysctl.kernel.hung_task_timeout_secs=` instead of rebuilding. +CONFIG_DEBUG_KERNEL=y +# CONFIG_DEBUG_MISC is not set +# CONFIG_RCU_TRACE is not set +# CONFIG_X86_DEBUG_FPU is not set +CONFIG_SOFTLOCKUP_DETECTOR=y +# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set +CONFIG_DETECT_HUNG_TASK=y +CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120 +# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set +CONFIG_RCU_CPU_STALL_TIMEOUT=21 +CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=0 +CONFIG_RCU_CPU_STALL_CPUTIME=y diff --git a/scripts/nvx.py b/scripts/nvx.py index a24618c2b..5b623d0e5 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -164,6 +164,7 @@ def _build_config(args: argparse.Namespace) -> BuildConfig: return BuildConfig( guest=getattr(args, "guest", InitramfsBuildConstants.DEFAULT_GUEST), native_guest=getattr(args, "native", False), + debug_kernel=getattr(args, "debug_kernel", False), openvmm=_openvmm_build_config(args), ) @@ -172,8 +173,12 @@ def command_build_guest(args: argparse.Namespace) -> None: build_guest(_build_config(args)) -def command_build_kernel(_: argparse.Namespace) -> None: - build_kernel(KernelBuildConfig()) +def command_build_kernel(args: argparse.Namespace) -> None: + build_kernel( + KernelBuildConfig.debug_variant() + if getattr(args, "debug", False) + else KernelBuildConfig() + ) def command_build_initramfs(args: argparse.Namespace) -> None: @@ -657,6 +662,14 @@ def _add_guest_options( action="store_true", help="build directly on Linux instead of using Docker", ) + parser.add_argument( + "--debug-kernel", + action="store_true", + help=( + "also build the CI debug kernel (build/vmlinux-debug), which adds " + "the soft-lockup, hung-task, and RCU stall diagnostics" + ), + ) def _add_openvmm_options(parser: argparse.ArgumentParser) -> None: @@ -686,6 +699,14 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: "build-kernel", help="fetch, patch, and build the pinned kernel natively on Linux", ) + kernel.add_argument( + "--debug", + action="store_true", + help=( + "build the CI debug variant (build/vmlinux-debug) by applying " + "kernel/config-microvm-debug" + ), + ) kernel.set_defaults(handler=command_build_kernel) initramfs = subparsers.add_parser( diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 404ed981a..062d51148 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -2,6 +2,7 @@ from __future__ import annotations +import hashlib import json import os import shlex @@ -126,13 +127,35 @@ def _assert_time_abi_kernel_config(path: Path) -> None: ) -def assert_required_kernel_config(path: Path) -> None: +def _assert_watchdog_kernel_config(path: Path, *, debug: bool) -> None: + if debug: + _assert_kernel_config( + path, + KernelBuildConstants.REQUIRED_DEBUG_CONFIG, + "debug kernel configuration lacks the CI watchdogs: ", + ) + return + configured = set(path.read_text(encoding="utf-8").splitlines()) + enabled = [ + setting + for setting in KernelBuildConstants.DEBUG_WATCHDOG_CONFIG + if setting in configured + ] + if enabled: + raise ScriptError( + "production kernel configuration enables debug-only watchdogs: " + + ", ".join(enabled) + ) + + +def assert_required_kernel_config(path: Path, *, debug: bool = False) -> None: """Validate the generated configuration required by the NVX platform.""" _assert_direct_boot_kernel_config(path) _assert_virtio_console_kernel_config(path) _assert_sandbox_kernel_config(path) _assert_shared_status_kernel_config(path) _assert_time_abi_kernel_config(path) + _assert_watchdog_kernel_config(path, debug=debug) def _require_linux(workflow: str) -> None: @@ -233,14 +256,21 @@ def _kernel_source_fingerprint() -> str: def _kernel_provenance_inputs( source_fingerprint: str, input_config_sha256: str, + debug_config_fragment_sha256: str | None = None, ) -> dict[str, object]: - return { + inputs: dict[str, object] = { "source": json.loads(source_fingerprint), "input_config": { "path": KernelBuildConstants.INPUT_CONFIG.as_posix(), "sha256": input_config_sha256, }, } + if debug_config_fragment_sha256 is not None: + inputs["debug_config_fragment"] = { + "path": KernelBuildConstants.DEBUG_CONFIG_FRAGMENT.as_posix(), + "sha256": debug_config_fragment_sha256, + } + return inputs def kernel_provenance_inputs() -> dict[str, object]: @@ -251,6 +281,43 @@ def kernel_provenance_inputs() -> dict[str, object]: ) +def _kernel_config_symbol(line: str) -> str | None: + if line.startswith("CONFIG_") and "=" in line: + return line.split("=", 1)[0] + if line.startswith("# CONFIG_") and line.endswith(" is not set"): + return line[2 : -len(" is not set")] + return None + + +def merge_kernel_config_fragment(base: str, fragment: str) -> str: + """Apply a Kconfig fragment the way ``merge_config.sh`` does. + + Every symbol the fragment assigns replaces the base assignment, so + ``olddefconfig`` sees one value per symbol. + """ + assignments: list[str] = [] + for number, line in enumerate(fragment.splitlines(), start=1): + symbol = _kernel_config_symbol(line) + if symbol is not None: + assignments.append(line) + elif line and not line.startswith("#"): + raise ScriptError( + f"kernel config fragment line {number} is invalid: {line}" + ) + if not assignments: + raise ScriptError("kernel config fragment assigns no symbols") + symbols = [_kernel_config_symbol(line) for line in assignments] + if len(set(symbols)) != len(symbols): + raise ScriptError("kernel config fragment assigns a symbol more than once") + replaced = set(symbols) + kept = [ + line + for line in base.splitlines() + if _kernel_config_symbol(line) not in replaced + ] + return "\n".join((*kept, *assignments)) + "\n" + + def _initramfs_source_files() -> tuple[Path, ...]: sources = [ BuildConstants.REPO_ROOT / DockerBuildConstants.DOCKERFILE, @@ -421,12 +488,16 @@ def build_guest(config: BuildConfig) -> None: "require Docker" ) build_kernel(config.kernel) + if config.debug_kernel: + build_kernel(KernelBuildConfig.debug_variant()) for guest in config.selected_guests(): build_initramfs(config.initramfs_config(guest)) if config.guest == "all": build_distro_layer(config.distro_layer_config()) return build_docker_artifacts(config.docker, config.guest) + if config.debug_kernel: + build_docker_debug_kernel(config.docker) def build_all(config: BuildConfig) -> None: @@ -1042,18 +1113,31 @@ def build_kernel(config: KernelBuildConfig) -> None: require_tool(tool) source, source_fingerprint = prepare_kernel_source(config) input_config = BuildConstants.REPO_ROOT / KernelBuildConstants.INPUT_CONFIG - input_config_sha256 = sha256_file(input_config) + input_config_bytes = input_config.read_bytes() + input_config_sha256 = hashlib.sha256(input_config_bytes).hexdigest() + fragment = ( + BuildConstants.REPO_ROOT / KernelBuildConstants.DEBUG_CONFIG_FRAGMENT + if config.debug + else None + ) + fragment_bytes = fragment.read_bytes() if fragment is not None else None + fragment_sha256 = ( + hashlib.sha256(fragment_bytes).hexdigest() + if fragment_bytes is not None + else None + ) provenance_inputs = _kernel_provenance_inputs( source_fingerprint, input_config_sha256, + fragment_sha256, ) - build_fingerprint = json.dumps( - { - "source": source_fingerprint, - "input_config_sha256": input_config_sha256, - }, - sort_keys=True, - ) + build_inputs: dict[str, str] = { + "source": source_fingerprint, + "input_config_sha256": input_config_sha256, + } + if fragment_sha256 is not None: + build_inputs["debug_config_fragment_sha256"] = fragment_sha256 + build_fingerprint = json.dumps(build_inputs, sort_keys=True) build_stamp = config.work / KernelBuildConstants.BUILD_STAMP_NAME cached_build_fingerprint = ( build_stamp.read_text(encoding="utf-8") if build_stamp.is_file() else None @@ -1063,16 +1147,24 @@ def build_kernel(config: KernelBuildConfig) -> None: config.work.mkdir(parents=True, exist_ok=True) build_stamp.write_text(build_fingerprint, encoding="utf-8") kernel_config = config.work / KernelBuildConstants.BUILD_CONFIG_NAME - provenance_path = config.output.with_name(KernelBuildConstants.PROVENANCE_NAME) + provenance_path = config.output.with_name(config.provenance_name) provenance_path.unlink(missing_ok=True) - shutil.copy2(input_config, kernel_config) - if sha256_file(kernel_config) != input_config_sha256: - raise ScriptError("kernel input config changed while it was being copied") + if fragment_bytes is None: + kernel_config.write_bytes(input_config_bytes) + else: + print(f">> applying {KernelBuildConstants.DEBUG_CONFIG_FRAGMENT.as_posix()}") + kernel_config.write_text( + merge_kernel_config_fragment( + input_config_bytes.decode("utf-8"), + fragment_bytes.decode("utf-8"), + ), + encoding="utf-8", + ) make = ["make", "-C", source, f"O={config.work}"] run_checked([*make, "olddefconfig"]) - assert_required_kernel_config(kernel_config) + assert_required_kernel_config(kernel_config, debug=config.debug) jobs = os.cpu_count() or 1 - print(f">> building vmlinux with {jobs} jobs") + print(f">> building {config.output.name} with {jobs} jobs") run_checked([*make, f"-j{jobs}", KernelBuildConstants.BINARY_NAME]) config.output.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(config.work / KernelBuildConstants.BINARY_NAME, config.output) @@ -1083,6 +1175,7 @@ def build_kernel(config: KernelBuildConfig) -> None: if ( _kernel_source_fingerprint() != source_fingerprint or sha256_file(input_config) != input_config_sha256 + or (fragment is not None and sha256_file(fragment) != fragment_sha256) ): config.output.unlink(missing_ok=True) generated_config.unlink(missing_ok=True) @@ -1190,6 +1283,31 @@ def build_docker_artifacts( f"(kernel {KernelBuildConstants.VERSION}, {guest_label})" ) run_checked(docker_build_command(config, target), cwd=BuildConstants.REPO_ROOT) + _require_docker_outputs(destination, expected) + + +def build_docker_debug_kernel(config: DockerBuildConfig) -> None: + """Build the CI debug kernel variant through the Docker pipeline.""" + require_tool( + "docker", + "docker was not found on PATH; install Docker with the Linux engine first", + ) + destination = _docker_destination(config.artifact_destination) + print( + f">> building the debug Linux kernel into '{destination}' " + f"(kernel {KernelBuildConstants.VERSION}, " + f"{KernelBuildConstants.DEBUG_CONFIG_FRAGMENT.as_posix()})" + ) + run_checked( + docker_build_command(config, DockerBuildConstants.DEBUG_KERNEL_TARGET), + cwd=BuildConstants.REPO_ROOT, + ) + _require_docker_outputs( + destination, DockerBuildConstants.DEBUG_KERNEL_ARTIFACT_NAMES + ) + + +def _require_docker_outputs(destination: Path, expected: tuple[str, ...]) -> None: _require_docker_artifacts(destination, expected) print(">> done:") for name in expected: diff --git a/scripts/nvx_tools/build_config.py b/scripts/nvx_tools/build_config.py index c79bf04f6..95ec7f99c 100644 --- a/scripts/nvx_tools/build_config.py +++ b/scripts/nvx_tools/build_config.py @@ -63,6 +63,25 @@ class KernelBuildConfig: work: Path = BuildConstants.BUILD_DIR / KernelBuildConstants.WORK_DIRECTORY_NAME output: Path = artifact_path(KernelBuildConstants.BINARY_NAME) cache_directory: Path = field(default_factory=_default_cache_directory) + debug: bool = False + + @staticmethod + def debug_variant() -> KernelBuildConfig: + """Return the CI debug kernel, which adds the debug config fragment.""" + return KernelBuildConfig( + work=BuildConstants.BUILD_DIR + / KernelBuildConstants.DEBUG_WORK_DIRECTORY_NAME, + output=artifact_path(KernelBuildConstants.DEBUG_BINARY_NAME), + debug=True, + ) + + @property + def provenance_name(self) -> str: + return ( + KernelBuildConstants.DEBUG_PROVENANCE_NAME + if self.debug + else KernelBuildConstants.PROVENANCE_NAME + ) @dataclass(frozen=True, slots=True) @@ -97,6 +116,7 @@ def openvmm_target_output(self, platform: OpenVmmPlatform) -> Path: class BuildConfig: guest: str = InitramfsBuildConstants.DEFAULT_GUEST native_guest: bool = False + debug_kernel: bool = False docker: DockerBuildConfig = field(default_factory=DockerBuildConfig) kernel: KernelBuildConfig = field(default_factory=KernelBuildConfig) openvmm: OpenVmmBuildConfig = field(default_factory=OpenVmmBuildConfig) diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index a7279ce72..49920f46e 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -35,11 +35,16 @@ class KernelBuildConstants: BUILD_STAMP_NAME: Final = ".nvx-build.json" BUILD_CONFIG_NAME: Final = ".config" INPUT_CONFIG: Final = Path("kernel") / "config-microvm" + DEBUG_CONFIG_FRAGMENT: Final = Path("kernel") / "config-microvm-debug" PATCH_DIRECTORY: Final = Path("kernel") / "patches" BINARY_NAME: Final = "vmlinux" CONFIG_NAME: Final = f"{BINARY_NAME}.config" PROVENANCE_NAME: Final = "vmlinux.provenance.json" PROVENANCE_FORMAT: Final = 1 + DEBUG_WORK_DIRECTORY_NAME: Final = "linux-debug" + DEBUG_BINARY_NAME: Final = "vmlinux-debug" + DEBUG_CONFIG_NAME: Final = f"{DEBUG_BINARY_NAME}.config" + DEBUG_PROVENANCE_NAME: Final = f"{DEBUG_BINARY_NAME}.provenance.json" REQUIRED_VIRTIO_CONSOLE_CONFIG: Final = ( "CONFIG_HVC_DRIVER=y", "CONFIG_VIRTIO=y", @@ -80,6 +85,15 @@ class KernelBuildConstants: "# CONFIG_HYPERV is not set", "# CONFIG_CPU_FREQ is not set", ) + DEBUG_WATCHDOG_CONFIG: Final = ( + "CONFIG_SOFTLOCKUP_DETECTOR=y", + "CONFIG_DETECT_HUNG_TASK=y", + ) + REQUIRED_DEBUG_CONFIG: Final = ( + "CONFIG_DEBUG_KERNEL=y", + *DEBUG_WATCHDOG_CONFIG, + "CONFIG_RCU_CPU_STALL_CPUTIME=y", + ) class OpenVMMBuildConstants: @@ -306,8 +320,14 @@ class DockerBuildConstants: AZURELINUX_TARGET: Final = "azurelinux-guest-artifacts" AZURELINUX_INITRAMFS_TARGET: Final = "azurelinux-initramfs-artifacts" ALL_GUESTS_TARGET: Final = "all-guest-artifacts" + DEBUG_KERNEL_TARGET: Final = "kernel-debug-artifacts" LINUX_SOURCE_TARGET: Final = "linux-source-artifacts" OUTPUT_TYPE: Final = "local" + DEBUG_KERNEL_ARTIFACT_NAMES: Final = ( + KernelBuildConstants.DEBUG_BINARY_NAME, + KernelBuildConstants.DEBUG_CONFIG_NAME, + KernelBuildConstants.DEBUG_PROVENANCE_NAME, + ) ALL_GUEST_ARTIFACT_NAMES: Final = ( KernelBuildConstants.BINARY_NAME, KernelBuildConstants.CONFIG_NAME, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 381279b3f..736872f82 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -640,6 +640,15 @@ def test_openvmm_help_describes_petri_vmm_tests(self): def test_guest_selection_cli_contract(self): default_build = nvx.parse_args(["build-guest"]) self.assertEqual(default_build.guest, "alpine") + self.assertFalse(default_build.debug_kernel) + + debug_guest = nvx.parse_args(["build-guest", "--debug-kernel"]) + self.assertTrue(debug_guest.debug_kernel) + self.assertTrue(nvx._build_config(debug_guest).debug_kernel) + self.assertTrue(nvx.parse_args(["build", "--debug-kernel"]).debug_kernel) + + self.assertFalse(nvx.parse_args(["build-kernel"]).debug) + self.assertTrue(nvx.parse_args(["build-kernel", "--debug"]).debug) all_guests = nvx.parse_args(["build-guest", "--guest", "all"]) self.assertEqual(all_guests.guest, "all") @@ -1725,6 +1734,13 @@ def test_build_commands_pass_specialized_configs_to_consumers(self): build_kernel.call_args.args[0], build_config.KernelBuildConfig, ) + self.assertFalse(build_kernel.call_args.args[0].debug) + with patch.object(nvx, "build_kernel") as build_debug_kernel: + nvx.command_build_kernel(argparse.Namespace(debug=True)) + self.assertEqual( + build_debug_kernel.call_args.args[0], + build_config.KernelBuildConfig.debug_variant(), + ) self.assertIsInstance( build_initramfs.call_args.args[0], build_config.InitramfsBuildConfig, @@ -2927,20 +2943,22 @@ def test_guest_artifact_restore_only_leaves_restore_and_scratch_steps_active(sel ] self.assertTrue(restore_steps) - for step in restore_steps: + scratch = _composite_action_step( + action, "Prepare Ubuntu sandbox scratch template" + ) + for step in (*restore_steps, scratch): try: name = _yaml_scalar(step, "name") except AssertionError: name = "" with self.subTest(step=name): - with self.assertRaisesRegex(AssertionError, "has no if field"): - _yaml_scalar(step, "if") - - scratch = _composite_action_step( - action, "Prepare Ubuntu sandbox scratch template" - ) - with self.assertRaisesRegex(AssertionError, "has no if field"): - _yaml_scalar(scratch, "if") + # Other inputs, such as debug-kernel, may select these steps, + # but restore-only must leave them active. + try: + condition = _yaml_scalar(step, "if") + except AssertionError: + continue + self.assertNotIn("restore-only", condition) def test_ci_runs_openvmm_tests_and_unit_tests_on_each_backend(self): workflow = ( @@ -3849,6 +3867,9 @@ def test_build_pins_and_artifact_names_have_one_literal_owner(self): KernelBuildConstants.BINARY_NAME, KernelBuildConstants.CONFIG_NAME, KernelBuildConstants.PROVENANCE_NAME, + KernelBuildConstants.DEBUG_BINARY_NAME, + KernelBuildConstants.DEBUG_CONFIG_NAME, + KernelBuildConstants.DEBUG_PROVENANCE_NAME, OpenVMMBuildConstants.GNU_RUST_TARGET, OpenVMMBuildConstants.MUSL_RUST_TARGET, OpenVMMBuildConstants.WINDOWS_RUST_TARGET, @@ -5603,10 +5624,58 @@ def test_kernel_input_config_uses_canonical_lf_line_endings(self): attributes = (BuildConstants.REPO_ROOT / ".gitattributes").read_text( encoding="utf-8" ) + for path in ( + KernelBuildConstants.INPUT_CONFIG, + KernelBuildConstants.DEBUG_CONFIG_FRAGMENT, + ): + with self.subTest(path=path): + self.assertIn( + f"{path.as_posix()} text eol=lf", + attributes.splitlines(), + ) + + def test_ci_debug_kernel_is_opt_in_and_keyed_by_its_fragment(self): + action = ( + BuildConstants.REPO_ROOT + / ".github" + / "actions" + / "build-guest-artifacts" + / "action.yml" + ).read_text(encoding="utf-8") + cache_input = next( + line + for line in action.splitlines() + if line.strip().startswith("DEBUG_KERNEL_INPUT_HASH:") + ) + for path in ( + "kernel/config-microvm", + "kernel/config-microvm-debug", + "kernel/patches/**", + "scripts/nvx_tools/build.py", + "scripts/nvx_tools/build_constants.py", + ): + with self.subTest(path=path): + self.assertIn(f"'{path}'", cache_input) + kernel_input = next( + line + for line in action.splitlines() + if line.strip().startswith("KERNEL_INPUT_HASH:") + ) + self.assertNotIn("config-microvm-debug", kernel_input) + build_step = _composite_action_step(action, "Build Linux debug kernel (Docker)") + self.assertIn("inputs.debug-kernel == 'true'", build_step) self.assertIn( - "kernel/config-microvm text eol=lf", - attributes.splitlines(), + f"--target {DockerBuildConstants.DEBUG_KERNEL_TARGET}", build_step ) + self.assertIn('default: "false"', action) + for name in DockerBuildConstants.DEBUG_KERNEL_ARTIFACT_NAMES: + with self.subTest(artifact=name): + self.assertEqual( + [line.strip() for line in action.splitlines()].count( + f"build/{name}" + ), + 2, + ) def test_ci_guest_cache_keys_include_shared_build_modules(self): action = ( @@ -5981,6 +6050,7 @@ def test_shared_status_kernel_config_is_required(self): def test_checked_in_config_meets_time_abi_kernel_contract(self): config = BuildConstants.REPO_ROOT / KernelBuildConstants.INPUT_CONFIG build._assert_time_abi_kernel_config(config) + build._assert_watchdog_kernel_config(config, debug=False) configured = config.read_text(encoding="utf-8").splitlines() for removed in ( "CONFIG_CPU_FREQ=y", @@ -6011,6 +6081,283 @@ def test_time_abi_kernel_config_requires_every_setting(self): with self.assertRaisesRegex(common.ScriptError, missing): build._assert_time_abi_kernel_config(config) + def test_watchdog_kernel_config_separates_production_and_debug(self): + with tempfile.TemporaryDirectory() as temporary: + config = Path(temporary) / ".config" + config.write_text( + "\n".join(KernelBuildConstants.REQUIRED_DEBUG_CONFIG) + "\n", + encoding="utf-8", + ) + build._assert_watchdog_kernel_config(config, debug=True) + with self.assertRaisesRegex(common.ScriptError, "debug-only watchdogs"): + build._assert_watchdog_kernel_config(config, debug=False) + + config.write_text("CONFIG_DEBUG_KERNEL=y\n", encoding="utf-8") + build._assert_watchdog_kernel_config(config, debug=False) + with self.assertRaisesRegex( + common.ScriptError, "CONFIG_SOFTLOCKUP_DETECTOR=y" + ): + build._assert_watchdog_kernel_config(config, debug=True) + + def test_kernel_config_fragment_replaces_base_assignments(self): + base = "\n".join( + ( + "#", + "# Kernel hacking", + "#", + "# CONFIG_DEBUG_KERNEL is not set", + "CONFIG_RCU_CPU_STALL_TIMEOUT=21", + "CONFIG_UNRELATED=y", + ) + ) + fragment = "\n".join( + ( + "# Comment lines and blank lines are ignored.", + "", + "CONFIG_DEBUG_KERNEL=y", + "# CONFIG_DEBUG_MISC is not set", + "CONFIG_RCU_CPU_STALL_TIMEOUT=21", + ) + ) + self.assertEqual( + build.merge_kernel_config_fragment(base, fragment).splitlines(), + [ + "#", + "# Kernel hacking", + "#", + "CONFIG_UNRELATED=y", + "CONFIG_DEBUG_KERNEL=y", + "# CONFIG_DEBUG_MISC is not set", + "CONFIG_RCU_CPU_STALL_TIMEOUT=21", + ], + ) + for invalid, message in ( + ("DEBUG_KERNEL=y\n", "line 1 is invalid"), + ("# only a comment\n", "assigns no symbols"), + ( + "CONFIG_DEBUG_KERNEL=y\n# CONFIG_DEBUG_KERNEL is not set\n", + "more than once", + ), + ): + with self.subTest(fragment=invalid): + with self.assertRaisesRegex(common.ScriptError, message): + build.merge_kernel_config_fragment(base, invalid) + + def test_checked_in_debug_fragment_only_adds_watchdogs(self): + base = (BuildConstants.REPO_ROOT / KernelBuildConstants.INPUT_CONFIG).read_text( + encoding="utf-8" + ) + fragment = ( + BuildConstants.REPO_ROOT / KernelBuildConstants.DEBUG_CONFIG_FRAGMENT + ).read_text(encoding="utf-8") + merged = build.merge_kernel_config_fragment(base, fragment).splitlines() + for setting in KernelBuildConstants.REQUIRED_DEBUG_CONFIG: + with self.subTest(setting=setting): + self.assertIn(setting, merged) + for setting in ( + "# CONFIG_DEBUG_MISC is not set", + "# CONFIG_RCU_TRACE is not set", + "# CONFIG_X86_DEBUG_FPU is not set", + "CONFIG_RCU_CPU_STALL_TIMEOUT=21", + ): + with self.subTest(setting=setting): + self.assertIn(setting, merged) + self.assertNotIn("# CONFIG_DEBUG_KERNEL is not set", merged) + for setting in KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG: + with self.subTest(time_abi=setting): + self.assertIn(setting, merged) + + def test_debug_kernel_build_applies_fragment_and_records_provenance(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + input_config = root / KernelBuildConstants.INPUT_CONFIG + input_config.parent.mkdir(parents=True) + input_config.write_text( + "\n".join( + ( + *KernelBuildConstants.REQUIRED_DIRECT_BOOT_CONFIG, + *KernelBuildConstants.REQUIRED_VIRTIO_CONSOLE_CONFIG, + *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, + *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, + *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + "# CONFIG_DEBUG_KERNEL is not set", + ) + ) + + "\n", + encoding="utf-8", + ) + fragment = root / KernelBuildConstants.DEBUG_CONFIG_FRAGMENT + fragment.write_text( + "# debug fixture\n" + + "\n".join(KernelBuildConstants.REQUIRED_DEBUG_CONFIG) + + "\n", + encoding="utf-8", + ) + patch_path = root / KernelBuildConstants.PATCH_DIRECTORY / "example.patch" + patch_path.parent.mkdir() + patch_path.write_text("patch", encoding="utf-8") + source = root / "source" + source.mkdir() + output_directory = root / "output" + output_directory.mkdir() + production_provenance = ( + output_directory / KernelBuildConstants.PROVENANCE_NAME + ) + production_provenance.write_text("production", encoding="utf-8") + config = build_config.KernelBuildConfig( + work=root / "work-debug", + output=output_directory / KernelBuildConstants.DEBUG_BINARY_NAME, + debug=True, + ) + + with patch.object(BuildConstants, "REPO_ROOT", root): + source_fingerprint = build._kernel_source_fingerprint() + + def run_build(command: object, **_kwargs: object) -> None: + if isinstance(command, list) and command[-1] == "vmlinux": + (config.work / "vmlinux").write_bytes(b"debug kernel") + + with ( + patch.object(BuildConstants, "REPO_ROOT", root), + patch.object(build, "_require_linux"), + patch.object(build, "require_tool", return_value="tool"), + patch.object( + build, + "prepare_kernel_source", + return_value=(source, source_fingerprint), + ), + patch.object(build, "run_checked", side_effect=run_build), + ): + build.build_kernel(config) + + generated = ( + output_directory / KernelBuildConstants.DEBUG_CONFIG_NAME + ).read_text(encoding="utf-8") + self.assertIn("CONFIG_DEBUG_KERNEL=y", generated.splitlines()) + self.assertNotIn("# CONFIG_DEBUG_KERNEL is not set", generated) + provenance = json.loads( + ( + output_directory / KernelBuildConstants.DEBUG_PROVENANCE_NAME + ).read_text(encoding="utf-8") + ) + self.assertEqual( + provenance["debug_config_fragment"], + { + "path": KernelBuildConstants.DEBUG_CONFIG_FRAGMENT.as_posix(), + "sha256": common.sha256_file(fragment), + }, + ) + self.assertEqual( + provenance["input_config"]["sha256"], common.sha256_file(input_config) + ) + self.assertEqual( + provenance["kernel_sha256"], + hashlib.sha256(b"debug kernel").hexdigest(), + ) + self.assertEqual( + production_provenance.read_text(encoding="utf-8"), "production" + ) + self.assertIn( + "debug_config_fragment_sha256", + (config.work / KernelBuildConstants.BUILD_STAMP_NAME).read_text( + encoding="utf-8" + ), + ) + self.assertEqual( + config.provenance_name, KernelBuildConstants.DEBUG_PROVENANCE_NAME + ) + self.assertEqual( + build_config.KernelBuildConfig().provenance_name, + KernelBuildConstants.PROVENANCE_NAME, + ) + self.assertNotIn("debug_config_fragment", build.kernel_provenance_inputs()) + + def test_guest_build_adds_debug_kernel_only_when_requested(self): + with ( + patch.object(build, "build_kernel") as kernel, + patch.object(build, "build_initramfs"), + patch.object(build, "build_docker_debug_kernel") as docker_debug, + ): + native = build_config.BuildConfig(native_guest=True, debug_kernel=True) + build.build_guest(native) + self.assertEqual( + kernel.call_args_list, + [ + call(native.kernel), + call(build_config.KernelBuildConfig.debug_variant()), + ], + ) + docker_debug.assert_not_called() + + with ( + patch.object(build, "build_docker_artifacts") as docker, + patch.object(build, "build_docker_debug_kernel") as docker_debug, + ): + portable = build_config.BuildConfig(debug_kernel=True) + build.build_guest(portable) + docker.assert_called_once_with(portable.docker, "alpine") + docker_debug.assert_called_once_with(portable.docker) + + with ( + patch.object(build, "build_docker_artifacts"), + patch.object(build, "build_docker_debug_kernel") as docker_debug, + ): + build.build_guest(build_config.BuildConfig()) + docker_debug.assert_not_called() + + debug = build_config.KernelBuildConfig.debug_variant() + self.assertTrue(debug.debug) + self.assertEqual(debug.output.name, KernelBuildConstants.DEBUG_BINARY_NAME) + self.assertEqual( + debug.work.name, KernelBuildConstants.DEBUG_WORK_DIRECTORY_NAME + ) + + def test_docker_debug_kernel_target_exports_debug_artifacts(self): + dockerfile = (BuildConstants.REPO_ROOT / "docker" / "Dockerfile").read_text( + encoding="utf-8" + ) + self.assertIn( + "FROM base AS kernel-debug\n" + "RUN python3 scripts/nvx.py build-kernel --debug\n", + dockerfile, + ) + self.assertIn( + f"FROM scratch AS {DockerBuildConstants.DEBUG_KERNEL_TARGET}", dockerfile + ) + for name in DockerBuildConstants.DEBUG_KERNEL_ARTIFACT_NAMES: + with self.subTest(artifact=name): + self.assertIn( + f"COPY --from=kernel-debug /repo/build/{name} /{name}", dockerfile + ) + + with tempfile.TemporaryDirectory() as temporary: + destination = Path(temporary) + + def export(_command: object, **_kwargs: object) -> None: + for name in DockerBuildConstants.DEBUG_KERNEL_ARTIFACT_NAMES: + (destination / name).write_bytes(b"artifact") + + config = build_config.DockerBuildConfig(artifact_destination=destination) + with ( + patch.object(build, "require_tool", return_value="docker"), + patch.object(build, "run_checked", side_effect=export) as run, + ): + build.build_docker_debug_kernel(config) + command = run.call_args.args[0] + self.assertEqual( + command[command.index("--target") + 1], + DockerBuildConstants.DEBUG_KERNEL_TARGET, + ) + + for name in DockerBuildConstants.DEBUG_KERNEL_ARTIFACT_NAMES: + (destination / name).unlink() + with ( + patch.object(build, "require_tool", return_value="docker"), + patch.object(build, "run_checked"), + self.assertRaisesRegex(common.ScriptError, "did not produce"), + ): + build.build_docker_debug_kernel(config) + def _posix_shell() -> str | None: shell = shutil.which("sh") From 3aec3060eed6aa9abe0550db1cfb88da8b92c5dd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:18:58 -0700 Subject: [PATCH 004/188] doc: fix the time ABI RCU stall release The draft released RCU stall suppression after membarrier(), assuming it waits for the grace period that was in flight at capture. Linux 6.18 skips synchronize_rcu() in MEMBARRIER_CMD_GLOBAL when one CPU is online, and with rcupdate.rcu_expedited=1 it runs an expedited grace period that does not end the in-flight normal one, so a restored guest could still report a false stall. Specify a release that works on any CPU count: set rcu_normal, then mount and unmount a private tmpfs, whose unmount waits for a full normal grace period, and release the suppression anyway after the stall timeout so a genuine stall is still reported. Add /sys/kernel/rcu_stall_count as a backstop to the capture, restore, and runtime checks, and settle the guest kernel options: drop SCHED_MC_PRIO with cpufreq, and drop the dormant KVM guest, paravirtual clock, and haltpoll options. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 45 +++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 76da731f4..6505292d9 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -670,7 +670,7 @@ CPUID and MSRs are read on every online CPU through `/dev/cpu//cpuid` and | `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | | `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | | `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | -| `C10` | The time daemon is running and has recorded no violation | capture, restore | +| `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0 | capture, restore | | `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | Capture and restore checks never wait. The boot `C7` check and the deferred @@ -716,7 +716,9 @@ contains every substring of a row: Before the boot check passes, a match is reported by `C4`. Afterwards it is a runtime violation: the daemon emits the event and powers off with status -194. +194. The daemon also reads `/sys/kernel/rcu_stall_count` at every discipline +poll and reports `G_RCU_STALL` if it is nonzero, so a stall is caught even if +its log record is lost. A violation event is one line of at most 512 bytes: @@ -769,9 +771,8 @@ in steps 2 and 3. Otherwise, in the restored process: 11. Run the restore checks (`C1`, `C2`, and `C5` on newly onlined CPUs; `C3` for CPU 0; `C6`; `C10`). They do not wait. 12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. -13. Signal the daemon to finish the restore off the latency path: - `membarrier(MEMBARRIER_CMD_GLOBAL, 0, 0)` (which waits for an RCU grace - period, so the grace period captured in flight has ended); restore the +13. Signal the daemon to finish the restore off the latency path: wait for + the [grace period release](#rcu-grace-period-release); restore the values saved in steps 2 and 3 (`G_REPAIR_SUPPRESSION`); run the deferred `C7` check; print `NVX-TIME-ABI: v=1 phase=restore status=ok`; and restart the discipline at the fast cadence. @@ -779,6 +780,28 @@ in steps 2 and 3. Otherwise, in the restored process: Steps 6, 7, and 8 run in one helper process. Repair failures emit a violation event and power off with status 195. +### RCU grace period release + +Stall suppression may be released only after the grace period that was in +flight at capture has ended, or the restored guest reports a false stall. +`membarrier(MEMBARRIER_CMD_GLOBAL)` does not guarantee this: Linux 6.18 skips +`synchronize_rcu()` when one CPU is online, and with `rcupdate.rcu_expedited=1` +it runs an expedited grace period, which does not end the in-flight normal +one. The daemon therefore: + +1. saves `/sys/kernel/rcu_normal` and writes 1, so every synchronous grace + period, expedited or not, waits for a normal grace period; +2. mounts a private `tmpfs` at `/run/nvx/rcu-sync` and unmounts it (on every + unmount, Linux 6.18 `namespace_unlock()` waits in + `synchronize_rcu_expedited()`, which now waits for a full normal grace + period started after the call, on any number of CPUs); +3. restores `/sys/kernel/rcu_normal`; and +4. releases the suppression. + +If step 2 has not returned after `rcu_cpu_stall_timeout` seconds, the daemon +releases the suppression anyway: the grace period is genuinely stuck, and the +kernel then reports the real stall. + ### Wall-clock discipline The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: @@ -953,7 +976,7 @@ fail with the listed code. | Case | Hosts | Expected | | --- | --- | --- | | Same host, immediate | All | Restored; `DOWNTIME_UTC` clear | -| Same host, downtime of 30 s (over the 21 s RCU stall timeout) | All | Restored; no RCU stall | +| Same host, downtime of 30 s (over the 21 s RCU stall timeout), at 1 and 8 vCPUs, with and without `rcupdate.rcu_expedited=1` | All | Restored; no RCU stall; `rcu_stall_count` stays 0 | | Same host under DVFS load | prometheus32 | Restored | | Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | @@ -1015,9 +1038,13 @@ Removed from OpenVMM: Removed from NVX: -- Kernel: patch 0004 (`lapic_timer_hz`), `CONFIG_CPU_FREQ`, and - `CONFIG_X86_INTEL_PSTATE`. `CONFIG_HYPERVISOR_GUEST` stays on and - `CONFIG_HYPERV` stays off. +- Kernel: patch 0004 (`lapic_timer_hz`); `CONFIG_CPU_FREQ`, + `CONFIG_X86_INTEL_PSTATE`, and `CONFIG_SCHED_MC_PRIO` (which selects both + and needs ACPI CPPC data the microVM lacks); and `CONFIG_KVM_GUEST` with + `CONFIG_PARAVIRT_CLOCK` and `CONFIG_HALTPOLL_CPUIDLE`, which are dormant + without a KVM signature. `CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` + stay on and `CONFIG_HYPERV` stays off. The CI debug kernel adds + `CONFIG_DEBUG_KERNEL` with the soft-lockup and hung-task detectors only. - Guest: RTC polling in `nvx-reseed`, and the restore packet v1 to v3 parser in `nvx-port-io`. - Harness: `tsc=reliable` and `no_timer_check` in `BASE_TUNING`; per-backend From 27c87f9a20699a70a83da62ea2a0f1c327d7dcbd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:22:05 -0700 Subject: [PATCH 005/188] doc: align the time ABI skew metrics with the warp probe The shared guest probe, nvx-time-probe warp, reports max_backward_ns from a spinlock handoff test and max_abs_offset_ns from ping-pong rounds, each run for at least 100 ms per CPU pair. Use the same names, method, and stall rule in the skew bound and in host qualification, so CI, qualification, and the specification check the same quantities. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6505292d9..cec928da9 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -345,15 +345,17 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this 4. **Backend live skew.** Each backend shows that live skew stays within the bound after release (TBD per backend; WHP is a decision gate). -The guest warp probe reports, for every ordered pair of online CPUs: +The guest warp probe (`nvx-time-probe warp` in the fleet tooling) runs two +tests on every pair of online CPUs, each for at least 100 ms per pair: -- `max_warp_ns`: the largest backward step observed when one CPU reads its - TSC after observing the other CPU's latest TSC through a shared cache line, - over at least 1,000,000 handoffs per pair; and -- `max_offset_ns`: the pairwise offset estimated by ping-pong exchanges, - taking each pair's minimum round trip. +- `max_backward_ns`: the largest backward TSC step observed when the two CPUs + alternately read the TSC under a shared spinlock (the Linux + `check_tsc_warp` method); a lower bound on skew; and +- `max_abs_offset_ns`: the largest pairwise offset estimated by ping-pong + rounds, `t2 - (t1 + t3) / 2` at the round with the minimum round trip. -Both must be at most 1,000 ns. Cycle values convert to nanoseconds with `F`. +Both must be at most 1,000 ns, and no pair may stall for more than 2 s. +Cycle values convert to nanoseconds with `F`. ## Downtime semantics and source selection @@ -922,8 +924,8 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, and the profile that `auto` selects (`E_PROFILE_HOST_UNKNOWN` fails) | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | | `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d` | -| `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_offset_ns <= 1000` | -| `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_warp_ns` and `max_offset_ns` at most 1,000 | +| `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | +| `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000 | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | | `H8` | Runner only: the runner's `cpu-` label equals the fingerprinted generation | From 71d3e067a03acb70be123c1922733e31dea1b97b Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:31:46 -0700 Subject: [PATCH 006/188] doc: apply the time ABI orchestrator decisions Detect the CPU generation at run time instead of using runner labels, bound the guest's UTC pairing uncertainty with retries and stable codes, leave the WHP invariant-TSC CPUID bit to the WHP spike, record the first WHP spike results, and accept the KVM cold_start_clocksource change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 105 +++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 26 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index cec928da9..aa518d4a9 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -124,12 +124,18 @@ Every CPU profile fixes these bits. A profile that violates them is rejected. | `0xa` | EAX, EBX, ECX, EDX | 0 (no PMU) | | `0x15`, `0x16` | EAX, EBX, ECX, EDX | 0, when within the maximum basic leaf | | `0x80000001` | EDX[27] RDTSCP | 1 | -| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only) | +| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only); on WHP, whether EDX[8] must be set is TBD(whp) | Leaves `0x15` and `0x16` are zeroed rather than synthesized. Linux takes both rates from the frequency MSRs, and zero leaves keep the profile's CPUID independent of any host's rate. +Linux trusts the TSC because of `AccessTscInvariantControls`, not because of +CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag. Until +the WHP report settles whether WHP can and must expose it, a WHP profile may +clear it, and WHP hosts qualify on measured invariance instead (see +[Host qualification](#host-qualification)). + ### Rates - **TSC.** The guest TSC runs at the destination's native rate `F_d` and is @@ -210,12 +216,12 @@ guest-visible result is identical; only the mechanism differs. | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | `ApicFrequency` property, 200 MHz — TBD(mshv) | Offloaded APIC only, 200 MHz; the emulated APIC is rejected — TBD(whp) | | TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; `IA32_TSC_ADJUST` raises #GP | Feature-bank bits cleared and CPUID bits cleared | Feature-bank bits cleared and CPUID bits cleared — TBD(whp) | -| Invariant TSC exposed | CPUID bit | `tsc_invariant_support` plus CPUID; hosts without it are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition — TBD(whp): CPUID override and qualification evidence | +| Invariant TSC exposed | CPUID bit | `tsc_invariant_support` plus CPUID; hosts without it are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition; hosts qualify on the measured warp and rate stability, and whether the CPUID bit must be set is TBD(whp) | | No paravirtual or synthetic features | No KVM leaves; no KVM Hyper-V enlightenment enabled beyond MSR routing | Synthetic processor features limited to what MSR routing needs — TBD(mshv) | `--hv` stays rejected for the microVM | | Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | VP 0 `IA32_TSC` read and host sample back to back — TBD(kvm) | TBD(mshv) | TBD(whp) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, computed from one host TSC read (Linux 5.16 or newer); no `IA32_TSC` writes | Freeze partition time, write the target to every created VP, thaw at release — TBD(mshv) | TBD(whp): suspend partition time and write the target to every VP | -| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value | Every created VP's TSC equals the target while time is frozen | TBD(whp) | -| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification | TBD(mshv) | TBD(whp): decision gate for RDTSC emulation | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, computed from one host TSC read (Linux 5.16 or newer); no `IA32_TSC` writes | Freeze partition time, write the target to every created VP, thaw at release — TBD(mshv) | Suspend partition time, write the target to every VP, resume at release (first VP run or `WHvResumePartitionTime`); `TscVirtualOffset` is unusable (writes fail) | +| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | +| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification | TBD(mshv) | Raw partitions: at most 67 ns after resume on prometheus28, `azure-windows-1`, and `azure-windows-3`, with no drift; guest-level confirmation and the RDTSC-emulation decision are TBD(whp) | | VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities | Derived from CPUID with the identity range masked: `hv1` and `kvm_clock` are false | Same | Same | | Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor — TBD(mshv) | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | @@ -271,11 +277,11 @@ an explicit `--cpu-profile` must name the same profile. (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the recorded one (`E_CPU_SURFACE`). -Initial generations are Skylake-SP (family 6, model 85: the bare-metal hosts), -Ice Lake-SP (6/106: the Xeon Platinum 8370C runners), and Emerald Rapids -(6/207: the 8573C runners). Profile IDs, contents, whether one profile serves -all backends of a generation, and the fate of the `MYSTERY_MSRS` stubs are -TBD(profiles). +Initial generations are `skylake-sp` (family 6, model 85: the bare-metal +hosts), `icelake-sp` (6/106: the Xeon Platinum 8370C runners), and +`emeraldrapids` (6/207: the 8573C runners). Profile IDs, contents, whether one +profile serves all backends of a generation, and the fate of the +`MYSTERY_MSRS` stubs are TBD(profiles). ## TSC rate policy and LAPIC rate rule @@ -632,6 +638,27 @@ a running VP's TSC on every backend, and frozen time on MSHV and WHP thaws at an unobservable instant, so the guest forms the UTC and TSC pair itself through this bracket; its `CLOCK_REALTIME` is derived from the TSC. +### Uncertainty bounds + +The guest accepts a pairing only if `epsilon` is within a bound, retries +otherwise, and then fails with a stable code: + +| Use | Bound | Attempts | When every attempt exceeds the bound | +| --- | --- | --- | --- | +| Restore repair (packet bracket, then time samples) | 1 ms | 3 | `G_REPAIR_SAMPLE`: event and power-off with status 195 | +| Initial synchronization at boot (time samples) | 1 ms | 3 | `G_CONFORMANCE_C12`: event and power-off with status 193 | +| Discipline poll (time samples) | 50 µs | 3 | `G_SAMPLE_UNCERTAIN`: the poll is skipped and the code is recorded in the state file and on the console; never fatal | + +`epsilon` is half the round trip of one PMIO write exit plus two +`CLOCK_REALTIME` reads, so it is expected to stay far below both bounds on +every backend. The backend agents measure it: + +| Backend | Expected `epsilon` (p50 / p99) | +| --- | --- | +| KVM | TBD(kvm) | +| MSHV | TBD(mshv) | +| WHP | TBD(whp) | + ### Generation counter and generation ID - The generation counter `g` is 0 in a cold-booted process and @@ -667,13 +694,14 @@ CPUID and MSRs are read on every online CPU through `/dev/cpu//cpuid` and | `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | | `C3` | MSR `0x40000002` equals the CPU's VP index; `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz; `0x40000023` equals 1,000,000,000 or 200,000,000; `0x40000118` equals 1; reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | | `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | -| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU | boot, restore (new CPUs) | +| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU (`nonstop_tsc` on WHP: TBD(whp)) | boot, restore (new CPUs) | | `C6` | `current_clocksource` is `tsc`; `available_clocksource` contains only `tsc`, `refined-jiffies`, and `jiffies` | boot, capture, restore | | `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | | `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | | `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | | `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0 | capture, restore | | `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | +| `C12` | A time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | Capture and restore checks never wait. The boot `C7` check and the deferred restore `C7` check may poll for at most 200 ms, because a CPU switches to @@ -690,9 +718,10 @@ NVX-TIME-ABI: v=1 phase= status=ok cpus= tsc_hz= detail=""` in the same format, emits a violation event with code `G_CONFORMANCE_`, and powers -off with status 193. After the boot check passes, init takes one time sample -and steps the clock to host UTC before any workload starts; this initial step -is not counted as a discontinuity. +off with status 193. The boot step of `C12` steps the clock before any +workload starts and is not counted as a discontinuity. The only other line +with this prefix is the non-fatal `phase=runtime status=uncertain` line of +the [wall-clock discipline](#wall-clock-discipline). ### Violation watcher @@ -811,7 +840,11 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: - **Cadence.** A poll every 16 s for the first four polls after the boot check or a restore, then every 64 s. - **Sample.** Up to three time samples per poll; keep the first with - `epsilon <= 50 µs`. Otherwise skip the poll and count a rejected sample. + `epsilon <= 50 µs`. Otherwise skip the poll, count a rejected sample, set + `last_sample_error=G_SAMPLE_UNCERTAIN`, and print + `NVX-TIME-ABI: v=1 phase=runtime status=uncertain code=G_SAMPLE_UNCERTAIN + epsilon_ns=` on the console. The next accepted sample resets + `last_sample_error` to `none`. A sample whose `g` differs from the recorded `g` is discarded: a restore happened, and restore repair resets the discipline. - **Step.** If `|theta| >= 128 ms`, apply `ADJ_SETOFFSET | ADJ_NANO` with @@ -849,6 +882,7 @@ atomically with `rename(2)` after every change. The sandbox agent bind-mounts | `uncertainty_ns` | Last accepted `epsilon` | | `frequency_ppb` | Current kernel frequency correction | | `samples`, `rejected_samples` | Sample counters | +| `last_sample_error` | `none` or `G_SAMPLE_UNCERTAIN` | | `violations` | 0; a violation powers the guest off | Workloads that need prompt notice of a step can also arm a `timerfd` with @@ -904,10 +938,13 @@ signals, and 255): | Status | Class | Guest codes | | ---: | --- | --- | -| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C11` | +| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C12` | | 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_KMSG_OVERRUN` | | 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | +`G_SAMPLE_UNCERTAIN` is the only non-fatal guest code: the discipline records +it and continues (see [Uncertainty bounds](#uncertainty-bounds)). + A workload can exit with any 8-bit status, so a harness classifies a time failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. @@ -921,19 +958,28 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | ID | Check | | --- | --- | | `H1` | The backend device or API is present and usable | -| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, and the profile that `auto` selects (`E_PROFILE_HOST_UNKNOWN` fails) | +| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`) | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | | `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d` | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | | `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000 | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | -| `H8` | Runner only: the runner's `cpu-` label equals the fingerprinted generation | -Runners carry exactly one generation label, `cpu-`, whose value is -the profile's generation key: Skylake-SP, Ice Lake-SP (8370C), or Emerald -Rapids (8573C); exact label values are TBD(profiles). Same-generation restore -tests select runners by label. Labels are assigned when a runner is -registered; agents never change runner registration. +Qualification gates on these measured properties and on the profile's +features. On WHP in particular, the warp probe (`H6`) and rate stability +(`H4`) are the evidence for TSC invariance; whether WHP must also expose the +CPUID invariant-TSC bit is TBD(whp). + +Generation names used in logs, reports, and job summaries are `skylake-sp` +(family 6, model 85), `icelake-sp` (6/106), and `emeraldrapids` (6/207). +`validate-runner` and `nvx.py doctor` detect the generation at run time and +report it, together with the selected profile, `F_d`, `L`, and the skew +metrics, in their log and the job summary; an unknown generation fails +qualification explicitly. Runner labels are not used and runners are not +re-registered: per-PR CI captures and restores on the same runner, and +same-generation cross-VM restore is validated by the fleet restore matrix +(`p6-restore-matrix`) on the registered hosts. Routing CI jobs by generation +labels is optional future work. ## Performance expectations and acceptance gate @@ -973,7 +1019,9 @@ at 1, 2, 4, and 8 vCPUs, plus the exhaustive CI check and the warp probe on every backend. **Restore matrix.** Every case runs with zero violations; rejected cases must -fail with the listed code. +fail with the listed code. Per-PR CI runs the same-host cases on one runner; +the full matrix, including the cross-VM cases, runs on the registered hosts +as the fleet restore matrix. | Case | Hosts | Expected | | --- | --- | --- | @@ -983,6 +1031,7 @@ fail with the listed code. | Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | +| Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | | Across VMs of one generation | `azure-kvm-1` to `-2`; `azure-azlinux-3` to `-4`; `azure-windows-1` to `-2`; `azure-windows-3` to `-4` | Restored | | Across generations | `azure-windows-1` (8370C) to `-3` (8573C) | `E_CPU_GENERATION` | | Host without invariant TSC | `azure-azlinux-2` | Cold boot and restore rejected: `E_PROFILE_UNSUPPORTED` | @@ -1013,6 +1062,7 @@ repeatable): | `dest-rate-offset-ppm=` | Perturb the measured `F_d` used by the rate policy and the reported deviation; the TSC is never scaled | | `downtime-add-s=` | Add `n` seconds to the measured `D` before the bounds check | | `utc-offset-ms=` | Add `n` milliseconds to every destination UTC reading: the downtime sample, the packet, and time samples | +| `sample-delay-us=` | Delay the handling of the first `0xa5` write and of every `0xa7` write by `n` µs, which widens the guest's bracket | Every active hook is logged at warning level and sets `TEST_HOOKS` in the packet and in every time sample. @@ -1072,9 +1122,12 @@ Migration impact: profile of their generation pins. - Hosts that fail qualification, such as `azure-azlinux-2`, cannot run microVMs until replaced. -- Same-generation restore tests need `cpu-` runner labels. +- Per-PR CI captures and restores on the same runner. Same-generation + cross-VM restore is validated by the fleet restore matrix on the + registered hosts, not in per-PR CI. - The `cold_start_clocksource` metric on KVM changes meaning (from - `kvm-clock` to `tsc`). + `kvm-clock` to `tsc`). The change is accepted without an exemption; a gate + failure on it is investigated as a regression. - Documentation updates: this document, [Snapshot and restore](snapshot-and-restore.md), [Machine and device ABI](machine-and-device-abi.md), [Cold boot](cold-boot.md), the From 4fd86484abd7973fc169f08deb8bc94cbcc9cdac Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:41:52 -0700 Subject: [PATCH 007/188] guest: add the nvx-time-probe guest time probe Time ABI v1 stops Linux from checking cross-vCPU TSC consistency (TSC_RELIABLE), so the 1 us skew bound must be enforced outside the kernel. CI, host qualification, and the spec use one guest warp probe for that: nvx-time-probe warp reports max_backward_ns from a spinlock handoff test and max_abs_offset_ns from ping-pong rounds for every CPU pair. It replaces restore-tsc-sync's reliance on the kernel's own warp check. Its latency and clock commands report TSC and clock read costs, clocksources, TSC flags, the hypervisor identity, and clock event devices. Add the perf agent's fleet probe (version 1.1, MIT) as guest/common/nvx-time-probe.c and build it from source into both guest initramfs images as /sbin/nvx-time-probe. This copy is now canonical. The only change from the fleet source is a 64-byte clock event device buffer: the parsed name holds up to 63 bytes, and gcc 12 at -O2 rejects the truncating snprintf under -Werror. The probe needs threads and was written for -O2, so it builds with its own flags instead of the -Os helper flags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 8 +- guest/common/nvx-time-probe.c | 1763 ++++++++++++++++++++++++++ scripts/nvx_tools/build.py | 10 +- scripts/nvx_tools/build_constants.py | 11 + scripts/test_nvx_tools.py | 55 + 5 files changed, 1844 insertions(+), 3 deletions(-) create mode 100644 guest/common/nvx-time-probe.c diff --git a/doc/build.md b/doc/build.md index 61297f0fa..696f7d2c2 100644 --- a/doc/build.md +++ b/doc/build.md @@ -139,8 +139,12 @@ complete per-scenario logs under `build/test-results/microvm` by default. The Alpine initramfs includes the sandbox PID-1 bootstrap, its container namespace -helpers, the static `nvx-device-io` benchmark helper, and the static -`nvx-port-io` restore packet helper under `/sbin`. The matching kernel enables +helpers, the static `nvx-device-io` benchmark helper, the static `nvx-port-io` +restore packet helper, and the static `nvx-time-probe` guest time probe under +`/sbin`. The probe measures TSC and clock read latency, reports the guest's +clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU +warp test that the [time ABI](design/time-abi.md#cross-vcpu-skew-bound) uses to +enforce its 1 µs skew bound. The matching kernel enables virtio-blk, compressed EROFS, overlayfs, ext4 scratch, memory cgroups, and cgroup BPF. The build fails if `olddefconfig` drops any required option. The APK manifest records the `blkid` and `util-linux` tools used by the bootstrap diff --git a/guest/common/nvx-time-probe.c b/guest/common/nvx-time-probe.c new file mode 100644 index 000000000..f77f9b254 --- /dev/null +++ b/guest/common/nvx-time-probe.c @@ -0,0 +1,1763 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +// nvx-time-probe: guest-side time diagnostics for the NVX time ABI. +// +// latency ns/op of TSC reads and clock_gettime(), p50/p99. +// warp pairwise cross-CPU TSC warp and offset checks. +// clock clocks, clocksources, TSC flags, hypervisor identity, and +// per-CPU clock event devices. +// +// Build (static, x86-64): cc -O2 -static -pthread -o nvx-time-probe +// nvx-time-probe.c + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define PROBE_VERSION "1.1" +#define EXIT_FAIL 1 +#define EXIT_USAGE 2 +#define EXIT_ERROR 3 +#define MAX_CPUS 1024 +#define KMSG_RECORD_MAX 8192 +#define SYSLOG_ACTION_READ_ALL 3 +#define SYSLOG_ACTION_SIZE_BUFFER 10 +#define CLOCKSOURCE_DIR "/sys/devices/system/clocksource/clocksource0" + +struct tsc_rate { + double hz; + const char *source; +}; + +static struct tsc_rate g_tsc; +static volatile uint64_t g_sink; + +// --------------------------------------------------------------------------- +// CPU primitives +// --------------------------------------------------------------------------- + +static inline uint64_t rdtsc_plain(void) +{ + uint32_t lo, hi; + + __asm__ __volatile__("rdtsc" : "=a"(lo), "=d"(hi)); + return ((uint64_t)hi << 32) | lo; +} + +static inline uint64_t rdtscp_plain(void) +{ + uint32_t lo, hi, aux; + + __asm__ __volatile__("rdtscp" : "=a"(lo), "=d"(hi), "=c"(aux)); + return ((uint64_t)hi << 32) | lo; +} + +// Linux rdtsc_ordered(): the read cannot pass earlier loads. +static inline uint64_t rdtsc_lfence(void) +{ + uint32_t lo, hi; + + __asm__ __volatile__("lfence\n\trdtsc" : "=a"(lo), "=d"(hi)::"memory"); + return ((uint64_t)hi << 32) | lo; +} + +// Fully fenced read for timing brackets: later work cannot start before it. +static inline uint64_t rdtsc_fenced(void) +{ + uint32_t lo, hi; + + __asm__ __volatile__("lfence\n\trdtsc\n\tlfence" + : "=a"(lo), "=d"(hi)::"memory"); + return ((uint64_t)hi << 32) | lo; +} + +static inline void cpu_pause(void) +{ + __asm__ __volatile__("pause" ::: "memory"); +} + +static void cpuid(uint32_t leaf, uint32_t subleaf, uint32_t regs[4]) +{ + __asm__ __volatile__("cpuid" + : "=a"(regs[0]), "=b"(regs[1]), "=c"(regs[2]), + "=d"(regs[3]) + : "a"(leaf), "c"(subleaf)); +} + +static bool cpu_has_rdtscp(void) +{ + uint32_t regs[4]; + + cpuid(0x80000000u, 0, regs); + if (regs[0] < 0x80000001u) + return false; + cpuid(0x80000001u, 0, regs); + return (regs[3] >> 27) & 1; +} + +static int pin_self(int cpu) +{ + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (sched_setaffinity(0, sizeof(set), &set) != 0) + return -1; + if (sched_getcpu() != cpu) { + errno = EINVAL; + return -1; + } + return 0; +} + +// --------------------------------------------------------------------------- +// Small helpers +// --------------------------------------------------------------------------- + +static double cycles_to_ns(double cycles) +{ + return g_tsc.hz > 0 ? cycles * 1e9 / g_tsc.hz : 0; +} + +static long long round_ll(double value) +{ + return (long long)(value >= 0 ? value + 0.5 : value - 0.5); +} + +static int read_first_line(const char *path, char *buf, size_t size) +{ + FILE *file = fopen(path, "r"); + + if (!file) + return -1; + if (!fgets(buf, (int)size, file)) { + fclose(file); + return -1; + } + fclose(file); + buf[strcspn(buf, "\n")] = '\0'; + return 0; +} + +static int cmp_double(const void *a, const void *b) +{ + double x = *(const double *)a; + double y = *(const double *)b; + + return (x > y) - (x < y); +} + +static int cmp_i64(const void *a, const void *b) +{ + int64_t x = *(const int64_t *)a; + int64_t y = *(const int64_t *)b; + + return (x > y) - (x < y); +} + +// Nearest-rank percentile over a sorted array, as the NVX harness reports. +static double nearest_rank(const double *sorted, size_t count, double pct) +{ + double exact = pct * (double)count; + size_t rank = (size_t)exact; + + if ((double)rank < exact) + rank++; + if (rank < 1) + rank = 1; + if (rank > count) + rank = count; + return sorted[rank - 1]; +} + +static int parse_u64_arg(const char *text, uint64_t *value) +{ + char *end; + + errno = 0; + *value = strtoull(text, &end, 0); + return errno == 0 && end != text && *end == '\0' ? 0 : -1; +} + +static int parse_cpu_list(const char *text, int *cpus, int max) +{ + int count = 0; + const char *p = text; + + while (*p) { + char *end; + long first = strtol(p, &end, 10); + long last = first; + + if (end == p || first < 0 || first >= MAX_CPUS) + return -1; + p = end; + if (*p == '-') { + last = strtol(p + 1, &end, 10); + if (end == p + 1 || last < first || last >= MAX_CPUS) + return -1; + p = end; + } + for (long cpu = first; cpu <= last; cpu++) { + for (int i = 0; i < count; i++) + if (cpus[i] == cpu) + return -1; + if (count >= max) + return -1; + cpus[count++] = (int)cpu; + } + if (*p == ',') + p++; + else if (*p) + return -1; + } + return count; +} + +static int affinity_cpus(int *cpus, int max) +{ + cpu_set_t set; + int count = 0; + + if (sched_getaffinity(0, sizeof(set), &set) != 0) + return -1; + for (int cpu = 0; cpu < CPU_SETSIZE && count < max; cpu++) + if (CPU_ISSET(cpu, &set)) + cpus[count++] = cpu; + return count; +} + +// Rejects CPUs outside the affinity mask (offline or excluded) before any +// measurement starts. +static int check_cpus_allowed(const char *command, const int *cpus, int count) +{ + cpu_set_t allowed; + + if (sched_getaffinity(0, sizeof(allowed), &allowed) != 0) { + fprintf(stderr, "nvx-time-probe %s: sched_getaffinity: %s\n", command, + strerror(errno)); + return -1; + } + for (int i = 0; i < count; i++) { + if (cpus[i] >= CPU_SETSIZE || !CPU_ISSET(cpus[i], &allowed)) { + fprintf(stderr, + "nvx-time-probe %s: cpu %d is not in the affinity mask" + " (offline or excluded)\n", + command, cpus[i]); + return -1; + } + } + return 0; +} + +static void format_cpu_list(const int *cpus, int count, char *buf, size_t size) +{ + size_t used = 0; + + buf[0] = '\0'; + for (int i = 0; i < count && used < size; i++) { + int first = cpus[i]; + int last = first; + + while (i + 1 < count && cpus[i + 1] == last + 1) + last = cpus[++i]; + used += (size_t)snprintf(buf + used, size - used, "%s%d", used ? "," : "", + first); + if (last != first && used < size) + used += (size_t)snprintf(buf + used, size - used, "-%d", last); + } +} + +// --------------------------------------------------------------------------- +// Kernel log access (/dev/kmsg, falling back to syslog(2)) +// --------------------------------------------------------------------------- + +typedef void (*kmsg_fn)(uint64_t usec, const char *text, void *ctx); + +static int kmsg_from_devkmsg(kmsg_fn fn, void *ctx) +{ + char record[KMSG_RECORD_MAX]; + int count = 0; + int fd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC); + + if (fd < 0) + return -1; + for (;;) { + ssize_t n = read(fd, record, sizeof(record) - 1); + char *semi; + char *p; + int field = 0; + + if (n < 0 && (errno == EINTR || errno == EPIPE)) + continue; + if (n <= 0) + break; + record[n] = '\0'; + // Record format: "prio,seq,usec,flags[,...];message\n[ KEY=VALUE\n]". + semi = strchr(record, ';'); + if (!semi) + continue; + for (p = record; p < semi && field < 2; p++) + if (*p == ',') + field++; + record[strcspn(record, "\n")] = '\0'; + fn(strtoull(p, NULL, 10), semi + 1, ctx); + count++; + } + close(fd); + return count > 0 ? 0 : -1; +} + +static int kmsg_from_syslog(kmsg_fn fn, void *ctx) +{ + int size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0); + char *buf; + char *line; + int n; + + if (size <= 0) + size = 1 << 20; + buf = malloc((size_t)size + 1); + if (!buf) + return -1; + n = klogctl(SYSLOG_ACTION_READ_ALL, buf, size); + if (n < 0) { + free(buf); + return -1; + } + buf[n] = '\0'; + for (line = buf; line && *line;) { + char *next = strchr(line, '\n'); + char *p = line; + uint64_t usec = 0; + + if (next) + *next++ = '\0'; + if (*p == '<') { + char *close_angle = strchr(p, '>'); + + if (close_angle) + p = close_angle + 1; + } + if (*p == '[') { + char *close_bracket = strchr(p, ']'); + + if (close_bracket) { + usec = (uint64_t)(strtod(p + 1, NULL) * 1e6 + 0.5); + p = close_bracket + 1; + if (*p == ' ') + p++; + } + } + fn(usec, p, ctx); + line = next; + } + free(buf); + return 0; +} + +static int kmsg_foreach(kmsg_fn fn, void *ctx) +{ + if (kmsg_from_devkmsg(fn, ctx) == 0) + return 0; + return kmsg_from_syslog(fn, ctx); +} + +// --------------------------------------------------------------------------- +// /proc/cpuinfo and TSC rate discovery +// --------------------------------------------------------------------------- + +static bool cpuinfo_has_flag(const char *flag) +{ + static char flags[8192]; + static bool loaded; + size_t len = strlen(flag); + const char *p; + + if (!loaded) { + char line[8192]; + FILE *file = fopen("/proc/cpuinfo", "r"); + + loaded = true; + while (file && fgets(line, sizeof(line), file)) { + if (strncmp(line, "flags", 5) == 0 && strchr(line, ':')) { + snprintf(flags, sizeof(flags), " %s", strchr(line, ':') + 1); + flags[strcspn(flags, "\n")] = '\0'; + strncat(flags, " ", sizeof(flags) - strlen(flags) - 1); + break; + } + } + if (file) + fclose(file); + } + for (p = flags; (p = strstr(p, flag)) != NULL; p += len) + if (p[-1] == ' ' && p[len] == ' ') + return true; + return false; +} + +static double cpuinfo_mhz(void) +{ + char line[512]; + double mhz = 0; + FILE *file = fopen("/proc/cpuinfo", "r"); + + if (!file) + return 0; + while (fgets(line, sizeof(line), file)) { + if (strncmp(line, "cpu MHz", 7) == 0 && strchr(line, ':')) { + mhz = strtod(strchr(line, ':') + 1, NULL); + break; + } + } + fclose(file); + return mhz; +} + +struct kmsg_tsc { + double refined_mhz; + double detected_tsc_mhz; + double detected_cpu_mhz; +}; + +static double mhz_after(const char *text, const char *prefix, const char *suffix) +{ + const char *p = strstr(text, prefix); + char *end; + double mhz; + + if (!p) + return 0; + p += strlen(prefix); + mhz = strtod(p, &end); + if (end == p || strncmp(end, suffix, strlen(suffix)) != 0) + return 0; + return mhz; +} + +static void kmsg_tsc_cb(uint64_t usec, const char *text, void *ctx) +{ + struct kmsg_tsc *k = ctx; + double mhz; + + (void)usec; + if ((mhz = mhz_after(text, "Refined TSC clocksource calibration: ", " MHz")) > 0) + k->refined_mhz = mhz; + else if ((mhz = mhz_after(text, "tsc: Detected ", " MHz TSC")) > 0) + k->detected_tsc_mhz = mhz; + else if ((mhz = mhz_after(text, "tsc: Detected ", " MHz processor")) > 0) + k->detected_cpu_mhz = mhz; +} + +// The kernel prints kHz precision as "NNNN.NNN MHz". +static double mhz_to_hz(double mhz) +{ + return (double)(uint64_t)(mhz * 1000.0 + 0.5) * 1000.0; +} + +static double measure_tsc_hz(unsigned ms) +{ + struct timespec a, b; + struct timespec pause = {ms / 1000, (long)(ms % 1000) * 1000000L}; + uint64_t t0, t1; + double ns; + + clock_gettime(CLOCK_MONOTONIC_RAW, &a); + t0 = rdtsc_fenced(); + nanosleep(&pause, NULL); + clock_gettime(CLOCK_MONOTONIC_RAW, &b); + t1 = rdtsc_fenced(); + ns = (double)(b.tv_sec - a.tv_sec) * 1e9 + (double)(b.tv_nsec - a.tv_nsec); + return ns > 0 ? (double)(t1 - t0) * 1e9 / ns : 0; +} + +// Precedence: option, sysfs tsc_freq_khz, kernel log (refined, then detected +// TSC, then detected processor), "cpu MHz" without APERF/MPERF, measurement. +static void detect_tsc_rate(double override_hz) +{ + struct kmsg_tsc k = {0}; + char line[64]; + + if (override_hz > 0) { + g_tsc = (struct tsc_rate){override_hz, "option"}; + return; + } + if (read_first_line("/sys/devices/system/cpu/cpu0/tsc_freq_khz", line, + sizeof(line)) == 0 && + strtoull(line, NULL, 10) > 0) { + g_tsc = (struct tsc_rate){(double)strtoull(line, NULL, 10) * 1000.0, + "sysfs-tsc_freq_khz"}; + return; + } + kmsg_foreach(kmsg_tsc_cb, &k); + if (k.refined_mhz > 0) { + g_tsc = (struct tsc_rate){mhz_to_hz(k.refined_mhz), "kmsg-refined"}; + return; + } + if (k.detected_tsc_mhz > 0) { + g_tsc = (struct tsc_rate){mhz_to_hz(k.detected_tsc_mhz), + "kmsg-detected-tsc"}; + return; + } + if (k.detected_cpu_mhz > 0) { + g_tsc = (struct tsc_rate){mhz_to_hz(k.detected_cpu_mhz), + "kmsg-detected-processor"}; + return; + } + // Without APERF/MPERF, "cpu MHz" reports cpu_khz, which equals tsc_khz. + if (!cpuinfo_has_flag("aperfmperf") && cpuinfo_mhz() > 0) { + g_tsc = (struct tsc_rate){mhz_to_hz(cpuinfo_mhz()), "cpuinfo-cpu-mhz"}; + return; + } + g_tsc = (struct tsc_rate){measure_tsc_hz(200), "measured-monotonic-raw"}; +} + +static void current_clocksource(char *buf, size_t size) +{ + if (read_first_line(CLOCKSOURCE_DIR "/current_clocksource", buf, size) != 0) + snprintf(buf, size, "unknown"); +} + +// --------------------------------------------------------------------------- +// Option parsing +// --------------------------------------------------------------------------- + +// Matches "--name value" or "--name=value" at argv[*index]. +static const char *option_value(int argc, char **argv, int *index, const char *name) +{ + const char *arg = argv[*index]; + size_t len = strlen(name); + + if (strncmp(arg, name, len) != 0) + return NULL; + if (arg[len] == '=') + return arg + len + 1; + if (arg[len] == '\0' && *index + 1 < argc) + return argv[++*index]; + return NULL; +} + +static int parse_hz_arg(const char *text, double *hz) +{ + char *end; + + errno = 0; + *hz = strtod(text, &end); + return errno == 0 && end != text && *end == '\0' && *hz > 0 ? 0 : -1; +} + +static int bad_option(const char *command, const char *arg) +{ + fprintf(stderr, "nvx-time-probe %s: invalid option or value: %s\n", command, arg); + return EXIT_USAGE; +} + +// --------------------------------------------------------------------------- +// latency +// --------------------------------------------------------------------------- + +enum method { + M_RDTSC, + M_RDTSCP, + M_LFENCE_RDTSC, + M_CLOCK_MONOTONIC, + M_CLOCK_REALTIME, + M_SYSCALL_MONOTONIC, + M_SYSCALL_GETPID, + M_SCHED_YIELD, + M_COUNT, +}; + +static const char *const method_names[M_COUNT] = { + "rdtsc", + "rdtscp", + "lfence_rdtsc", + "clock_gettime_monotonic", + "clock_gettime_realtime", + "syscall_clock_gettime_monotonic", + "syscall_getpid", + "sched_yield", +}; + +struct latency_result { + double p50; + double p99; + double min; + double mean; + size_t samples; +}; + +static uint64_t run_batch(enum method method, unsigned batch) +{ + struct timespec ts; + uint64_t sink = 0; + unsigned i; + + switch (method) { + case M_RDTSC: + for (i = 0; i < batch; i++) + sink += rdtsc_plain(); + break; + case M_RDTSCP: + for (i = 0; i < batch; i++) + sink += rdtscp_plain(); + break; + case M_LFENCE_RDTSC: + for (i = 0; i < batch; i++) + sink += rdtsc_lfence(); + break; + case M_CLOCK_MONOTONIC: + for (i = 0; i < batch; i++) { + clock_gettime(CLOCK_MONOTONIC, &ts); + sink += (uint64_t)ts.tv_nsec; + } + break; + case M_CLOCK_REALTIME: + for (i = 0; i < batch; i++) { + clock_gettime(CLOCK_REALTIME, &ts); + sink += (uint64_t)ts.tv_nsec; + } + break; + case M_SYSCALL_MONOTONIC: + for (i = 0; i < batch; i++) { + syscall(SYS_clock_gettime, CLOCK_MONOTONIC, &ts); + sink += (uint64_t)ts.tv_nsec; + } + break; + case M_SYSCALL_GETPID: + for (i = 0; i < batch; i++) + sink += (uint64_t)syscall(SYS_getpid); + break; + case M_SCHED_YIELD: + for (i = 0; i < batch; i++) + sink += (uint64_t)sched_yield(); + break; + default: + break; + } + return sink; +} + +// Median cost of an empty fenced timing bracket, subtracted from samples. +static double bracket_overhead_cycles(size_t iterations) +{ + double *samples = malloc(iterations * sizeof(*samples)); + double median; + + if (!samples) + return 0; + for (size_t i = 0; i < iterations; i++) { + uint64_t t0 = rdtsc_fenced(); + uint64_t t1 = rdtsc_fenced(); + + samples[i] = (double)(t1 - t0); + } + qsort(samples, iterations, sizeof(*samples), cmp_double); + median = nearest_rank(samples, iterations, 0.50); + free(samples); + return median; +} + +static int measure_method(enum method method, size_t iterations, unsigned batch, + double overhead_cycles, struct latency_result *out) +{ + double *per_op = malloc(iterations * sizeof(*per_op)); + double sum = 0; + + if (!per_op) + return -1; + g_sink += run_batch(method, batch); + for (size_t i = 0; i < iterations; i++) { + uint64_t t0 = rdtsc_fenced(); + uint64_t sink = run_batch(method, batch); + uint64_t t1 = rdtsc_fenced(); + double cycles = (double)(t1 - t0) - overhead_cycles; + + g_sink += sink; + if (cycles < 0) + cycles = 0; + per_op[i] = cycles_to_ns(cycles) / batch; + sum += per_op[i]; + } + qsort(per_op, iterations, sizeof(*per_op), cmp_double); + out->p50 = nearest_rank(per_op, iterations, 0.50); + out->p99 = nearest_rank(per_op, iterations, 0.99); + out->min = per_op[0]; + out->mean = sum / (double)iterations; + out->samples = iterations; + free(per_op); + return 0; +} + +static int cmd_latency(int argc, char **argv) +{ + uint64_t iterations = 2000; + uint64_t batch = 50; + uint64_t cpu_arg = UINT64_MAX; + double tsc_hz = 0; + struct latency_result results[M_COUNT] = {0}; + char clocksource[64]; + bool has_rdtscp = cpu_has_rdtscp(); + bool has_vdso = getauxval(AT_SYSINFO_EHDR) != 0; + double overhead; + int cpu; + + for (int i = 2; i < argc; i++) { + const char *value; + + if ((value = option_value(argc, argv, &i, "--iterations"))) { + if (parse_u64_arg(value, &iterations) || iterations == 0 || + iterations > 10000000) + return bad_option("latency", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--batch"))) { + if (parse_u64_arg(value, &batch) || batch == 0 || batch > 1000000) + return bad_option("latency", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--cpu"))) { + if (parse_u64_arg(value, &cpu_arg) || cpu_arg >= MAX_CPUS) + return bad_option("latency", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--tsc-hz"))) { + if (parse_hz_arg(value, &tsc_hz)) + return bad_option("latency", argv[i]); + } else { + return bad_option("latency", argv[i]); + } + } + + cpu = cpu_arg == UINT64_MAX ? sched_getcpu() : (int)cpu_arg; + if (cpu >= 0 && check_cpus_allowed("latency", &cpu, 1) != 0) + return EXIT_USAGE; + if (cpu < 0 || pin_self(cpu) != 0) { + fprintf(stderr, "nvx-time-probe latency: cannot pin to cpu %d: %s\n", cpu, + strerror(errno)); + return EXIT_ERROR; + } + detect_tsc_rate(tsc_hz); + if (g_tsc.hz <= 0) { + fprintf(stderr, "nvx-time-probe latency: unknown TSC rate; pass --tsc-hz\n"); + return EXIT_ERROR; + } + current_clocksource(clocksource, sizeof(clocksource)); + overhead = bracket_overhead_cycles((size_t)iterations); + + printf("nvx-time-probe %s latency: cpu=%d iterations=%" PRIu64 " batch=%" PRIu64 + " tsc_hz=%.0f tsc_hz_source=%s clocksource=%s vdso=%d" + " bracket_overhead_ns=%.1f\n", + PROBE_VERSION, cpu, iterations, batch, g_tsc.hz, g_tsc.source, + clocksource, has_vdso, cycles_to_ns(overhead)); + printf("%-32s %10s %10s %10s %10s\n", "method", "p50_ns", "p99_ns", "min_ns", + "mean_ns"); + for (int m = 0; m < M_COUNT; m++) { + if (m == M_RDTSCP && !has_rdtscp) { + printf("%-32s unsupported (CPUID.80000001H:EDX[27] clear)\n", + method_names[m]); + continue; + } + if (measure_method((enum method)m, (size_t)iterations, (unsigned)batch, + overhead, &results[m]) != 0) { + fprintf(stderr, "nvx-time-probe latency: out of memory\n"); + return EXIT_ERROR; + } + printf("%-32s %10.1f %10.1f %10.1f %10.1f\n", method_names[m], + results[m].p50, results[m].p99, results[m].min, results[m].mean); + fflush(stdout); + } + for (int m = 0; m < M_COUNT; m++) { + if (!results[m].samples) { + printf("NVX-TIME-PROBE latency method=%s unsupported=1\n", method_names[m]); + continue; + } + printf("NVX-TIME-PROBE latency method=%s p50_ns=%.1f p99_ns=%.1f min_ns=%.1f" + " mean_ns=%.1f samples=%zu batch=%" PRIu64 " cpu=%d clocksource=%s\n", + method_names[m], results[m].p50, results[m].p99, results[m].min, + results[m].mean, results[m].samples, batch, cpu, clocksource); + } + return 0; +} + +// --------------------------------------------------------------------------- +// warp +// --------------------------------------------------------------------------- + +#define PP_STOP UINT64_MAX +#define PP_MAX_SAMPLES (1u << 18) +#define STALL_SECONDS 2.0 +#define SPIN_CHECK_MASK 0xfffffu + +// (a) Linux check_tsc_warp(): both CPUs take one spinlock, read the TSC, and +// compare it with the last value either CPU stored. A backward step proves a +// skew larger than the lock handoff latency, so it is a lower bound. +static struct { + _Alignas(64) atomic_int lock; + uint64_t last_tsc; + uint64_t max_warp; + uint64_t warps; + _Alignas(64) atomic_int ready; + atomic_int abort; + atomic_int stop; +} g_warp; + +// (b) Ping-pong: cpu_a reads t1 and signals, cpu_b reads t2 and answers, and +// cpu_a reads t3. The offset of cpu_b relative to cpu_a lies in [t2-t3, t2-t1]; +// the point estimate is t2-(t1+t3)/2 with uncertainty (t3-t1)/2. +static struct { + _Alignas(64) _Atomic uint64_t seq; + _Atomic uint64_t t2; + _Alignas(64) atomic_int ready; + atomic_int abort; +} g_pp; + +struct pair_result { + int cpu_a; + int cpu_b; + uint64_t warp_iterations; + uint64_t warps; + uint64_t max_warp_cycles; + uint64_t rounds; + uint64_t min_rtt_cycles; + int64_t min_rtt_offset_x2; + int64_t lo_cycles; + int64_t hi_cycles; + double median_offset_cycles; + double offset_cycles; + double uncertainty_cycles; + bool consistent; + bool stalled; +}; + +struct thread_arg { + int cpu; + bool leader; + uint64_t budget_cycles; + uint64_t stall_cycles; + uint64_t iterations; + int pin_errno; + struct pair_result *result; + int64_t *offsets_x2; +}; + +static bool pair_barrier(atomic_int *ready, atomic_int *abort_flag, bool failed) +{ + if (failed) + atomic_store(abort_flag, 1); + atomic_fetch_add(ready, 1); + while (atomic_load(ready) < 2) + cpu_pause(); + return !atomic_load(abort_flag); +} + +static uint64_t coarse_ns(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC_COARSE, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec; +} + +static void *warp_thread(void *p) +{ + struct thread_arg *arg = p; + bool pinned = pin_self(arg->cpu) == 0; + uint64_t end; + uint64_t deadline_ns; + uint64_t i; + + if (!pinned) + arg->pin_errno = errno; + if (!pair_barrier(&g_warp.ready, &g_warp.abort, !pinned)) + return NULL; + end = rdtsc_lfence() + arg->budget_cycles; + // Guard against a TSC that never reaches the deadline. + deadline_ns = coarse_ns() + + (uint64_t)(cycles_to_ns((double)arg->budget_cycles) + STALL_SECONDS * 1e9); + for (i = 0;; i++) { + uint64_t prev, now; + + while (atomic_exchange_explicit(&g_warp.lock, 1, memory_order_acquire)) + while (atomic_load_explicit(&g_warp.lock, memory_order_relaxed)) + cpu_pause(); + prev = g_warp.last_tsc; + now = rdtsc_lfence(); + g_warp.last_tsc = now; + if (prev > now) { + if (prev - now > g_warp.max_warp) + g_warp.max_warp = prev - now; + g_warp.warps++; + } + atomic_store_explicit(&g_warp.lock, 0, memory_order_release); + if (!(i & 7)) { + if (arg->leader && + (rdtsc_plain() > end || (!(i & 4095) && coarse_ns() > deadline_ns))) + atomic_store_explicit(&g_warp.stop, 1, memory_order_relaxed); + if (atomic_load_explicit(&g_warp.stop, memory_order_relaxed)) + break; + } + } + arg->iterations = i + 1; + return NULL; +} + +static void *pp_leader(void *p) +{ + struct thread_arg *arg = p; + struct pair_result *r = arg->result; + bool pinned = pin_self(arg->cpu) == 0; + uint64_t stored = 0; + uint64_t end; + uint64_t k; + + if (!pinned) + arg->pin_errno = errno; + if (!pair_barrier(&g_pp.ready, &g_pp.abort, !pinned)) + return NULL; + end = rdtsc_lfence() + arg->budget_cycles; + for (k = 1;; k++) { + uint64_t spins = 0; + uint64_t t1, t2, t3, rtt; + int64_t lo, hi; + + t1 = rdtsc_fenced(); + atomic_store_explicit(&g_pp.seq, 2 * k - 1, memory_order_release); + while (atomic_load_explicit(&g_pp.seq, memory_order_acquire) != 2 * k) { + if ((++spins & SPIN_CHECK_MASK) == 0 && + rdtsc_plain() - t1 > arg->stall_cycles) { + r->stalled = true; + goto done; + } + } + t3 = rdtsc_lfence(); + t2 = atomic_load_explicit(&g_pp.t2, memory_order_relaxed); + rtt = t3 - t1; + lo = (int64_t)(t2 - t3); + hi = (int64_t)(t2 - t1); + if (rtt < r->min_rtt_cycles) { + r->min_rtt_cycles = rtt; + r->min_rtt_offset_x2 = lo + hi; + } + if (lo > r->lo_cycles) + r->lo_cycles = lo; + if (hi < r->hi_cycles) + r->hi_cycles = hi; + if (stored < PP_MAX_SAMPLES) + arg->offsets_x2[stored++] = lo + hi; + r->rounds = k; + if ((k & 63) == 0 && rdtsc_plain() > end) + break; + } +done: + atomic_store_explicit(&g_pp.seq, PP_STOP, memory_order_release); + arg->iterations = stored; + return NULL; +} + +static void *pp_follower(void *p) +{ + struct thread_arg *arg = p; + bool pinned = pin_self(arg->cpu) == 0; + + if (!pinned) + arg->pin_errno = errno; + if (!pair_barrier(&g_pp.ready, &g_pp.abort, !pinned)) + return NULL; + for (uint64_t k = 1;; k++) { + uint64_t spins = 0; + uint64_t wait_start = 0; + uint64_t value; + + while ((value = atomic_load_explicit(&g_pp.seq, memory_order_acquire)) != + 2 * k - 1) { + if (value == PP_STOP) + return NULL; + if ((++spins & SPIN_CHECK_MASK) == 0) { + uint64_t now = rdtsc_plain(); + + if (!wait_start) + wait_start = now; + else if (now - wait_start > arg->stall_cycles) + return NULL; + } + } + atomic_store_explicit(&g_pp.t2, rdtsc_fenced(), memory_order_relaxed); + atomic_store_explicit(&g_pp.seq, 2 * k, memory_order_release); + } +} + +static int run_threads(void *(*fn_a)(void *), void *(*fn_b)(void *), + struct thread_arg *a, struct thread_arg *b) +{ + pthread_t ta, tb; + int err; + + if ((err = pthread_create(&ta, NULL, fn_a, a)) != 0) { + errno = err; + return -1; + } + if ((err = pthread_create(&tb, NULL, fn_b, b)) != 0) { + // Release the first thread from its barrier before joining it. + atomic_store(&g_warp.abort, 1); + atomic_store(&g_pp.abort, 1); + atomic_fetch_add(&g_warp.ready, 1); + atomic_fetch_add(&g_pp.ready, 1); + pthread_join(ta, NULL); + errno = err; + return -1; + } + pthread_join(ta, NULL); + pthread_join(tb, NULL); + if (a->pin_errno || b->pin_errno) { + errno = a->pin_errno ? a->pin_errno : b->pin_errno; + return -1; + } + return 0; +} + +static int run_pair(int cpu_a, int cpu_b, uint64_t budget_cycles, + uint64_t stall_cycles, int64_t *offsets_x2, + struct pair_result *r) +{ + struct thread_arg a = {cpu_a, true, budget_cycles, stall_cycles, 0, 0, r, offsets_x2}; + struct thread_arg b = {cpu_b, false, budget_cycles, stall_cycles, 0, 0, r, NULL}; + + memset(r, 0, sizeof(*r)); + r->cpu_a = cpu_a; + r->cpu_b = cpu_b; + + atomic_store(&g_warp.lock, 0); + atomic_store(&g_warp.ready, 0); + atomic_store(&g_warp.abort, 0); + atomic_store(&g_warp.stop, 0); + g_warp.last_tsc = 0; + g_warp.max_warp = 0; + g_warp.warps = 0; + if (run_threads(warp_thread, warp_thread, &a, &b) != 0) + return -1; + r->warp_iterations = a.iterations + b.iterations; + r->warps = g_warp.warps; + r->max_warp_cycles = g_warp.max_warp; + + atomic_store(&g_pp.seq, 0); + atomic_store(&g_pp.t2, 0); + atomic_store(&g_pp.ready, 0); + atomic_store(&g_pp.abort, 0); + r->min_rtt_cycles = UINT64_MAX; + r->lo_cycles = INT64_MIN; + r->hi_cycles = INT64_MAX; + a.iterations = 0; + if (run_threads(pp_leader, pp_follower, &a, &b) != 0) + return -1; + if (a.iterations == 0) { + r->stalled = true; + return 0; + } + qsort(offsets_x2, (size_t)a.iterations, sizeof(*offsets_x2), cmp_i64); + r->median_offset_cycles = (double)offsets_x2[(a.iterations - 1) / 2] / 2.0; + r->consistent = r->lo_cycles <= r->hi_cycles; + if (r->consistent) { + r->offset_cycles = ((double)r->lo_cycles + (double)r->hi_cycles) / 2.0; + r->uncertainty_cycles = ((double)r->hi_cycles - (double)r->lo_cycles) / 2.0; + } else { + r->offset_cycles = (double)r->min_rtt_offset_x2 / 2.0; + r->uncertainty_cycles = (double)r->min_rtt_cycles / 2.0; + } + return 0; +} + +static double abs_double(double value) +{ + return value < 0 ? -value : value; +} + +static int cmd_warp(int argc, char **argv) +{ + static int cpus[MAX_CPUS]; + uint64_t duration_ms = 100; + uint64_t bound_ns = 1000; + double tsc_hz = 0; + int ncpus = -1; + int pairs = 0; + int inconsistent = 0; + int stalled = 0; + uint64_t total_warps = 0; + uint64_t max_backward = 0; + double max_abs_offset = 0; + double max_uncertainty = 0; + double max_skew_bound = 0; + uint64_t budget_cycles; + uint64_t stall_cycles; + int64_t *offsets_x2; + char cpu_text[256]; + bool pass; + bool conclusive; + + for (int i = 2; i < argc; i++) { + const char *value; + + if ((value = option_value(argc, argv, &i, "--duration-ms"))) { + if (parse_u64_arg(value, &duration_ms) || duration_ms == 0 || + duration_ms > 600000) + return bad_option("warp", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--cpus"))) { + if ((ncpus = parse_cpu_list(value, cpus, MAX_CPUS)) <= 0) + return bad_option("warp", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--bound-ns"))) { + if (parse_u64_arg(value, &bound_ns)) + return bad_option("warp", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--tsc-hz"))) { + if (parse_hz_arg(value, &tsc_hz)) + return bad_option("warp", argv[i]); + } else { + return bad_option("warp", argv[i]); + } + } + if (ncpus < 0 && (ncpus = affinity_cpus(cpus, MAX_CPUS)) <= 0) { + fprintf(stderr, "nvx-time-probe warp: sched_getaffinity: %s\n", strerror(errno)); + return EXIT_ERROR; + } + if (check_cpus_allowed("warp", cpus, ncpus) != 0) + return EXIT_USAGE; + detect_tsc_rate(tsc_hz); + if (g_tsc.hz <= 0) { + fprintf(stderr, "nvx-time-probe warp: unknown TSC rate; pass --tsc-hz\n"); + return EXIT_ERROR; + } + budget_cycles = (uint64_t)((double)duration_ms * g_tsc.hz / 1000.0); + stall_cycles = (uint64_t)(STALL_SECONDS * g_tsc.hz) + budget_cycles; + offsets_x2 = malloc(PP_MAX_SAMPLES * sizeof(*offsets_x2)); + if (!offsets_x2) { + fprintf(stderr, "nvx-time-probe warp: out of memory\n"); + return EXIT_ERROR; + } + format_cpu_list(cpus, ncpus, cpu_text, sizeof(cpu_text)); + printf("nvx-time-probe %s warp: cpus=%s pairs=%d duration_ms=%" PRIu64 + " (per test per pair) tsc_hz=%.0f tsc_hz_source=%s bound_ns=%" PRIu64 "\n", + PROBE_VERSION, cpu_text, ncpus * (ncpus - 1) / 2, duration_ms, g_tsc.hz, + g_tsc.source, bound_ns); + fflush(stdout); + + for (int i = 0; i < ncpus; i++) { + for (int j = i + 1; j < ncpus; j++) { + struct pair_result r; + double skew_bound; + + if (run_pair(cpus[i], cpus[j], budget_cycles, stall_cycles, offsets_x2, + &r) != 0) { + fprintf(stderr, "nvx-time-probe warp: pair %d-%d: %s\n", cpus[i], + cpus[j], strerror(errno)); + free(offsets_x2); + return EXIT_ERROR; + } + pairs++; + total_warps += r.warps; + if (r.max_warp_cycles > max_backward) + max_backward = r.max_warp_cycles; + if (r.stalled) { + stalled++; + printf("pair %d-%d: warp_iterations=%" PRIu64 " warps=%" PRIu64 + " max_backward_cycles=%" PRIu64 " ping_pong=stalled\n", + r.cpu_a, r.cpu_b, r.warp_iterations, r.warps, r.max_warp_cycles); + continue; + } + if (!r.consistent) + inconsistent++; + skew_bound = r.consistent + ? (abs_double((double)r.lo_cycles) > abs_double((double)r.hi_cycles) + ? abs_double((double)r.lo_cycles) + : abs_double((double)r.hi_cycles)) + : abs_double(r.offset_cycles) + r.uncertainty_cycles; + if (abs_double(r.offset_cycles) > max_abs_offset) + max_abs_offset = abs_double(r.offset_cycles); + if (r.uncertainty_cycles > max_uncertainty) + max_uncertainty = r.uncertainty_cycles; + if (skew_bound > max_skew_bound) + max_skew_bound = skew_bound; + printf("pair %d-%d: warp_iterations=%" PRIu64 " warps=%" PRIu64 + " max_backward_cycles=%" PRIu64 " max_backward_ns=%lld rounds=%" PRIu64 + " min_rtt_ns=%lld offset_cycles=%.1f offset_ns=%lld uncertainty_ns=%lld" + " bounds_ns=[%lld,%lld] median_offset_ns=%lld consistent=%d\n", + r.cpu_a, r.cpu_b, r.warp_iterations, r.warps, r.max_warp_cycles, + round_ll(cycles_to_ns((double)r.max_warp_cycles)), r.rounds, + round_ll(cycles_to_ns((double)r.min_rtt_cycles)), r.offset_cycles, + round_ll(cycles_to_ns(r.offset_cycles)), + round_ll(cycles_to_ns(r.uncertainty_cycles)), + round_ll(cycles_to_ns((double)r.lo_cycles)), + round_ll(cycles_to_ns((double)r.hi_cycles)), + round_ll(cycles_to_ns(r.median_offset_cycles)), r.consistent); + fflush(stdout); + } + } + free(offsets_x2); + if (pairs == 0) + printf("single CPU: no pairs to compare\n"); + + pass = stalled == 0 && cycles_to_ns((double)max_backward) <= (double)bound_ns && + cycles_to_ns(max_abs_offset) <= (double)bound_ns; + // The offset estimate decides nothing when its uncertainty reaches the bound + // (for example, under trapped RDTSC, where every read exits to the VMM). + conclusive = pairs == 0 || (stalled == 0 && cycles_to_ns(max_uncertainty) < (double)bound_ns); + if (!conclusive) + printf("warning: offset uncertainty (+/-%lld ns) is not below the bound (%" PRIu64 + " ns); the verdict is not conclusive (trapped RDTSC or a descheduled vCPU?)\n", + round_ll(cycles_to_ns(max_uncertainty)), bound_ns); + printf("NVX-TIME-PROBE warp max_backward_cycles=%" PRIu64 " max_backward_ns=%lld" + " max_abs_offset_ns=%lld pairs=%d verdict=%s bound_ns=%" PRIu64 "\n", + max_backward, round_ll(cycles_to_ns((double)max_backward)), + round_ll(cycles_to_ns(max_abs_offset)), pairs, pass ? "PASS" : "FAIL", + bound_ns); + printf("NVX-TIME-PROBE warp-detail max_abs_offset_cycles=%.1f max_uncertainty_ns=%lld" + " max_skew_bound_ns=%lld total_warps=%" PRIu64 " inconsistent_pairs=%d" + " stalled_pairs=%d cpus=%s duration_ms=%" PRIu64 " tsc_hz=%.0f" + " tsc_hz_source=%s conclusive=%d\n", + max_abs_offset, round_ll(cycles_to_ns(max_uncertainty)), + round_ll(cycles_to_ns(max_skew_bound)), total_warps, inconsistent, stalled, + cpu_text, duration_ms, g_tsc.hz, g_tsc.source, conclusive); + return pass ? 0 : EXIT_FAIL; +} + +// --------------------------------------------------------------------------- +// clock +// --------------------------------------------------------------------------- + +static const char *const tsc_flag_names[] = { + "tsc", "rdtscp", "constant_tsc", "nonstop_tsc", "nonstop_tsc_s3", + "tsc_known_freq", "tsc_reliable", "tsc_deadline_timer", "tsc_adjust", "art", + "hypervisor", "arat", "aperfmperf", "hwp", "x2apic", + NULL, +}; + +static const char *const hypervisor_patterns[] = { + "Hypervisor detected", "Hyper-V", "Booting paravirtualized kernel", "kvm-guest", + NULL, +}; + +static const char *const time_patterns[] = { + "tsc", "TSC", "clocksource", "clockevent", "kvm-clock", + "APIC timer", "LAPIC", "lapic", "calibrat", "Calibrat", + "unchecked MSR", "hpet", "HPET", "timekeeping", "sched_clock", + "detected stall", "watchdog", "lockup", "hung_task", "jiffies", + "rtc", "RTC", NULL, +}; + +static const char *const param_files[] = { + "/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress", + "/sys/module/rcupdate/parameters/rcu_cpu_stall_timeout", + "/proc/sys/kernel/watchdog", + "/proc/sys/kernel/soft_watchdog", + "/proc/sys/kernel/nmi_watchdog", + "/proc/sys/kernel/watchdog_thresh", + "/proc/sys/kernel/hung_task_timeout_secs", + NULL, +}; + +struct kmsg_filter { + const char *const *patterns; + int matches; + int tsc_unstable; + int unchecked_msr; +}; + +struct tick_info { + bool seen; + int tick_mode; + int state; + int hres_active; + int nohz; + char device[64]; + char handler[64]; +}; + +static void add_unique(char *list, size_t size, const char *item) +{ + size_t len = strlen(item); + + for (const char *p = list; (p = strstr(p, item)) != NULL; p += len) + if ((p == list || p[-1] == ',') && (p[len] == ',' || p[len] == '\0')) + return; + if (list[0]) + strncat(list, ",", size - strlen(list) - 1); + strncat(list, item, size - strlen(list) - 1); +} + +static void sanitize_token(char *text) +{ + for (char *p = text; *p; p++) + if (!((*p >= 'a' && *p <= 'z') || (*p >= 'A' && *p <= 'Z') || + (*p >= '0' && *p <= '9') || *p == '#' || *p == '-' || *p == '.')) + *p = '_'; +} + +static void print_clock(const char *name, clockid_t id, struct timespec *out) +{ + struct timespec ts; + + if (clock_gettime(id, &ts) != 0) { + printf("%-20s unavailable (%s)\n", name, strerror(errno)); + return; + } + if (out) + *out = ts; + printf("%-20s %lld.%09ld", name, (long long)ts.tv_sec, ts.tv_nsec); + if (id == CLOCK_REALTIME) { + struct tm tm; + char iso[32]; + time_t seconds = ts.tv_sec; + + gmtime_r(&seconds, &tm); + strftime(iso, sizeof(iso), "%Y-%m-%dT%H:%M:%S", &tm); + printf(" (%s.%09ldZ)", iso, ts.tv_nsec); + } + printf("\n"); +} + +static void kmsg_filter_cb(uint64_t usec, const char *text, void *ctx) +{ + struct kmsg_filter *filter = ctx; + + if (strstr(text, "Marking TSC unstable") || strstr(text, "TSC unstable")) + filter->tsc_unstable++; + if (strstr(text, "unchecked MSR access")) + filter->unchecked_msr++; + for (const char *const *pattern = filter->patterns; *pattern; pattern++) { + if (strstr(text, *pattern)) { + printf("[%5llu.%06llu] %s\n", (unsigned long long)(usec / 1000000), + (unsigned long long)(usec % 1000000), text); + filter->matches++; + return; + } + } +} + +static const char *tick_mode_name(int mode) +{ + return mode == 0 ? "periodic" : mode == 1 ? "oneshot" : "unknown"; +} + +static const char *clockevent_state_name(int state) +{ + static const char *const names[] = {"detached", "shutdown", "periodic", "oneshot", + "oneshot_stopped"}; + + return state >= 0 && state <= 4 ? names[state] : "unknown"; +} + +static void print_timer_list(char *summary, size_t size) +{ + static struct tick_info cpus[MAX_CPUS]; + struct tick_info broadcast = {false, -1, -1, -1, -1, "", ""}; + struct tick_info *cur = NULL; + char devices[256] = ""; + char modes[64] = ""; + char states[128] = ""; + char line[512]; + int hr_cpu = -1; + int tick_mode = -1; + int count = 0; + FILE *file = fopen("/proc/timer_list", "r"); + + if (!file) { + printf("timer_list: unavailable (%s)\n", strerror(errno)); + snprintf(summary, size, "clockevents=unavailable"); + return; + } + for (int i = 0; i < MAX_CPUS; i++) + cpus[i] = (struct tick_info){false, -1, -1, -1, -1, "", ""}; + while (fgets(line, sizeof(line), file)) { + char name[64]; + int value; + + line[strcspn(line, "\n")] = '\0'; + if (strncmp(line, "cpu: ", 5) == 0) { + hr_cpu = atoi(line + 5); + cur = NULL; + } else if (strncmp(line, "Tick Device: mode:", 18) == 0) { + tick_mode = atoi(line + 18); + hr_cpu = -1; + cur = NULL; + } else if (strncmp(line, "Broadcast device", 16) == 0) { + cur = &broadcast; + cur->seen = true; + cur->tick_mode = tick_mode; + } else if (sscanf(line, "Per CPU device: %d", &value) == 1) { + cur = value >= 0 && value < MAX_CPUS ? &cpus[value] : NULL; + if (cur) { + cur->seen = true; + cur->tick_mode = tick_mode; + } + } else if (cur && sscanf(line, "Clock Event Device: %63s", name) == 1) { + snprintf(cur->device, sizeof(cur->device), "%s", name); + } else if (cur && sscanf(line, " mode: %d", &value) == 1) { + cur->state = value; + } else if (cur && sscanf(line, " event_handler: %63s", name) == 1) { + snprintf(cur->handler, sizeof(cur->handler), "%s", name); + } else if (hr_cpu >= 0 && hr_cpu < MAX_CPUS) { + if (sscanf(line, " .hres_active : %d", &value) == 1) + cpus[hr_cpu].hres_active = value; + else if (sscanf(line, " .nohz_mode : %d", &value) == 1 || + sscanf(line, " .nohz : %d", &value) == 1) + cpus[hr_cpu].nohz = value; + } + } + fclose(file); + + if (broadcast.seen) + printf("broadcast: device=%s state=%s tick_mode=%s handler=%s\n", + broadcast.device[0] ? broadcast.device : "none", + clockevent_state_name(broadcast.state), tick_mode_name(broadcast.tick_mode), + broadcast.handler[0] ? broadcast.handler : "none"); + for (int i = 0; i < MAX_CPUS; i++) { + if (!cpus[i].seen) + continue; + count++; + printf("cpu%d: device=%s state=%s tick_mode=%s handler=%s hres_active=%d nohz=%d\n", + i, cpus[i].device[0] ? cpus[i].device : "none", + clockevent_state_name(cpus[i].state), tick_mode_name(cpus[i].tick_mode), + cpus[i].handler[0] ? cpus[i].handler : "none", cpus[i].hres_active, + cpus[i].nohz); + add_unique(devices, sizeof(devices), cpus[i].device[0] ? cpus[i].device : "none"); + add_unique(modes, sizeof(modes), tick_mode_name(cpus[i].tick_mode)); + add_unique(states, sizeof(states), clockevent_state_name(cpus[i].state)); + } + if (!broadcast.seen && count == 0) + printf("timer_list: no tick devices found (file empty or masked)\n"); + snprintf(summary, size, + "clockevent_cpus=%d clockevent_devices=%s tick_modes=%s clockevent_states=%s" + " broadcast=%s:%s", + count, devices[0] ? devices : "none", modes[0] ? modes : "none", + states[0] ? states : "none", + broadcast.seen && broadcast.device[0] ? broadcast.device : "none", + broadcast.seen ? clockevent_state_name(broadcast.state) : "none"); +} + +struct cpuid_summary { + char vendor[16]; + uint32_t max_leaf; + uint32_t features_eax; + uint32_t features_edx; + bool hypervisor; +}; + +static void print_cpuid(struct cpuid_summary *s) +{ + uint32_t r[4]; + uint32_t max_basic, max_ext, family, model; + + memset(s, 0, sizeof(*s)); + cpuid(0, 0, r); + max_basic = r[0]; + cpuid(0x80000000u, 0, r); + max_ext = r[0]; + cpuid(1, 0, r); + family = (r[0] >> 8) & 0xf; + model = (r[0] >> 4) & 0xf; + if (family == 0x6 || family == 0xf) + model |= (r[0] >> 12) & 0xf0; + if (family == 0xf) + family += (r[0] >> 20) & 0xff; + s->hypervisor = (r[2] >> 31) & 1; + printf("cpuid.1: family=%u model=%u stepping=%u edx.tsc=%u ecx.tsc_deadline=%u" + " ecx.hypervisor=%u\n", + family, model, r[0] & 0xf, (r[3] >> 4) & 1, (r[2] >> 24) & 1, s->hypervisor); + if (max_basic >= 6) { + cpuid(6, 0, r); + printf("cpuid.6: eax.arat=%u eax.hwp=%u ecx.aperfmperf=%u\n", (r[0] >> 2) & 1, + (r[0] >> 7) & 1, r[2] & 1); + } + if (max_basic >= 7) { + cpuid(7, 0, r); + printf("cpuid.7.0: ebx.tsc_adjust=%u\n", (r[1] >> 1) & 1); + } + if (max_basic >= 0xa) { + cpuid(0xa, 0, r); + printf("cpuid.0xa: eax=%#010x (pmu_version=%u)\n", r[0], r[0] & 0xff); + } + if (max_basic >= 0x15) { + cpuid(0x15, 0, r); + printf("cpuid.0x15: eax.denominator=%u ebx.numerator=%u ecx.crystal_hz=%u", r[0], + r[1], r[2]); + if (r[0] && r[1] && r[2]) + printf(" (tsc_hz=%.0f)", (double)r[2] * r[1] / r[0]); + printf("\n"); + } + if (max_basic >= 0x16) { + cpuid(0x16, 0, r); + printf("cpuid.0x16: eax.base_mhz=%u ebx.max_mhz=%u ecx.bus_mhz=%u\n", + r[0] & 0xffff, r[1] & 0xffff, r[2] & 0xffff); + } + if (max_ext >= 0x80000001u) { + cpuid(0x80000001u, 0, r); + printf("cpuid.0x80000001: edx.rdtscp=%u\n", (r[3] >> 27) & 1); + } + if (max_ext >= 0x80000007u) { + cpuid(0x80000007u, 0, r); + printf("cpuid.0x80000007: edx.invariant_tsc=%u\n", (r[3] >> 8) & 1); + } + if (!s->hypervisor) { + printf("cpuid.0x40000000: not queried (hypervisor bit clear)\n"); + snprintf(s->vendor, sizeof(s->vendor), "none"); + return; + } + cpuid(0x40000000u, 0, r); + s->max_leaf = r[0]; + memcpy(s->vendor, &r[1], 4); + memcpy(s->vendor + 4, &r[2], 4); + memcpy(s->vendor + 8, &r[3], 4); + s->vendor[12] = '\0'; + for (int i = 0; i < 12 && s->vendor[i]; i++) + if (s->vendor[i] < 0x20 || s->vendor[i] > 0x7e) + s->vendor[i] = '.'; + printf("cpuid.0x40000000: eax.max_leaf=%#x vendor=\"%s\"\n", s->max_leaf, s->vendor); + for (uint32_t leaf = 0x40000001u; leaf <= s->max_leaf && leaf <= 0x40000010u; leaf++) { + cpuid(leaf, 0, r); + printf("cpuid.%#x: eax=%#010x ebx=%#010x ecx=%#010x edx=%#010x\n", leaf, r[0], + r[1], r[2], r[3]); + if (leaf == 0x40000003u) { + s->features_eax = r[0]; + s->features_edx = r[3]; + } + } + if (strcmp(s->vendor, "Microsoft Hv") == 0 && s->max_leaf >= 0x40000003u) + printf("hyperv.features: hypercall=%u vp_index=%u access_frequency_msrs=%u" + " access_tsc_invariant=%u frequency_msrs_available=%u\n", + (s->features_eax >> 5) & 1, (s->features_eax >> 6) & 1, + (s->features_eax >> 11) & 1, (s->features_eax >> 15) & 1, + (s->features_edx >> 8) & 1); +} + +// Opt-in: reads go through the msr driver's rdmsr_safe(), so a #GP returns +// EIO instead of faulting in the guest. +static void print_msrs(void) +{ + static const struct { + uint32_t index; + const char *name; + } msrs[] = { + {0x10, "IA32_TSC"}, + {0x3b, "IA32_TSC_ADJUST"}, + {0x40000002u, "HV_X64_MSR_VP_INDEX"}, + {0x40000022u, "HV_X64_MSR_TSC_FREQUENCY"}, + {0x40000023u, "HV_X64_MSR_APIC_FREQUENCY"}, + {0x40000118u, "HV_X64_MSR_TSC_INVARIANT_CONTROL"}, + {0x4b564d01u, "MSR_KVM_SYSTEM_TIME_NEW"}, + }; + int fd = open("/dev/cpu/0/msr", O_RDONLY | O_CLOEXEC); + + if (fd < 0) { + printf("msr: /dev/cpu/0/msr unavailable (%s)\n", strerror(errno)); + return; + } + for (size_t i = 0; i < sizeof(msrs) / sizeof(msrs[0]); i++) { + uint64_t value; + + if (pread(fd, &value, sizeof(value), (off_t)msrs[i].index) == sizeof(value)) + printf("msr %#010x %-34s = %#" PRIx64 " (%" PRIu64 ")\n", msrs[i].index, + msrs[i].name, value, value); + else + printf("msr %#010x %-34s : %s\n", msrs[i].index, msrs[i].name, + errno == EIO ? "read faulted (#GP)" : strerror(errno)); + } + close(fd); +} + +static int cmd_clock(int argc, char **argv) +{ + uint64_t measure_ms = 200; + double tsc_hz = 0; + bool read_msrs = false; + struct timespec mono = {0}; + struct kmsg_filter hv_filter = {hypervisor_patterns, 0, 0, 0}; + struct kmsg_filter time_filter = {time_patterns, 0, 0, 0}; + struct cpuid_summary ids; + char clocksource[64]; + char available[256] = ""; + char present[256] = ""; + char absent[256] = ""; + char timer_summary[512]; + char line[4096]; + uint64_t tsc; + int cpu; + + for (int i = 2; i < argc; i++) { + const char *value; + + if ((value = option_value(argc, argv, &i, "--measure-ms"))) { + if (parse_u64_arg(value, &measure_ms) || measure_ms > 60000) + return bad_option("clock", argv[i]); + } else if ((value = option_value(argc, argv, &i, "--tsc-hz"))) { + if (parse_hz_arg(value, &tsc_hz)) + return bad_option("clock", argv[i]); + } else if (strcmp(argv[i], "--msr") == 0) { + read_msrs = true; + } else { + return bad_option("clock", argv[i]); + } + } + detect_tsc_rate(tsc_hz); + + printf("nvx-time-probe %s clock\n== clocks\n", PROBE_VERSION); + cpu = sched_getcpu(); + tsc = rdtsc_fenced(); + print_clock("CLOCK_REALTIME", CLOCK_REALTIME, NULL); + print_clock("CLOCK_MONOTONIC", CLOCK_MONOTONIC, &mono); + print_clock("CLOCK_MONOTONIC_RAW", CLOCK_MONOTONIC_RAW, NULL); + print_clock("CLOCK_BOOTTIME", CLOCK_BOOTTIME, NULL); + printf("%-20s %" PRIu64 " on cpu %d", "TSC", tsc, cpu); + if (g_tsc.hz > 0) + printf(" (%.9f s; tsc_minus_monotonic_ns=%lld)", (double)tsc / g_tsc.hz, + round_ll((double)tsc * 1e9 / g_tsc.hz - + ((double)mono.tv_sec * 1e9 + (double)mono.tv_nsec))); + printf("\ntsc_hz=%.0f tsc_hz_source=%s\n", g_tsc.hz, g_tsc.source); + if (measure_ms > 0) + printf("tsc_hz_measured=%.0f (over %" PRIu64 " ms against CLOCK_MONOTONIC_RAW)\n", + measure_tsc_hz((unsigned)measure_ms), measure_ms); + + printf("== clocksource\n"); + current_clocksource(clocksource, sizeof(clocksource)); + if (read_first_line(CLOCKSOURCE_DIR "/available_clocksource", line, sizeof(line)) == 0) + for (char *token = strtok(line, " "); token; token = strtok(NULL, " ")) + add_unique(available, sizeof(available), token); + printf("current: %s\navailable: %s\n", clocksource, available[0] ? available : "unknown"); + + printf("== cpuinfo\n"); + { + FILE *file = fopen("/proc/cpuinfo", "r"); + int processors = 0; + bool model_printed = false; + char mhz[512] = ""; + + while (file && fgets(line, sizeof(line), file)) { + char *colon = strchr(line, ':'); + + line[strcspn(line, "\n")] = '\0'; + if (strncmp(line, "processor", 9) == 0) + processors++; + else if (!model_printed && strncmp(line, "model name", 10) == 0 && colon) { + printf("model name:%s\n", colon + 1); + model_printed = true; + } else if (strncmp(line, "cpu MHz", 7) == 0 && colon) + add_unique(mhz, sizeof(mhz), colon + 2); + } + if (file) + fclose(file); + printf("processors: %d\ncpu MHz: %s\n", processors, mhz[0] ? mhz : "unknown"); + } + for (const char *const *flag = tsc_flag_names; *flag; flag++) + add_unique(cpuinfo_has_flag(*flag) ? present : absent, + cpuinfo_has_flag(*flag) ? sizeof(present) : sizeof(absent), *flag); + printf("time flags present: %s\ntime flags absent: %s\n", present[0] ? present : "none", + absent[0] ? absent : "none"); + + printf("== cpuid\n"); + print_cpuid(&ids); + if (read_msrs) { + printf("== msr (cpu 0)\n"); + print_msrs(); + } + + printf("== kernel\n"); + if (read_first_line("/proc/cmdline", line, sizeof(line)) == 0) + printf("cmdline: %s\n", line); + if (read_first_line("/sys/devices/system/cpu/online", line, sizeof(line)) == 0) + printf("online cpus: %s\n", line); + for (const char *const *path = param_files; *path; path++) { + if (read_first_line(*path, line, sizeof(line)) == 0) + printf("%s = %s\n", *path, line); + else + printf("%s: absent\n", *path); + } + + printf("== kmsg: hypervisor\n"); + if (kmsg_foreach(kmsg_filter_cb, &hv_filter) != 0) + printf("(kernel log unavailable: %s)\n", strerror(errno)); + else if (!hv_filter.matches) + printf("(none)\n"); + printf("== kmsg: time\n"); + if (kmsg_foreach(kmsg_filter_cb, &time_filter) != 0) + printf("(kernel log unavailable: %s)\n", strerror(errno)); + else if (!time_filter.matches) + printf("(none)\n"); + + printf("== timer_list\n"); + print_timer_list(timer_summary, sizeof(timer_summary)); + + sanitize_token(ids.vendor); + printf("NVX-TIME-PROBE clock clocksource=%s available=%s tsc_hz=%.0f tsc_hz_source=%s" + " hv_vendor=%s hv_max_leaf=%#x hv_features_eax=%#x hv_features_edx=%#x" + " flags=%s missing_flags=%s tsc_unstable_msgs=%d unchecked_msr_msgs=%d %s\n", + clocksource, available[0] ? available : "unknown", g_tsc.hz, g_tsc.source, + ids.vendor, ids.max_leaf, ids.features_eax, ids.features_edx, + present[0] ? present : "none", absent[0] ? absent : "none", + time_filter.tsc_unstable, time_filter.unchecked_msr, timer_summary); + return 0; +} + +// --------------------------------------------------------------------------- +// main +// --------------------------------------------------------------------------- + +static void usage(FILE *out) +{ + fprintf(out, + "usage: nvx-time-probe COMMAND [OPTIONS]\n" + "\n" + "Commands:\n" + " latency [--iterations N] [--batch B] [--cpu C] [--tsc-hz HZ]\n" + " Per-operation cost (p50/p99/min/mean ns) of rdtsc, rdtscp,\n" + " lfence;rdtsc, clock_gettime(MONOTONIC/REALTIME) through the vDSO,\n" + " and syscall controls. Default: 2000 timed batches of 50 operations\n" + " on the current CPU.\n" + " warp [--duration-ms N] [--cpus LIST] [--bound-ns N] [--tsc-hz HZ]\n" + " Pairwise cross-CPU TSC checks: a check_tsc_warp-style spinlock test\n" + " and a ping-pong offset estimate, each running N ms per pair\n" + " (default 100). LIST looks like 0-3,6 (default: every CPU in the\n" + " affinity mask). The bound defaults to 1000 ns.\n" + " clock [--measure-ms N] [--tsc-hz HZ] [--msr]\n" + " Clocks, clocksources, TSC flags, CPUID time leaves, hypervisor and\n" + " time kernel messages, and the /proc/timer_list clock event summary.\n" + " --msr also reads the TSC, Hyper-V time, and kvmclock MSRs on CPU 0\n" + " through /dev/cpu/0/msr (a #GP is reported, not raised).\n" + " version\n" + "\n" + "The TSC rate comes from --tsc-hz, sysfs tsc_freq_khz, the kernel log, or\n" + "/proc/cpuinfo, in that order, and is measured as a last resort.\n" + "Exit status: 0 success, 1 warp FAIL, 2 usage error, 3 runtime error.\n"); +} + +int main(int argc, char **argv) +{ + setvbuf(stdout, NULL, _IOLBF, 0); + if (argc < 2) { + usage(stderr); + return EXIT_USAGE; + } + if (strcmp(argv[1], "latency") == 0) + return cmd_latency(argc, argv); + if (strcmp(argv[1], "warp") == 0) + return cmd_warp(argc, argv); + if (strcmp(argv[1], "clock") == 0) + return cmd_clock(argc, argv); + if (strcmp(argv[1], "version") == 0 || strcmp(argv[1], "--version") == 0) { + printf("nvx-time-probe %s\n", PROBE_VERSION); + return 0; + } + if (strcmp(argv[1], "help") == 0 || strcmp(argv[1], "--help") == 0 || + strcmp(argv[1], "-h") == 0) { + usage(stdout); + return 0; + } + usage(stderr); + return EXIT_USAGE; +} diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 062d51148..772e47277 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -576,13 +576,14 @@ def _build_static_helper( work: Path, source: Path, destination: Path, + cflags: tuple[str, ...] = InitramfsBuildConstants.STATIC_HELPER_CFLAGS, ) -> dict[str, str]: compiler = require_tool("cc") output = work / source.stem run_checked( [ compiler, - *InitramfsBuildConstants.STATIC_HELPER_CFLAGS, + *cflags, "-o", output, source, @@ -840,6 +841,13 @@ def _install_guest_files( common / f"{name}.c", root / "sbin" / name, ) + probe = InitramfsBuildConstants.TIME_PROBE_NAME + helpers[probe] = _build_static_helper( + config.work, + common / f"{probe}.c", + root / "sbin" / probe, + InitramfsBuildConstants.TIME_PROBE_CFLAGS, + ) helpers["nvx-device-io"] = _build_device_io_helper( config.work, root / "sbin" / "nvx-device-io", diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 49920f46e..4fca1017e 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -311,6 +311,17 @@ class InitramfsBuildConstants: "nvx-console-pending", "nvx-managed-agent", ) + TIME_PROBE_NAME: Final = "nvx-time-probe" + TIME_PROBE_CFLAGS: Final = ( + "-std=gnu11", + "-static", + "-O2", + "-pthread", + "-s", + "-Wall", + "-Wextra", + "-Werror", + ) class DockerBuildConstants: diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 736872f82..0572afc44 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -6312,6 +6312,61 @@ def test_guest_build_adds_debug_kernel_only_when_requested(self): debug.work.name, KernelBuildConstants.DEBUG_WORK_DIRECTORY_NAME ) + def test_guest_files_build_time_probe_with_its_own_flags(self): + source = BuildConstants.REPO_ROOT / "guest" / "common" / "nvx-time-probe.c" + header = source.read_text(encoding="utf-8").splitlines()[:2] + self.assertEqual( + header, + [ + "// Copyright (c) Microsoft Corporation.", + "// Licensed under the MIT License.", + ], + ) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) / "root" + work = Path(temporary) / "work" + for directory in (root / "sbin", root / "etc", work): + directory.mkdir(parents=True) + commands: list[list[str]] = [] + + def compile_helper( + command: list[str | os.PathLike[str]], **_kwargs: object + ) -> None: + commands.append([os.fspath(part) for part in command]) + output = Path(command[command.index("-o") + 1]) + output.write_bytes(f"static-elf {output.name}".encode()) + + with ( + patch.object(build, "require_tool", return_value="cc"), + patch.object(build, "run_checked", side_effect=compile_helper), + ): + helpers = build._install_guest_files( + build_config.InitramfsBuildConfig(work=work), + root, + guests.guest_descriptor("alpine"), + ) + + probe = InitramfsBuildConstants.TIME_PROBE_NAME + flags_by_source = { + Path(command[-1]).stem: command[1 : command.index("-o")] + for command in commands + } + self.assertEqual( + flags_by_source[probe], list(InitramfsBuildConstants.TIME_PROBE_CFLAGS) + ) + self.assertIn("-pthread", flags_by_source[probe]) + for name in InitramfsBuildConstants.STATIC_HELPERS: + with self.subTest(helper=name): + self.assertEqual( + flags_by_source[name], + list(InitramfsBuildConstants.STATIC_HELPER_CFLAGS), + ) + self.assertEqual(helpers[probe]["source_sha256"], common.sha256_file(source)) + self.assertEqual( + helpers[probe]["binary_sha256"], + hashlib.sha256(f"static-elf {probe}".encode()).hexdigest(), + ) + def test_docker_debug_kernel_target_exports_debug_artifacts(self): dockerfile = (BuildConstants.REPO_ROOT / "docker" / "Dockerfile").read_text( encoding="utf-8" From 751a03cb5c53ac54cc36b37b580f4faeba01db3d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 02:38:30 -0700 Subject: [PATCH 008/188] doc: finalize the time ABI backend obligations from the spikes Fill in the KVM, MSHV, and WHP mechanisms that the Phase 1 spikes demonstrated: KVM's MSR filter and common TSC offset, MSHV's MSR-index intercepts and frozen TSC writes, and WHP's suspended TSC writes with an explicit resume. WHP met the 1 us skew gate natively, so the RDTSC emulation interim is removed. Also program explicit zero leaves that KVM cannot otherwise list, make IA32_TSC_ADJUST and IA32_TSC_DEADLINE fault on every backend, treat the invariant-TSC bit as profile policy, re-apply the LAPIC state after the synchronized TSC set, and require a tsc host clocksource on Linux. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 120 ++++++++++++++++++++++++----------------- 1 file changed, 70 insertions(+), 50 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index aa518d4a9..6cdc77393 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -47,8 +47,7 @@ Goals: Non-goals: - Restore across backends or across CPU generations. Both are rejected. -- TSC scaling, and RDTSC trapping outside the WHP interim described in - [Backend obligations](#backend-obligations). +- TSC scaling and RDTSC trapping. - Paravirtual clocks: kvmclock, the Hyper-V reference TSC page and reference counter, synthetic timers, VMGenID, and VMClock. - A virtual PMU. AMD profiles are deferred until an AMD host is registered. @@ -86,8 +85,11 @@ and sets the spinlock retry count to "never notify". Every other leaf in `0x40000006..=0x400000ff` returns either all zeros or the vendor's architectural out-of-range result (on Intel, the result of the -highest basic leaf for the same subleaf). It never returns Hyper-V feature -data, a `VS#1` interface signature at `0x40000081`, or any hypervisor +highest basic leaf for the same subleaf). OpenVMM programs explicit zero +results for `0x40000006..=0x4000000f` and `0x40000080..=0x40000082`, which +Linux and Hyper-V-aware software probe, so they read zero on every backend; +KVM cannot list the whole range in its CPUID table. No leaf returns Hyper-V +feature data, a `VS#1` interface signature at `0x40000081`, or any hypervisor signature. No base `0x40000100..=0x4000ff00` (step `0x100`) carries a KVM, Xen, VMware, or other hypervisor signature, so Linux selects only the Hyper-V platform. @@ -108,6 +110,11 @@ on every backend: `HV_X64_MSR_TSC_INVARIANT_CONTROL` is one partition-wide value. It is guest state: it is saved with the VM and restored with it. Linux writes 1 at boot with an unchecked `wrmsrq`, so this write must never fault on any backend. +Clearing it after setting it is accepted. + +The CPU time bits hide two architectural timer MSRs, and every backend makes +them raise #GP: `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` +(`0x6e0`). ### CPU time bits @@ -131,10 +138,13 @@ rates from the frequency MSRs, and zero leaves keep the profile's CPUID independent of any host's rate. Linux trusts the TSC because of `AccessTscInvariantControls`, not because of -CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag. Until +CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag and +spares each AP a delay-loop calibration (about 150 ms per AP on Azure MSHV). +The bit is policy, not a fingerprint feature: profiles set it even where a +nested hypervisor hides it from host fingerprints (Azure's MSHV and WHP L1 +partitions), and host qualification backs it with measured invariance. Until the WHP report settles whether WHP can and must expose it, a WHP profile may -clear it, and WHP hosts qualify on measured invariance instead (see -[Host qualification](#host-qualification)). +clear it (see [Host qualification](#host-qualification)). ### Rates @@ -189,6 +199,8 @@ Because the kernel no longer checks cross-vCPU TSC consistency, the outside the contract; the guest never executes it. - `HV_X64_MSR_TSC_FREQUENCY` returns the declared rate `F`, which differs from the physical rate `F_d` by up to the rate tolerance after a restore. +- `HV_X64_MSR_TSC_INVARIANT_CONTROL` can be cleared after it is set; Hyper-V + and KVM raise #GP instead. - CPUID `0x80000007` EDX[8] is set by the CPU profile from boot. It does not depend on writes to `HV_X64_MSR_TSC_INVARIANT_CONTROL`. - `0x40000002` carries an NVX signature and the time ABI version, not a @@ -205,33 +217,35 @@ the raw TSC. ## Backend obligations Each backend implements these mechanisms behind one OpenVMM interface. The -guest-visible result is identical; only the mechanism differs. +guest-visible result is identical; only the mechanism differs. The spikes +(`p1-spike-kvm`, `p1-spike-mshv`, `p1-spike-whp-tsc`) demonstrated every +settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity leaves; every KVM `0x4xxxxxxx` entry removed | CPUID intercept results (`always_override`) for `0x40000000..=0x400000ff` — TBD(mshv) | CPUID result list or CPUID exits for `0x40000000..=0x400000ff` — TBD(whp) | -| Profile CPUID and time bits | `KVM_SET_CPUID2` | Processor feature banks plus CPUID intercept results | Processor feature banks plus CPUID result list or exits — TBD(whp) | -| Identity MSRs routed to OpenVMM | `KVM_X86_SET_MSR_FILTER` denying `0x40000000..=0x400001ff`, with user-space MSR exits for filtered and unknown MSRs — TBD(kvm) confirm precedence over in-kernel Hyper-V MSRs | MSR intercept for the range, or native frequency MSRs plus an intercept for `0x40000118` only if native values match the table exactly — TBD(mshv) | Unhandled-MSR exits enabled for the offloaded APIC; no `hv1_emulator` — TBD(whp) | -| Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight | CPUID result list or CPUID exits for the identity and explicit zero leaves — TBD(whp) | +| Profile CPUID and time bits | `KVM_SET_CPUID2` | Processor feature banks plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks plus CPUID result list or exits — TBD(whp) | +| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | Unhandled-MSR exits enabled for the offloaded APIC; no `hv1_emulator` — TBD(whp) | +| Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | -| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | `ApicFrequency` property, 200 MHz — TBD(mshv) | Offloaded APIC only, 200 MHz; the emulated APIC is rejected — TBD(whp) | -| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; `IA32_TSC_ADJUST` raises #GP | Feature-bank bits cleared and CPUID bits cleared | Feature-bank bits cleared and CPUID bits cleared — TBD(whp) | -| Invariant TSC exposed | CPUID bit | `tsc_invariant_support` plus CPUID; hosts without it are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition; hosts qualify on the measured warp and rate stability, and whether the CPUID bit must be set is TBD(whp) | -| No paravirtual or synthetic features | No KVM leaves; no KVM Hyper-V enlightenment enabled beyond MSR routing | Synthetic processor features limited to what MSR routing needs — TBD(mshv) | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | VP 0 `IA32_TSC` read and host sample back to back — TBD(kvm) | TBD(mshv) | TBD(whp) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, computed from one host TSC read (Linux 5.16 or newer); no `IA32_TSC` writes | Freeze partition time, write the target to every created VP, thaw at release — TBD(mshv) | Suspend partition time, write the target to every VP, resume at release (first VP run or `WHvResumePartitionTime`); `TscVirtualOffset` is unusable (writes fail) | -| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | -| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification | TBD(mshv) | Raw partitions: at most 67 ns after resume on prometheus28, `azure-windows-1`, and `azure-windows-3`, with no drift; guest-level confirmation and the RDTSC-emulation decision are TBD(whp) | +| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC only, 200 MHz; the emulated APIC is rejected — TBD(whp) | +| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits cleared and CPUID bits cleared — TBD(whp) | +| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition; hosts qualify on the measured warp and rate stability, and whether the CPUID bit must be set is TBD(whp) | +| No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | +| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads (0.1 to 0.6 µs) | TBD(mshv) | TBD(whp) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and thaw at the first VP run (56 to 312 µs for 1 to 8 VPs) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | +| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | +| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | -| Partition capabilities | Derived from CPUID with the identity range masked: `hv1` and `kvm_clock` are false | Same | Same | -| Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor — TBD(mshv) | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | -| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ` | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` (unless the interim applies), leaf `0x15` synthesis | +| Partition capabilities | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | +| Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | +| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | -**WHP interim.** If no WHP primitive keeps live cross-VP skew within 1 µs -after release, WHP keeps its partition-reference-time TSC model for restored -SMP partitions (RDTSC, RDTSCP, and `IA32_TSC` read exits) as a documented -interim, and the case is escalated. The guest-visible ABI is unchanged. -TBD(whp). +**WHP decision gate: met.** Suspending partition time, writing the target, +verifying the frozen values, and resuming kept every pair of vCPUs within +70 ns over 60 s on bare metal and on both Azure runner generations, so WHP +never traps RDTSC. The emulated clock it replaces cost 45 to 75 µs per guest +timestamp read. ## CPU profiles @@ -240,7 +254,8 @@ generation. Profiles are pinned in the OpenVMM source tree, derived by intersecting host fingerprints (the Firecracker `cpu-template-helper` workflow), and immutable once released: any change creates a new revision ID. -**Format.** A profile is a canonical protobuf message: +**Format.** A profile has these fields, in a canonical, deterministic +encoding defined by OpenVMM's `cpu_profile` crate: | Field | Content | | --- | --- | @@ -296,8 +311,7 @@ The boundary is accepted. Beyond it, restore is rejected tolerance is recorded in the manifest as `tsc_tolerance_ppm = 250`; any other recorded value is rejected. Within the tolerance: -- the TSC is never scaled, and RDTSC is never trapped (except the - [WHP interim](#backend-obligations)); +- the TSC is never scaled, and RDTSC is never trapped; - MSR `0x40000022` keeps returning `F_s`, so the guest's `tsc_khz` is unchanged; - guest clocks run fast or slow by at most the deviation (measured hosts @@ -349,7 +363,9 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this 3. **CI warp probe.** Conformance, restore-matrix, and soak runs execute the guest warp probe after boot and after every restore and fail above 1 µs. 4. **Backend live skew.** Each backend shows that live skew stays within the - bound after release (TBD per backend; WHP is a decision gate). + bound after release. The spikes measured at most 63 ns on KVM, at most + 516 ns on MSHV (dual-socket bare metal, bounded by the probe's round + trip), and at most 70 ns on WHP. The guest warp probe (`nvx-time-probe warp` in the fleet tooling) runs two tests on every pair of online CPUs, each for at least 100 ms per pair: @@ -397,11 +413,12 @@ path, a UTC delta that differs from `D` by more than 1 s is logged as a host wall-clock step and does not change `D`. The packet reports the source, so the guest and tests can tell the two paths apart. -The restore anchor is the instant of the synchronized TSC set. On KVM the -guest TSC runs from that instant; on MSHV and WHP it is frozen until the -first VP runs. Either way, the guest monotonic advance across a restore lies -in `[D, D + R]`, where `R` is the time from the restore anchor to the first -restored instruction. Wall-clock repair absorbs `R` for `CLOCK_REALTIME`. +The restore anchor is the instant of the synchronized TSC set. On KVM and +WHP the guest TSC runs from that instant (WHP resumes partition time right +after the read-back); on MSHV it is frozen until the first VP runs. Either +way, the guest monotonic advance across a restore lies in `[D, D + R]`, +where `R` is the time from the restore anchor to the first restored +instruction. Wall-clock repair absorbs `R` for `CLOCK_REALTIME`. ## Restore algorithm @@ -466,14 +483,17 @@ In the worker, with every VP stopped: from saved VP state are superseded and never applied afterwards. 13. Read back: every instantiated VP holds the synchronized value as defined by its backend (`E_TSC_SYNC_READBACK`). -14. Advance every VP's counting-mode LAPIC timer by `D` at `L`. +14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every + VP's LAPIC state again after the synchronized set, even when no timer is + armed: KVM derives its timer deadline from the guest TSC when the LAPIC + state is set. 15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT catches up from its saved VM-time cursor; it is idle. 16. Seal the time fields of the restore packet: `D`, its source, the rate deviation, `g`, and the test-hook flag. 17. Start the state units, with host input gated when the restore requires an - acknowledgement; publish the readiness event; release the VPs. On MSHV - and WHP, partition time thaws when the first VP runs. + acknowledgement; publish the readiness event; release the VPs. On MSHV, + partition time thaws when the first VP runs. The guest then repairs its clocks (see [Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. @@ -634,9 +654,9 @@ theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 The same bracket applies to `utc_ns` in the restore packet, around the first `0xa5` write. A sample costs one write and four reads. OpenVMM cannot observe -a running VP's TSC on every backend, and frozen time on MSHV and WHP thaws at -an unobservable instant, so the guest forms the UTC and TSC pair itself -through this bracket; its `CLOCK_REALTIME` is derived from the TSC. +a running VP's TSC on every backend, and frozen time on MSHV thaws at an +unobservable instant, so the guest forms the UTC and TSC pair itself through +this bracket; its `CLOCK_REALTIME` is derived from the TSC. ### Uncertainty bounds @@ -707,8 +727,9 @@ Capture and restore checks never wait. The boot `C7` check and the deferred restore `C7` check may poll for at most 200 ms, because a CPU switches to one-shot mode at its first tick. The CI conformance suite adds an exhaustive check: every leaf `0x40000006..=0x400000ff`, writes of 0 and 1 to -`0x40000118` and of 2 (which must fail), and writes to every read-only -identity MSR (which must fail). +`0x40000118` and of 2 (which must fail), writes to every read-only +identity MSR (which must fail), and reads of `IA32_TSC_ADJUST` and +`IA32_TSC_DEADLINE` (which must fail). On success the check prints one line to the console: @@ -960,7 +981,7 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | `H1` | The backend device or API is present and usable | | `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`) | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | -| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d` | +| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d`; on Linux the host clocksource is `tsc` (KVM rewrites per-vCPU TSC offsets on a host with an unstable TSC) | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | | `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000 | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | @@ -1082,10 +1103,10 @@ Removed from OpenVMM: advancement (`advance_tsc`), TSC-deadline advancement, periodic LAPIC advancement, and wall-clock-only downtime (`calculate_snapshot_downtime`). - KVM: `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, KVM CPUID leaves, - and `KVM_SET_TSC_KHZ`. + `KVM_SET_TSC_KHZ`, and restore-time `IA32_TSC` writes. - MSHV: BSP-copy TSC alignment and exact rate equality. - WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` - unless the WHP interim applies. + with its RDTSC, RDTSCP, and `IA32_TSC` exits. - Restore packets v1 to v3 and manifest versions 2 to 5. Removed from NVX: @@ -1143,8 +1164,7 @@ implementation lands: 2. **Restoring a snapshot.** Replace the paragraphs on MSHV `IA32_TSC_ADJUST`, KVM TSC offset advancement, MSHV and WHP time freezing and BSP alignment, and the WHP partition-reference-time TSC with one - paragraph on the synchronized TSC set and its read-back. Keep the WHP - interim paragraph only if it applies. + paragraph on the synchronized TSC set and its read-back. 3. **Generation ID.** Extend the portb paragraph with selector `0xa7`, window port `0xeb`, status bit 6, four-byte reads, and packet v4. 4. **Device configuration on restore.** Replace the CPU contract, TSC From 17ba6fb4402f5c2d15bf16778c2db940215ede2d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 02:26:51 -0700 Subject: [PATCH 009/188] kernel: keep the debug kernel's RCU stall timeout fixed The debug-kernel fragment and doc/build.md suggested shortening the RCU stall timeout per boot with rcupdate.rcu_cpu_stall_timeout=. The time ABI's conformance check C8 requires rcu_cpu_stall_timeout to be 21 s, and the guest's RCU grace-period release waits for that timeout, so a shortened value would fail the boot check on the debug kernel CI uses to prove that restores never trip the watchdogs. Only suggest the soft-lockup and hung-task threshold overrides and say why the RCU stall timeout stays at the production value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 9 ++++++--- kernel/config-microvm-debug | 7 ++++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/doc/build.md b/doc/build.md index 696f7d2c2..63de46925 100644 --- a/doc/build.md +++ b/doc/build.md @@ -192,9 +192,12 @@ matching base assignment before `olddefconfig`. It enables `DEBUG_KERNEL`, the soft-lockup and hung-task detectors, and extra RCU stall diagnostics. It pins the production RCU stall timeouts and turns off the debug options that `DEBUG_KERNEL` would otherwise enable by default. The build fails if the -detectors are missing from the generated config. CI can shorten the thresholds -for one boot with `watchdog_thresh=`, `rcupdate.rcu_cpu_stall_timeout=`, and -`sysctl.kernel.hung_task_timeout_secs=`. +detectors are missing from the generated config. CI can shorten the soft-lockup +and hung-task thresholds for one boot with `watchdog_thresh=` and +`sysctl.kernel.hung_task_timeout_secs=`. The RCU stall timeout stays at the +production value because the time ABI's +[conformance check C8](design/time-abi.md#conformance-checks-and-the-nvx-time-abi-marker) +requires it. The variant builds in `build/linux-debug`, so it never invalidates the production object cache, and it produces: diff --git a/kernel/config-microvm-debug b/kernel/config-microvm-debug index 68994a1f5..fcf84f980 100644 --- a/kernel/config-microvm-debug +++ b/kernel/config-microvm-debug @@ -5,9 +5,10 @@ # CI can prove that snapshot restores never trip them. Nothing else may differ # from production: DEBUG_KERNEL would otherwise default DEBUG_MISC, RCU_TRACE, # and X86_DEBUG_FPU on, and the RCU stall settings are pinned to the production -# values. CI can shorten the thresholds per boot with `watchdog_thresh=`, -# `rcupdate.rcu_cpu_stall_timeout=`, and -# `sysctl.kernel.hung_task_timeout_secs=` instead of rebuilding. +# values. CI can shorten the soft-lockup and hung-task thresholds per boot with +# `watchdog_thresh=` and `sysctl.kernel.hung_task_timeout_secs=` instead of +# rebuilding. The RCU stall timeout stays at 21 s because the time ABI's +# conformance check C8 requires it. CONFIG_DEBUG_KERNEL=y # CONFIG_DEBUG_MISC is not set # CONFIG_RCU_TRACE is not set From ead100cca8a4716776395025eef5bba07e597a41 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 03:26:03 -0700 Subject: [PATCH 010/188] guest: add the nvx-time guest time component Time ABI v1 gives the guest four obligations: boot, capture, and restore conformance checks with the NVX-TIME-ABI marker; a kernel-log violation watcher; the snapshot agent's time steps around restore packet v4; and wall-clock discipline from the portb time sample. nvx-time implements them in one static helper, so the checks, the watcher, and the discipline share one daemon and one view of the generation counter. - boot runs C1 to C12 once, publishes /run/nvx/time/state, and starts the daemon only after it reports its descriptors ready. check runs the capture and restore subsets, which never wait. /proc/timer_list and /proc/cpuinfo are streamed, because their size depends on the workload's timers and the vCPU count. - The daemon follows /dev/kmsg for the spec's violation patterns, polls rcu_stall_count, and disciplines CLOCK_REALTIME through adjtimex at the spec's cadence and bounds. - pre-capture, capture, cancel-capture, and restore-finish implement snapshot steps 1 to 3 and 5 to 13: stall suppression, packet v4 with four-byte reads, the bracketed UTC step, the acknowledgement, and the RCU grace-period release off the latency path. - capture holds the portb lock from the capture request to the restored clock, and a discipline poll holds it from its sample to the published state, so a snapshot never contains a half-applied poll. - Sampling first checks the time-window status bit, so a VMM without the window never sees selector 0xa7 or port 0xeb. - --report-only, or nvx_time_abi=report-only on the kernel command line, reports violations under the NVX-TIME-REPORT prefix without powering off, for VMMs that predate the ABI. Nothing invokes nvx-time yet. init and nvx-snapshot switch to it with the OpenVMM change that implements restore packet v4 and the time sample. scripts/test_guest_time.py checks the decoders and the correction and watcher logic against fixtures derived from doc/design/time-abi.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-nvx/action.yml | 4 +- doc/build.md | 17 +- guest/common/nvx-time.c | 3422 +++++++++++++++++++++++ scripts/nvx_tools/build_constants.py | 1 + scripts/test_guest_time.py | 543 ++++ 5 files changed, 3980 insertions(+), 7 deletions(-) create mode 100644 guest/common/nvx-time.c create mode 100644 scripts/test_guest_time.py diff --git a/.github/actions/validate-nvx/action.yml b/.github/actions/validate-nvx/action.yml index 718f92f19..24f580f5b 100644 --- a/.github/actions/validate-nvx/action.yml +++ b/.github/actions/validate-nvx/action.yml @@ -10,7 +10,7 @@ runs: run: | set -euo pipefail python3 -m compileall -q scripts - python3 -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py -v + python3 -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py -v python3 scripts/test_adversarial.py -v python3 .github/skills/nvx-host-connect/scripts/test_hosts.py -v python3 scripts/nvx.py --help @@ -28,7 +28,7 @@ runs: $ErrorActionPreference = "Stop" python -m compileall -q scripts if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - python -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py -v + python -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py -v if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\test_adversarial.py -v if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/doc/build.md b/doc/build.md index 63de46925..227928606 100644 --- a/doc/build.md +++ b/doc/build.md @@ -140,11 +140,18 @@ complete per-scenario logs under The Alpine initramfs includes the sandbox PID-1 bootstrap, its container namespace helpers, the static `nvx-device-io` benchmark helper, the static `nvx-port-io` -restore packet helper, and the static `nvx-time-probe` guest time probe under -`/sbin`. The probe measures TSC and clock read latency, reports the guest's -clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU -warp test that the [time ABI](design/time-abi.md#cross-vcpu-skew-bound) uses to -enforce its 1 µs skew bound. The matching kernel enables +restore packet helper, the static `nvx-time-probe` guest time probe, and the +static `nvx-time` guest time component under `/sbin`. The probe measures TSC and +clock read latency, reports the guest's clock, clocksource, TSC flags, and +hypervisor identity, and runs the cross-vCPU warp test that the +[time ABI](design/time-abi.md#cross-vcpu-skew-bound) uses to enforce its 1 µs +skew bound. `nvx-time` implements the time ABI's +[guest obligations](design/time-abi.md#guest-obligations): the conformance +checks, the violation watcher, the restore packet v4 and time-sample parsers, +the snapshot agent's time steps, and the wall-clock discipline. Its +`--report-only` flag (or the `nvx_time_abi=report-only` kernel command-line +token) reports violations without powering off, for runs against VMMs that +predate the ABI. The matching kernel enables virtio-blk, compressed EROFS, overlayfs, ext4 scratch, memory cgroups, and cgroup BPF. The build fails if `olddefconfig` drops any required option. The APK manifest records the `blkid` and `util-linux` tools used by the bootstrap diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c new file mode 100644 index 000000000..943b3c137 --- /dev/null +++ b/guest/common/nvx-time.c @@ -0,0 +1,3422 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +// nvx-time: the guest time component of NVX time ABI v1. +// +// doc/design/time-abi.md ("Guest obligations") is the contract this helper +// enforces. It runs the conformance checks, the violation watcher and the +// wall-clock discipline (one daemon), and the snapshot agent's time steps: +// +// boot boot checks, initial clock step, daemon start +// check --phase PHASE [--new-cpus LIST] +// capture or restore checks, for diagnosis +// pre-capture capture checks and watchdog suppression +// capture REQUEST GENERATION_ID ENTROPY_FILE +// capture request, then restore repair +// cancel-capture undo pre-capture after a rejected capture +// restore-finish [--ack] [--new-cpus LIST] +// restore checks, acknowledgement, release +// sample print one host time sample +// generation-id print the VM generation ID +// test NAME ARGUMENTS... pure decoders and checks, for unit tests +// +// --report-only, or nvx_time_abi=report-only on the kernel command line, +// reports failures without powering the VM off and prints NVX-TIME-REPORT +// instead of NVX-TIME-ABI, so that its output never passes for a conformant +// run. Every portb transfer uses four-byte reads: a restore packet without +// memory ranges costs 24 data-port exits. + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef SYS_pidfd_open +#define SYS_pidfd_open 434 +#endif + +#define ARRAY_SIZE(array) (sizeof(array) / sizeof((array)[0])) + +#define NSEC_PER_SEC INT64_C(1000000000) +#define NSEC_PER_MSEC INT64_C(1000000) +#define NSEC_PER_USEC INT64_C(1000) +#define MAX_CPUS 64 + +#define STATUS_CONFORMANCE 193 +#define STATUS_VIOLATION 194 +#define STATUS_REPAIR 195 + +#define PORTB_DATA 0xe9 +#define PORTB_STATUS 0xea +#define PORTB_WINDOW 0xeb +#define PORT_SHUTDOWN 0x604 +#define PORT_SNAPSHOT 0x605 +#define SELECT_PACKET 0xa5 +#define SELECT_GENERATION_ID 0xa6 +#define SELECT_TIME_SAMPLE 0xa7 +#define PORTB_PACKET_AVAILABLE 0x02 +#define PORTB_GENERATION_ID_AVAILABLE 0x20 +#define PORTB_TIME_WINDOW 0x40 +#define SNAPSHOT_ACKNOWLEDGE 2 + +#define PACKET_HEADER_SIZE 32 +#define PACKET_RANGE_SIZE 16 +#define PACKET_ENTROPY_SIZE 64 +#define PACKET_MAX_RANGES 255 +#define PACKET_MAX_SIZE \ + (PACKET_HEADER_SIZE + PACKET_MAX_RANGES * PACKET_RANGE_SIZE + \ + PACKET_ENTROPY_SIZE) +#define PACKET_DOWNTIME_UTC 0x01 +#define PACKET_MEMORY_TARGET 0x02 +#define PACKET_ACK_REQUIRED 0x04 +#define PACKET_TEST_HOOKS 0x08 +#define PACKET_KNOWN_FLAGS 0x0f +#define SAMPLE_SIZE 16 +#define SAMPLE_TEST_HOOKS 0x08 +#define GENERATION_ID_SIZE 16 + +#define RATE_DEVIATION_LIMIT INT64_C(16384000) +#define FREQUENCY_LIMIT (INT64_C(500) << 16) +#define DOWNTIME_LIMIT_NS (INT64_C(2592000) * NSEC_PER_SEC) +#define REPAIR_BOUND_NS NSEC_PER_MSEC +#define BOOT_BOUND_NS NSEC_PER_MSEC +#define POLL_BOUND_NS (50 * NSEC_PER_USEC) +#define SAMPLE_ATTEMPTS 3 +#define STEP_THRESHOLD_NS (128 * NSEC_PER_MSEC) +#define SYNCHRONIZED_AGE_NS (128 * NSEC_PER_SEC) +#define FAST_PERIOD_S 16 +#define SLOW_PERIOD_S 64 +#define FAST_POLLS 4 +#define FAST_TIME_CONSTANT 4 +#define SLOW_TIME_CONSTANT 6 +#define TIMER_LIST_WAIT_NS (200 * NSEC_PER_MSEC) +#define DEFAULT_STALL_TIMEOUT_S 21 +#define EVENT_MAX 512 +#define DETAIL_MAX 384 +#define TEXT_MAX 65536 + +#define RUN_DIR "/run/nvx" +#define TIME_DIR RUN_DIR "/time" +#define STATE_PATH TIME_DIR "/state" +#define STATE_TEMP_PATH TIME_DIR "/.state" +#define STATE_LOCK_PATH TIME_DIR "/state.lock" +#define RATES_PATH TIME_DIR "/rates" +#define DAEMON_PID_PATH TIME_DIR "/daemon.pid" +#define SUPPRESSION_PATH TIME_DIR "/suppression" +#define RESTORE_PATH TIME_DIR "/restore" +#define PORTB_LOCK_PATH RUN_DIR "/portb.lock" +#define RCU_SYNC_PATH RUN_DIR "/rcu-sync" + +#define RCU_STALL_SUPPRESS "/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress" +#define RCU_STALL_TIMEOUT "/sys/module/rcupdate/parameters/rcu_cpu_stall_timeout" +#define RCU_NORMAL "/sys/kernel/rcu_normal" +#define RCU_STALL_COUNT "/sys/kernel/rcu_stall_count" +#define SOFT_WATCHDOG "/proc/sys/kernel/soft_watchdog" +#define HUNG_TASK_TIMEOUT "/proc/sys/kernel/hung_task_timeout_secs" +#define CLOCKSOURCE_DIR "/sys/devices/system/clocksource/clocksource0" + +enum phase { PHASE_BOOT, PHASE_CAPTURE, PHASE_RESTORE, PHASE_RUNTIME }; + +static const char *const k_phase_names[] = {"boot", "capture", "restore", + "runtime"}; + +static bool g_report_only; +static bool g_test; +static uint32_t g_generation; +static bool g_ports; + +// --------------------------------------------------------------------------- +// Small utilities +// --------------------------------------------------------------------------- + +static int64_t clock_ns(clockid_t clock) +{ + struct timespec now; + + clock_gettime(clock, &now); + return (int64_t)now.tv_sec * NSEC_PER_SEC + now.tv_nsec; +} + +static int write_all(int fd, const void *data, size_t size) +{ + const char *bytes = data; + + while (size > 0) { + ssize_t count = write(fd, bytes, size); + + if (count < 0 && errno == EINTR) + continue; + if (count <= 0) + return -1; + bytes += count; + size -= (size_t)count; + } + return 0; +} + +static ssize_t read_bytes(const char *path, void *buffer, size_t size) +{ + int fd = open(path, O_RDONLY | O_CLOEXEC); + size_t total = 0; + + if (fd < 0) + return -1; + while (total < size) { + ssize_t count = read(fd, (char *)buffer + total, size - total); + + if (count < 0 && errno == EINTR) + continue; + if (count < 0) { + int error = errno; + + close(fd); + errno = error; + return -1; + } + if (count == 0) + break; + total += (size_t)count; + } + close(fd); + return (ssize_t)total; +} + +static ssize_t read_text(const char *path, char *buffer, size_t size) +{ + ssize_t count = read_bytes(path, buffer, size - 1); + + if (count < 0) + return -1; + buffer[count] = '\0'; + return count; +} + +typedef void (*line_consumer)(void *context, char *line); + +// Passes each line of TEXT to CONSUME, splitting TEXT in place. +static void for_each_text_line(char *text, line_consumer consume, + void *context) +{ + for (char *line = text; line != NULL && *line != '\0';) { + char *next = strchr(line, '\n'); + + if (next != NULL) + *next++ = '\0'; + consume(context, line); + line = next; + } +} + +// Streams the file at PATH into CONSUME one line at a time, so files whose +// size depends on the workload (/proc/timer_list) or on the CPU count +// (/proc/cpuinfo) are never truncated. Longer lines than the line buffer are +// cut. Returns -1 with errno set on failure. +static int for_each_file_line(const char *path, line_consumer consume, + void *context) +{ + char chunk[16384]; + char line[8192]; + size_t length = 0; + int fd = open(path, O_RDONLY | O_CLOEXEC); + + if (fd < 0) + return -1; + for (;;) { + ssize_t count = read(fd, chunk, sizeof(chunk)); + + if (count < 0 && errno == EINTR) + continue; + if (count < 0) { + int error = errno; + + close(fd); + errno = error; + return -1; + } + if (count == 0) + break; + for (ssize_t index = 0; index < count; index++) { + if (chunk[index] == '\n') { + line[length] = '\0'; + consume(context, line); + length = 0; + } else if (length < sizeof(line) - 1) { + line[length++] = chunk[index]; + } + } + } + close(fd); + if (length > 0) { + line[length] = '\0'; + consume(context, line); + } + return 0; +} + +static int parse_int64(const char *text, int64_t *value) +{ + char *end; + + errno = 0; + *value = strtoll(text, &end, 10); + if (errno != 0 || end == text) + return -1; + while (*end == '\n' || *end == ' ') + end++; + return *end == '\0' ? 0 : -1; +} + +static int read_int64(const char *path, int64_t *value) +{ + char text[64]; + + if (read_text(path, text, sizeof(text)) < 0) + return -1; + return parse_int64(text, value); +} + +static int write_text(const char *path, const char *text) +{ + int fd = open(path, O_WRONLY | O_CLOEXEC); + int result; + + if (fd < 0) + return -1; + result = write_all(fd, text, strlen(text)); + if (close(fd) != 0) + result = -1; + return result; +} + +static int write_int64(const char *path, int64_t value) +{ + char text[32]; + + snprintf(text, sizeof(text), "%" PRId64 "\n", value); + return write_text(path, text); +} + +// Atomically replaces PATH with TEXT through a temporary file in its directory. +static int replace_file(const char *path, const char *temporary, + const char *text) +{ + int fd = open(temporary, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644); + int result; + + if (fd < 0) + return -1; + result = write_all(fd, text, strlen(text)); + if (close(fd) != 0) + result = -1; + if (result == 0 && rename(temporary, path) != 0) + result = -1; + if (result != 0) + unlink(temporary); + return result; +} + +static void make_runtime_directories(void) +{ + (void)mkdir(RUN_DIR, 0755); + (void)mkdir(TIME_DIR, 0755); +} + +static bool ends_with(const char *text, const char *suffix) +{ + size_t length = strlen(text); + size_t suffix_length = strlen(suffix); + + return length >= suffix_length && + strcmp(text + length - suffix_length, suffix) == 0; +} + +static uint32_t read_le32(const uint8_t *bytes) +{ + return (uint32_t)bytes[0] | (uint32_t)bytes[1] << 8 | + (uint32_t)bytes[2] << 16 | (uint32_t)bytes[3] << 24; +} + +static uint64_t read_le64(const uint8_t *bytes) +{ + return (uint64_t)read_le32(bytes) | (uint64_t)read_le32(bytes + 4) << 32; +} + +static void format_hex(const uint8_t *bytes, size_t size, char *text) +{ + static const char digits[] = "0123456789abcdef"; + + for (size_t index = 0; index < size; index++) { + text[index * 2] = digits[bytes[index] >> 4]; + text[index * 2 + 1] = digits[bytes[index] & 0x0f]; + } + text[size * 2] = '\0'; +} + +static int parse_hex(const char *text, uint8_t *bytes, size_t size) +{ + if (strlen(text) != size * 2) + return -1; + for (size_t index = 0; index < size * 2; index++) { + char c = text[index]; + int value; + + if (c >= '0' && c <= '9') + value = c - '0'; + else if (c >= 'a' && c <= 'f') + value = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') + value = c - 'A' + 10; + else + return -1; + if (index % 2 == 0) + bytes[index / 2] = (uint8_t)(value << 4); + else + bytes[index / 2] |= (uint8_t)value; + } + return 0; +} + +// Parses a CPU list such as "0-3,5". Returns the number of CPUs in it. +static int parse_cpu_list(const char *text, bool *cpus) +{ + const char *cursor = text; + int count = 0; + + memset(cpus, 0, sizeof(bool) * MAX_CPUS); + while (*cursor != '\0' && *cursor != '\n') { + char *end; + long first = strtol(cursor, &end, 10); + long last = first; + + if (end == cursor || first < 0) + return -1; + cursor = end; + if (*cursor == '-') { + last = strtol(cursor + 1, &end, 10); + if (end == cursor + 1 || last < first) + return -1; + cursor = end; + } + if (last >= MAX_CPUS) + return -1; + for (long cpu = first; cpu <= last; cpu++) { + if (!cpus[cpu]) { + cpus[cpu] = true; + count++; + } + } + if (*cursor == ',') + cursor++; + else if (*cursor != '\0' && *cursor != '\n') + return -1; + } + return count; +} + +// --------------------------------------------------------------------------- +// Console output, events, and power-off +// --------------------------------------------------------------------------- + +static const char *abi_prefix(void) +{ + return g_report_only ? "NVX-TIME-REPORT" : "NVX-TIME-ABI"; +} + +// Escapes TEXT into OUT, which holds LIMIT characters plus a terminator. An +// escape sequence that would not fit is dropped whole, never split. +static void escape_text(const char *text, char *out, size_t limit) +{ + size_t length = 0; + + for (const unsigned char *c = (const unsigned char *)text; *c != '\0'; + c++) { + char piece[5]; + size_t size; + + if (*c == '"' || *c == '\\') { + piece[0] = '\\'; + piece[1] = (char)*c; + size = 2; + } else if (*c < 0x20 || *c > 0x7e) { + snprintf(piece, sizeof(piece), "\\x%02x", *c); + size = 4; + } else { + piece[0] = (char)*c; + size = 1; + } + if (length + size > limit) + break; + memcpy(out + length, piece, size); + length += size; + } + out[length] = '\0'; +} + +static void console_write(const char *line) +{ + int fd = g_test ? -1 : open("/dev/console", O_WRONLY | O_NOCTTY | O_CLOEXEC); + + if (fd < 0) { + (void)write_all(g_test ? STDOUT_FILENO : STDERR_FILENO, line, + strlen(line)); + return; + } + (void)write_all(fd, line, strlen(line)); + close(fd); +} + +static int enable_ports(void) +{ + if (g_ports) + return 0; + if (g_test) { + errno = EPERM; + return -1; + } + // A four-byte read at port P needs permission for P to P + 3, so the + // reads at 0xe9 and 0xeb cover 0xe9 to 0xee. + if (ioperm(PORTB_DATA, PORTB_WINDOW + 4 - PORTB_DATA, 1) != 0 || + ioperm(PORT_SHUTDOWN, 2, 1) != 0) + return -1; + g_ports = true; + return 0; +} + +// Reads SIZE bytes from PORT four bytes per exit; the device zero-fills +// past the end of the selected record. +static void portb_read(uint16_t port, uint8_t *buffer, size_t size) +{ + for (size_t offset = 0; offset < size; offset += 4) { + uint32_t value = inl(port); + size_t count = size - offset < 4 ? size - offset : 4; + + memcpy(buffer + offset, &value, count); + } +} + +static int lock_file(const char *path) +{ + int fd; + + make_runtime_directories(); + fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0600); + if (fd < 0) + return -1; + while (flock(fd, LOCK_EX) != 0) { + if (errno != EINTR) { + close(fd); + return -1; + } + } + return fd; +} + +static void power_off(int status) +{ + fflush(NULL); + if (enable_ports() == 0) { + outb((unsigned char)status, PORT_SHUTDOWN); + } else { + int fd = open("/dev/port", O_WRONLY | O_CLOEXEC); + unsigned char byte = (unsigned char)status; + + if (fd >= 0) { + ssize_t written = pwrite(fd, &byte, 1, PORT_SHUTDOWN); + + (void)written; + close(fd); + } + } + for (;;) + pause(); +} + +static void format_event(char *line, const char *code, const char *source, + enum phase phase, uint32_t generation, + int64_t boottime_ns, const char *detail) +{ + int prefix = snprintf(line, EVENT_MAX + 1, + "%s-VIOLATION: v=1 code=%s source=%s phase=%s " + "generation=%" PRIu32 " boottime_ns=%" PRId64 + " detail=\"", + abi_prefix(), code, source, k_phase_names[phase], + generation, boottime_ns); + + escape_text(detail, line + prefix, EVENT_MAX - (size_t)prefix - 2); + strcat(line, "\"\n"); +} + +static void count_violation(void); + +// Writes a violation event to the kernel log, the portb data port, and the +// console. Consumers treat identical lines as one event. +static void emit_event(const char *code, const char *source, enum phase phase, + const char *detail) +{ + char line[EVENT_MAX + 1]; + char record[EVENT_MAX + 8]; + int fd; + + format_event(line, code, source, phase, g_generation, + clock_ns(CLOCK_BOOTTIME), detail); + if (g_test) { + console_write(line); + return; + } + fd = open("/dev/kmsg", O_WRONLY | O_CLOEXEC); + if (fd >= 0) { + snprintf(record, sizeof(record), "<2>%s", line); + (void)write_all(fd, record, strlen(record)); + close(fd); + } + if (enable_ports() == 0) { + for (const char *c = line; *c != '\0'; c++) + outb((unsigned char)*c, PORTB_DATA); + } + console_write(line); + count_violation(); +} + +// Reports a fatal time-ABI failure. In report-only mode it returns, and the +// caller abandons the failed step. +static void fail_fatal(int status, const char *code, const char *source, + enum phase phase, const char *format, ...) + __attribute__((format(printf, 5, 6))); + +static void fail_fatal(int status, const char *code, const char *source, + enum phase phase, const char *format, ...) +{ + char detail[DETAIL_MAX]; + va_list arguments; + + va_start(arguments, format); + vsnprintf(detail, sizeof(detail), format, arguments); + va_end(arguments); + emit_event(code, source, phase, detail); + if (!g_report_only) + power_off(status); +} + +// --------------------------------------------------------------------------- +// Published state (/run/nvx/time/state) and private records +// --------------------------------------------------------------------------- + +struct time_state { + uint64_t generation; + uint64_t discontinuities; + char last_discontinuity[16]; + int64_t last_step_ns; + int64_t last_step_realtime_ns; + uint64_t last_downtime_ns; + char last_downtime_source[16]; + uint64_t synchronized; + int64_t offset_ns; + int64_t uncertainty_ns; + int64_t frequency_ppb; + uint64_t samples; + uint64_t rejected_samples; + char last_sample_error[24]; + uint64_t violations; +}; + +static void state_init(struct time_state *state) +{ + memset(state, 0, sizeof(*state)); + snprintf(state->last_discontinuity, sizeof(state->last_discontinuity), + "none"); + snprintf(state->last_downtime_source, sizeof(state->last_downtime_source), + "none"); + snprintf(state->last_sample_error, sizeof(state->last_sample_error), + "none"); +} + +static void copy_word(char *destination, size_t size, const char *value) +{ + snprintf(destination, size, "%.*s", (int)strcspn(value, "\n"), value); +} + +// Parses key=value lines. Unknown keys are ignored; a malformed number in a +// known numeric key fails the parse. +static int state_parse(char *text, struct time_state *state) +{ + struct { + const char *key; + uint64_t *unsigned_value; + int64_t *signed_value; + } const numbers[] = { + {"generation", &state->generation, NULL}, + {"discontinuities", &state->discontinuities, NULL}, + {"last_step_ns", NULL, &state->last_step_ns}, + {"last_step_realtime_ns", NULL, &state->last_step_realtime_ns}, + {"last_downtime_ns", &state->last_downtime_ns, NULL}, + {"synchronized", &state->synchronized, NULL}, + {"offset_ns", NULL, &state->offset_ns}, + {"uncertainty_ns", NULL, &state->uncertainty_ns}, + {"frequency_ppb", NULL, &state->frequency_ppb}, + {"samples", &state->samples, NULL}, + {"rejected_samples", &state->rejected_samples, NULL}, + {"violations", &state->violations, NULL}, + }; + struct { + const char *key; + char *value; + size_t size; + } const words[] = { + {"last_discontinuity", state->last_discontinuity, + sizeof(state->last_discontinuity)}, + {"last_downtime_source", state->last_downtime_source, + sizeof(state->last_downtime_source)}, + {"last_sample_error", state->last_sample_error, + sizeof(state->last_sample_error)}, + }; + + state_init(state); + for (char *line = text; line != NULL && *line != '\0';) { + char *next = strchr(line, '\n'); + char *value = strchr(line, '='); + + if (next != NULL) + *next++ = '\0'; + if (value != NULL) { + *value++ = '\0'; + for (size_t index = 0; index < ARRAY_SIZE(words); index++) { + if (strcmp(line, words[index].key) == 0) + copy_word(words[index].value, words[index].size, value); + } + for (size_t index = 0; index < ARRAY_SIZE(numbers); index++) { + int64_t number; + + if (strcmp(line, numbers[index].key) != 0) + continue; + if (parse_int64(value, &number) != 0) + return -1; + if (numbers[index].unsigned_value != NULL) + *numbers[index].unsigned_value = (uint64_t)number; + else + *numbers[index].signed_value = number; + } + } + line = next; + } + return 0; +} + +static void state_format(const struct time_state *state, char *text, + size_t size) +{ + snprintf(text, size, + "version=1\n" + "generation=%" PRIu64 "\n" + "discontinuities=%" PRIu64 "\n" + "last_discontinuity=%s\n" + "last_step_ns=%" PRId64 "\n" + "last_step_realtime_ns=%" PRId64 "\n" + "last_downtime_ns=%" PRIu64 "\n" + "last_downtime_source=%s\n" + "synchronized=%" PRIu64 "\n" + "offset_ns=%" PRId64 "\n" + "uncertainty_ns=%" PRId64 "\n" + "frequency_ppb=%" PRId64 "\n" + "samples=%" PRIu64 "\n" + "rejected_samples=%" PRIu64 "\n" + "last_sample_error=%s\n" + "violations=%" PRIu64 "\n", + state->generation, state->discontinuities, + state->last_discontinuity, state->last_step_ns, + state->last_step_realtime_ns, state->last_downtime_ns, + state->last_downtime_source, state->synchronized, + state->offset_ns, state->uncertainty_ns, state->frequency_ppb, + state->samples, state->rejected_samples, + state->last_sample_error, state->violations); +} + +static int state_load(struct time_state *state) +{ + char text[2048]; + + if (read_text(STATE_PATH, text, sizeof(text)) < 0) + return -1; + return state_parse(text, state); +} + +static int state_store(const struct time_state *state) +{ + char text[2048]; + + state_format(state, text, sizeof(text)); + return replace_file(STATE_PATH, STATE_TEMP_PATH, text); +} + +typedef void (*state_update)(struct time_state *state, const void *context); + +// Applies UPDATE to the published state under its lock. The helpers and the +// daemon all write the state this way, so updates never interleave. +static int state_apply(state_update update, const void *context) +{ + struct time_state state; + int lock = lock_file(STATE_LOCK_PATH); + int result; + + if (lock < 0) + return -1; + if (state_load(&state) != 0) + state_init(&state); + update(&state, context); + result = state_store(&state); + close(lock); + return result; +} + +static void add_violation(struct time_state *state, const void *context) +{ + (void)context; + state->violations++; +} + +static void count_violation(void) +{ + if (!g_test && access(STATE_PATH, F_OK) == 0) + (void)state_apply(add_violation, NULL); +} + +static int64_t frequency_to_ppb(int64_t frequency) +{ + int64_t scaled = frequency * 1000; + + return (scaled + (scaled >= 0 ? 32768 : -32768)) / 65536; +} + +static int rates_load(uint64_t *tsc_hz, uint64_t *lapic_hz) +{ + char text[256]; + unsigned long long tsc; + unsigned long long lapic; + + if (read_text(RATES_PATH, text, sizeof(text)) < 0 || + sscanf(text, "tsc_hz=%llu lapic_hz=%llu", &tsc, &lapic) != 2) + return -1; + *tsc_hz = tsc; + *lapic_hz = lapic; + return 0; +} + +static int rates_store(uint64_t tsc_hz, uint64_t lapic_hz) +{ + char text[128]; + + snprintf(text, sizeof(text), "tsc_hz=%" PRIu64 "\nlapic_hz=%" PRIu64 "\n", + tsc_hz, lapic_hz); + return replace_file(RATES_PATH, RATES_PATH ".tmp", text); +} + +// The watchdog settings saved before a capture, in the order they are +// written. Missing debug-kernel sysctls are skipped. +static const char *const k_suppressed[] = {RCU_STALL_SUPPRESS, SOFT_WATCHDOG, + HUNG_TASK_TIMEOUT}; +static const int64_t k_suppressed_values[] = {1, 0, 0}; + +static int suppression_save(void) +{ + char text[512] = ""; + size_t length = 0; + int fd; + + for (size_t index = 0; index < ARRAY_SIZE(k_suppressed); index++) { + int64_t value; + + if (access(k_suppressed[index], F_OK) != 0) { + if (index == 0) + return -1; + continue; + } + if (read_int64(k_suppressed[index], &value) != 0) + return -1; + length += (size_t)snprintf(text + length, sizeof(text) - length, + "%s=%" PRId64 "\n", k_suppressed[index], + value); + } + fd = open(SUPPRESSION_PATH, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC, 0600); + if (fd < 0) + return -1; + if (write_all(fd, text, length) != 0) { + close(fd); + unlink(SUPPRESSION_PATH); + return -1; + } + close(fd); + for (size_t index = 0; index < ARRAY_SIZE(k_suppressed); index++) { + if (access(k_suppressed[index], F_OK) == 0 && + write_int64(k_suppressed[index], k_suppressed_values[index]) != 0) + return -1; + } + return 0; +} + +// Writes back every value saved by suppression_save, then forgets them. +static int suppression_restore(char *detail, size_t size) +{ + char text[512]; + char *line = text; + + if (read_text(SUPPRESSION_PATH, text, sizeof(text)) < 0) { + snprintf(detail, size, "read %s: %s", SUPPRESSION_PATH, + strerror(errno)); + return -1; + } + while (*line != '\0') { + char *next = strchr(line, '\n'); + char *value = strchr(line, '='); + + if (next != NULL) + *next++ = '\0'; + if (value == NULL) + break; + *value++ = '\0'; + if (write_text(line, value) != 0) { + snprintf(detail, size, "restore %.200s: %s", line, + strerror(errno)); + return -1; + } + line = next != NULL ? next : line + strlen(line); + } + if (unlink(SUPPRESSION_PATH) != 0) { + snprintf(detail, size, "remove %s: %s", SUPPRESSION_PATH, + strerror(errno)); + return -1; + } + return 0; +} + +// --------------------------------------------------------------------------- +// Decoders and decisions (pure; exercised by `nvx-time test`) +// --------------------------------------------------------------------------- + +struct restore_packet { + uint8_t flags; + uint8_t online_vp_count; + uint8_t range_count; + uint32_t generation; + int32_t rate_deviation; + uint64_t downtime_ns; + uint64_t utc_ns; + uint64_t ranges[PACKET_MAX_RANGES][2]; + uint8_t entropy[PACKET_ENTROPY_SIZE]; +}; + +static int parse_packet_header(const uint8_t *bytes, + struct restore_packet *packet, char *detail, + size_t size) +{ + uint8_t online; + + memset(packet, 0, sizeof(*packet)); + if (memcmp(bytes, "OVR", 3) != 0) { + snprintf(detail, size, "restore packet magic is %02x%02x%02x", + bytes[0], bytes[1], bytes[2]); + return -1; + } + if (bytes[3] != 4) { + snprintf(detail, size, "restore packet version is %u", bytes[3]); + return -1; + } + packet->flags = bytes[4]; + online = bytes[5]; + packet->online_vp_count = online; + packet->range_count = bytes[6]; + packet->generation = read_le32(bytes + 8); + packet->rate_deviation = (int32_t)read_le32(bytes + 12); + packet->downtime_ns = read_le64(bytes + 16); + packet->utc_ns = read_le64(bytes + 24); + if ((packet->flags & ~PACKET_KNOWN_FLAGS) != 0) { + snprintf(detail, size, "restore packet flags 0x%02x set reserved bits", + packet->flags); + return -1; + } + if (online != 0 && online != 1 && online != 2 && online != 4 && + online != 8) { + snprintf(detail, size, "restore packet online VP count is %u", online); + return -1; + } + if ((packet->flags & PACKET_MEMORY_TARGET) == 0 && packet->range_count != 0) { + snprintf(detail, size, + "restore packet has %u memory ranges without a memory target", + packet->range_count); + return -1; + } + if (bytes[7] != 0) { + snprintf(detail, size, "restore packet reserved byte is %u", bytes[7]); + return -1; + } + if (packet->generation == 0) { + snprintf(detail, size, "restore packet generation is 0"); + return -1; + } + if (packet->rate_deviation > RATE_DEVIATION_LIMIT || + packet->rate_deviation < -RATE_DEVIATION_LIMIT) { + snprintf(detail, size, + "restore packet rate deviation %" PRId32 + " exceeds the 250 ppm tolerance", + packet->rate_deviation); + return -1; + } + if (packet->downtime_ns > (uint64_t)DOWNTIME_LIMIT_NS) { + snprintf(detail, size, + "restore packet downtime %" PRIu64 " ns exceeds 30 days", + packet->downtime_ns); + return -1; + } + if (packet->utc_ns == 0 || packet->utc_ns > (uint64_t)INT64_MAX) { + snprintf(detail, size, "restore packet UTC %" PRIu64 " is invalid", + packet->utc_ns); + return -1; + } + return 0; +} + +static size_t packet_body_size(const struct restore_packet *packet) +{ + return (size_t)packet->range_count * PACKET_RANGE_SIZE + PACKET_ENTROPY_SIZE; +} + +static int parse_packet_body(const uint8_t *bytes, + struct restore_packet *packet, char *detail, + size_t size) +{ + for (unsigned index = 0; index < packet->range_count; index++) { + uint64_t start = read_le64(bytes + index * PACKET_RANGE_SIZE); + uint64_t length = read_le64(bytes + index * PACKET_RANGE_SIZE + 8); + + if (length == 0 || start + length < start) { + snprintf(detail, size, "restore memory range %u is invalid", index); + return -1; + } + packet->ranges[index][0] = start; + packet->ranges[index][1] = length; + } + memcpy(packet->entropy, bytes + packet->range_count * PACKET_RANGE_SIZE, + PACKET_ENTROPY_SIZE); + return 0; +} + +struct time_sample { + uint8_t flags; + uint32_t generation; + uint64_t utc_ns; + int64_t t0; + int64_t t1; +}; + +static int parse_sample(const uint8_t *bytes, struct time_sample *sample, + char *detail, size_t size) +{ + uint16_t reserved = (uint16_t)(bytes[2] | bytes[3] << 8); + + sample->flags = bytes[1]; + sample->generation = read_le32(bytes + 4); + sample->utc_ns = read_le64(bytes + 8); + if (bytes[0] != 1) { + snprintf(detail, size, "time sample version is %u", bytes[0]); + return -1; + } + if ((sample->flags & ~SAMPLE_TEST_HOOKS) != 0 || reserved != 0) { + snprintf(detail, size, "time sample flags 0x%02x or reserved 0x%04x", + sample->flags, reserved); + return -1; + } + if (sample->utc_ns == 0 || sample->utc_ns > (uint64_t)INT64_MAX) { + snprintf(detail, size, "time sample UTC %" PRIu64 " is invalid", + sample->utc_ns); + return -1; + } + return 0; +} + +// The host latched UTC between the guest's CLOCK_REALTIME reads T0 and T1. +static void pair_clocks(int64_t t0, int64_t t1, uint64_t utc_ns, + int64_t *theta, int64_t *epsilon) +{ + *epsilon = (t1 - t0) / 2; + *theta = (int64_t)utc_ns - (t0 + (t1 - t0) / 2); +} + +static void offset_to_timeval(int64_t offset_ns, struct timeval *time) +{ + time->tv_sec = (time_t)(offset_ns / NSEC_PER_SEC); + time->tv_usec = (suseconds_t)(offset_ns % NSEC_PER_SEC); + if (time->tv_usec < 0) { + time->tv_sec -= 1; + time->tv_usec += NSEC_PER_SEC; + } +} + +static int64_t restore_frequency(int32_t rate_deviation) +{ + int64_t frequency = -(int64_t)rate_deviation; + + if (frequency > FREQUENCY_LIMIT) + return FREQUENCY_LIMIT; + if (frequency < -FREQUENCY_LIMIT) + return -FREQUENCY_LIMIT; + return frequency; +} + +static int64_t ceil_us(int64_t nanoseconds) +{ + return (nanoseconds + NSEC_PER_USEC - 1) / NSEC_PER_USEC; +} + +// Builds the discipline's correction for an accepted sample: a step at or +// beyond 128 ms, otherwise a PLL slew. Returns true for a step. +static bool plan_correction(int64_t theta, int64_t epsilon, bool fast, + struct timex *tx) +{ + int64_t magnitude = theta < 0 ? -theta : theta; + + memset(tx, 0, sizeof(*tx)); + if (magnitude >= STEP_THRESHOLD_NS) { + tx->modes = ADJ_SETOFFSET | ADJ_NANO; + offset_to_timeval(theta, &tx->time); + return true; + } + tx->modes = ADJ_OFFSET | ADJ_STATUS | ADJ_NANO | ADJ_TIMECONST | + ADJ_MAXERROR | ADJ_ESTERROR; + tx->offset = (long)theta; + tx->status = STA_PLL | STA_NANO; + tx->constant = fast ? FAST_TIME_CONSTANT : SLOW_TIME_CONSTANT; + tx->maxerror = (long)ceil_us(magnitude + epsilon); + tx->esterror = (long)ceil_us(epsilon); + return false; +} + +struct watch_rule { + const char *code; + const char *all[2]; + const char *any[4]; +}; + +static const struct watch_rule k_watch_rules[] = { + {"G_TSC_UNSTABLE", {"Marking TSC unstable due to ", NULL}, {NULL}}, + {"G_CLOCKSOURCE_UNSTABLE", + {"timekeeping watchdog on CPU", "as unstable because the skew is too large"}, + {NULL}}, + {"G_CLOCKSOURCE_SWITCH", {"clocksource: Switched to clocksource ", NULL}, + {NULL}}, + {"G_CLOCKSOURCE_SKEW", + {"clocksource: ", NULL}, + {" ahead of CPU ", " behind CPU ", NULL}}, + {"G_TSC_WARP", + {NULL, NULL}, + {"TSC synchronization [CPU#", "cycles TSC warp between CPUs", + "TSC warped randomly between CPUs", NULL}}, + {"G_TSC_ADJUST", {"TSC ADJUST", NULL}, {NULL}}, + {"G_RCU_STALL", + {"rcu: INFO: ", NULL}, + {" detected stalls on CPUs/tasks", " self-detected stall on CPU", + " detected expedited stalls", NULL}}, + {"G_RCU_STARVED", + {"rcu: ", NULL}, + {" kthread starved for ", " kthread timer wakeup didn't happen for ", + NULL}}, + {"G_SOFT_LOCKUP", {"watchdog: BUG: soft lockup - CPU#", NULL}, {NULL}}, + {"G_HARD_LOCKUP", {"Watchdog detected hard LOCKUP", NULL}, {NULL}}, + {"G_HUNG_TASK", {"INFO: task ", " blocked for more than "}, {NULL}}, + {"G_UNCHECKED_MSR", {"unchecked MSR access error", NULL}, {NULL}}, +}; + +// Returns the code of the first rule whose substrings all appear in MESSAGE. +// The guest's own NVX-TIME lines quote kernel text and are never matched. +static const char *match_watch_rule(const char *message) +{ + if (strncmp(message, "NVX-TIME-", 9) == 0) + return NULL; + for (size_t index = 0; index < ARRAY_SIZE(k_watch_rules); index++) { + const struct watch_rule *rule = &k_watch_rules[index]; + bool all = true; + bool any = rule->any[0] == NULL; + + for (size_t part = 0; part < ARRAY_SIZE(rule->all) && rule->all[part]; + part++) + all = all && strstr(message, rule->all[part]) != NULL; + for (size_t part = 0; part < ARRAY_SIZE(rule->any) && rule->any[part]; + part++) + any = any || strstr(message, rule->any[part]) != NULL; + if (all && any) + return rule->code; + } + return NULL; +} + +struct boot_log { + bool hypervisor; + bool privileges; + bool lapic; + bool overrun; + char last_switch[48]; + char problem[DETAIL_MAX]; +}; + +static void boot_log_add(struct boot_log *log, const char *message) +{ + static const char *const forbidden[] = { + "Fast TSC calibration", "Refined TSC clocksource calibration", + "kvm-clock", "APIC timer: using supplied frequency"}; + static const char switched[] = "clocksource: Switched to clocksource "; + const char *name = strstr(message, switched); + const char *code; + + if (strstr(message, "Hypervisor detected: Microsoft Hyper-V") != NULL) + log->hypervisor = true; + if (strstr(message, "Hyper-V: privilege flags low 0x8860,") != NULL) + log->privileges = true; + if (ends_with(message, "Hyper-V: LAPIC Timer Frequency: 0x989680") || + ends_with(message, "Hyper-V: LAPIC Timer Frequency: 0x1e8480")) + log->lapic = true; + if (name != NULL) { + name += sizeof(switched) - 1; + snprintf(log->last_switch, sizeof(log->last_switch), "%.*s", + (int)strcspn(name, " \n"), name); + } + if (log->problem[0] != '\0') + return; + code = match_watch_rule(message); + if (code != NULL && strcmp(code, "G_CLOCKSOURCE_SWITCH") != 0) { + snprintf(log->problem, sizeof(log->problem), "%s: %.300s", code, + message); + return; + } + for (size_t index = 0; index < ARRAY_SIZE(forbidden); index++) { + if (strncmp(message, "NVX-TIME-", 9) != 0 && + strstr(message, forbidden[index]) != NULL) { + snprintf(log->problem, sizeof(log->problem), + "forbidden record: %.300s", message); + return; + } + } +} + +static bool boot_log_verdict(const struct boot_log *log, char *detail, + size_t size) +{ + if (log->overrun) + snprintf(detail, size, "kernel log records were lost (EPIPE)"); + else if (log->problem[0] != '\0') + snprintf(detail, size, "%s", log->problem); + else if (!log->hypervisor) + snprintf(detail, size, "no 'Hypervisor detected: Microsoft Hyper-V'"); + else if (!log->privileges) + snprintf(detail, size, "no 'Hyper-V: privilege flags low 0x8860,'"); + else if (!log->lapic) + snprintf(detail, size, "no 'Hyper-V: LAPIC Timer Frequency' of 1 GHz " + "or 200 MHz"); + else if (strcmp(log->last_switch, "tsc") != 0) + snprintf(detail, size, "last clocksource switch is to '%s', not tsc", + log->last_switch); + else + return true; + return false; +} + +// --------------------------------------------------------------------------- +// Kernel log +// --------------------------------------------------------------------------- + +typedef void (*kmsg_consumer)(void *context, const char *message); + +// Reads every available /dev/kmsg record from the non-blocking FD and passes +// each message text to CONSUME. Returns 0 when drained, -EPIPE when records +// were overwritten before they were read, or another negative errno. +static int kmsg_drain(int fd, kmsg_consumer consume, void *context) +{ + char record[8192]; + + for (;;) { + ssize_t count = read(fd, record, sizeof(record) - 1); + char *message; + + if (count < 0 && errno == EINTR) + continue; + if (count < 0) + return errno == EAGAIN ? 0 : -errno; + if (count == 0) + return 0; + record[count] = '\0'; + message = strchr(record, ';'); + if (message == NULL) + continue; + message++; + message[strcspn(message, "\n")] = '\0'; + consume(context, message); + } +} + +static void boot_log_consume(void *context, const char *message) +{ + boot_log_add(context, message); +} + +// --------------------------------------------------------------------------- +// Conformance checks +// --------------------------------------------------------------------------- + +struct checks { + enum phase phase; + int failures; + int possible; + int online_count; + bool online[MAX_CPUS]; + uint64_t cpu_khz[MAX_CPUS]; + bool cpu_khz_known[MAX_CPUS]; + uint64_t tsc_hz; + uint64_t lapic_hz; +}; + +static bool check_failed(struct checks *checks, const char *id, + const char *format, ...) + __attribute__((format(printf, 3, 4))); + +// Prints the failure line, emits G_CONFORMANCE_, and powers off with +// status 193. In report-only mode it returns false and the run continues. +static bool check_failed(struct checks *checks, const char *id, + const char *format, ...) +{ + char detail[DETAIL_MAX]; + char escaped[DETAIL_MAX]; + char line[EVENT_MAX + 64]; + char code[32]; + va_list arguments; + + va_start(arguments, format); + vsnprintf(detail, sizeof(detail), format, arguments); + va_end(arguments); + escape_text(detail, escaped, sizeof(escaped) - 1); + snprintf(line, sizeof(line), + "%s: v=1 phase=%s status=fail check=%s detail=\"%s\"\n", + abi_prefix(), k_phase_names[checks->phase], id, escaped); + console_write(line); + snprintf(code, sizeof(code), "G_CONFORMANCE_%s", id); + checks->failures++; + emit_event(code, "conformance", checks->phase, detail); + if (!g_report_only) + power_off(STATUS_CONFORMANCE); + return false; +} + +static int checks_init(struct checks *checks, enum phase phase) +{ + char text[256]; + bool possible[MAX_CPUS]; + + memset(checks, 0, sizeof(*checks)); + checks->phase = phase; + if (read_text("/sys/devices/system/cpu/possible", text, sizeof(text)) < 0) + return -1; + checks->possible = parse_cpu_list(text, possible); + if (read_text("/sys/devices/system/cpu/online", text, sizeof(text)) < 0) + return -1; + checks->online_count = parse_cpu_list(text, checks->online); + return checks->possible > 0 && checks->online_count > 0 ? 0 : -1; +} + +static void cpuid(uint32_t leaf, uint32_t subleaf, uint32_t registers[4]) +{ + __asm__ __volatile__("cpuid" + : "=a"(registers[0]), "=b"(registers[1]), + "=c"(registers[2]), "=d"(registers[3]) + : "a"(leaf), "c"(subleaf)); +} + +// Runs the calling thread on CPU only, so inline CPUID and the msr driver +// read that CPU without inter-processor calls. +static int pin_to_cpu(int cpu) +{ + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (sched_setaffinity(0, sizeof(set), &set) != 0) + return -1; + return sched_getcpu() == cpu ? 0 : -1; +} + +static const uint32_t k_kvm_signature[3] = {0x4b4d564b, 0x564b4d56, + 0x0000004d}; + +static bool all_zero(const uint32_t value[4]) +{ + return (value[0] | value[1] | value[2] | value[3]) == 0; +} + +// C1: the identity leaves, the out-of-range rule, and (on one CPU, because it +// costs 255 CPUID exits) the absence of a KVM signature at any base. +static bool check_identity(struct checks *checks, int cpu, bool scan_bases) +{ + const uint32_t capacity = (uint32_t)checks->possible; + const uint32_t expected[6][4] = { + {0x40000005, 0x7263694d, 0x666f736f, 0x76482074}, + {0x31237648, 0, 0, 0}, + {0x0058564e, 0x00010000, 0, 0}, + {0x00008860, 0, 0, 0x00000100}, + {0, 0xffffffff, 0, 0}, + {capacity, capacity, 0, 0}, + }; + static const uint32_t out_of_range[] = {0x40000006, 0x40000081}; + uint32_t value[4]; + uint32_t highest[4]; + + for (uint32_t index = 0; index < 6; index++) { + cpuid(0x40000000 + index, 0, value); + if (memcmp(value, expected[index], sizeof(value)) != 0) + return check_failed( + checks, "C1", + "cpu %d leaf 0x%08" PRIx32 " is %08" PRIx32 ":%08" PRIx32 + ":%08" PRIx32 ":%08" PRIx32 ", expected %08" PRIx32 + ":%08" PRIx32 ":%08" PRIx32 ":%08" PRIx32, + cpu, 0x40000000 + index, value[0], value[1], value[2], + value[3], expected[index][0], expected[index][1], + expected[index][2], expected[index][3]); + } + cpuid(0, 0, value); + cpuid(value[0], 0, highest); + for (size_t index = 0; index < ARRAY_SIZE(out_of_range); index++) { + cpuid(out_of_range[index], 0, value); + if (!all_zero(value) && memcmp(value, highest, sizeof(value)) != 0) + return check_failed(checks, "C1", + "cpu %d leaf 0x%08" PRIx32 + " is neither zero nor out of range", + cpu, out_of_range[index]); + } + if (!scan_bases) + return true; + for (uint32_t base = 0x40000100; base <= 0x4000ff00; base += 0x100) { + cpuid(base, 0, value); + if (value[1] == k_kvm_signature[0] && value[2] == k_kvm_signature[1] && + value[3] == k_kvm_signature[2]) + return check_failed(checks, "C1", + "cpu %d leaf 0x%08" PRIx32 + " carries the KVM signature", + cpu, base); + } + return true; +} + +static bool check_bit(struct checks *checks, int cpu, const char *name, + uint32_t reg, unsigned bit, bool expected) +{ + if (((reg >> bit) & 1) == (expected ? 1u : 0u)) + return true; + return check_failed(checks, "C2", "cpu %d %s is %s", cpu, name, + expected ? "clear" : "set"); +} + +static bool check_leaf(struct checks *checks, int cpu, uint32_t leaf, + const uint32_t expected[4]) +{ + uint32_t value[4]; + + cpuid(leaf, 0, value); + if (memcmp(value, expected, sizeof(value)) == 0) + return true; + return check_failed(checks, "C2", + "cpu %d leaf 0x%08" PRIx32 " is %08" PRIx32 + ":%08" PRIx32 ":%08" PRIx32 ":%08" PRIx32 + ", expected %08" PRIx32 ":%08" PRIx32 ":%08" PRIx32 + ":%08" PRIx32, + cpu, leaf, value[0], value[1], value[2], value[3], + expected[0], expected[1], expected[2], expected[3]); +} + +// C2: the CPU time bits every profile fixes. +static bool check_time_bits(struct checks *checks, int cpu) +{ + static const uint32_t arat_only[4] = {4, 0, 0, 0}; + static const uint32_t zero[4] = {0, 0, 0, 0}; + static const uint32_t invariant_tsc[4] = {0, 0, 0, 0x100}; + uint32_t value[4]; + uint32_t max_basic; + bool ok; + + cpuid(0, 0, value); + max_basic = value[0]; + cpuid(1, 0, value); + ok = check_bit(checks, cpu, "CPUID.1:ECX[31] (hypervisor)", value[2], 31, + true) && + check_bit(checks, cpu, "CPUID.1:ECX[24] (TSC-deadline)", value[2], + 24, false) && + check_bit(checks, cpu, "CPUID.1:ECX[15] (PDCM)", value[2], 15, + false) && + check_bit(checks, cpu, "CPUID.1:EDX[4] (TSC)", value[3], 4, true); + if (!ok) + return false; + if (max_basic < 6) + return check_failed(checks, "C2", "cpu %d lacks leaf 0x6 (ARAT)", cpu); + if (!check_leaf(checks, cpu, 6, arat_only)) + return false; + if (max_basic >= 7) { + cpuid(7, 0, value); + if (!check_bit(checks, cpu, "CPUID.7.0:EBX[1] (TSC_ADJUST)", value[1], + 1, false)) + return false; + } + if (max_basic >= 0xa && !check_leaf(checks, cpu, 0xa, zero)) + return false; + if (max_basic >= 0x15 && !check_leaf(checks, cpu, 0x15, zero)) + return false; + if (max_basic >= 0x16 && !check_leaf(checks, cpu, 0x16, zero)) + return false; + cpuid(0x80000000, 0, value); + if (value[0] < 0x80000007) + return check_failed(checks, "C2", "cpu %d lacks leaf 0x80000007", cpu); + cpuid(0x80000001, 0, value); + if (!check_bit(checks, cpu, "CPUID.80000001:EDX[27] (RDTSCP)", value[3], + 27, true)) + return false; + return check_leaf(checks, cpu, 0x80000007, invariant_tsc); +} + +static int msr_read(int fd, uint32_t msr, uint64_t *value) +{ + ssize_t count = pread(fd, value, sizeof(*value), (off_t)msr); + + if (count == (ssize_t)sizeof(*value)) + return 0; + return count < 0 ? -errno : -EIO; +} + +// C3: the synthetic MSRs. FULL checks every row; otherwise only the +// declared TSC rate, as the restore phase requires for CPU 0. +static bool check_msrs(struct checks *checks, int cpu, bool full) +{ + static const uint32_t faulting[] = {0x40000000, 0x40000001, 0x40000020}; + char path[64]; + uint64_t value = 0; + bool ok = true; + int error = 0; + int fd; + + snprintf(path, sizeof(path), "/dev/cpu/%d/msr", cpu); + fd = open(path, O_RDONLY | O_CLOEXEC); + if (fd < 0) + return check_failed(checks, "C3", "cpu %d: open %s: %s", cpu, path, + strerror(errno)); + if (full && ((error = msr_read(fd, 0x40000002, &value)) != 0 || + value != (uint64_t)cpu)) + ok = check_failed(checks, "C3", + "cpu %d: MSR 0x40000002 (VP index) is %" PRIu64 + " (error %d)", + cpu, value, error); + if (ok && (error = msr_read(fd, 0x40000022, &value)) != 0) + ok = check_failed(checks, "C3", "cpu %d: MSR 0x40000022 read: %s", + cpu, strerror(-error)); + if (ok && checks->tsc_hz != 0 && value != checks->tsc_hz) + ok = check_failed(checks, "C3", + "cpu %d: MSR 0x40000022 is %" PRIu64 + " Hz, expected %" PRIu64 " Hz", + cpu, value, checks->tsc_hz); + if (ok && checks->tsc_hz == 0) + checks->tsc_hz = value; + if (ok && checks->cpu_khz_known[cpu] && + value / 1000 != checks->cpu_khz[cpu]) + ok = check_failed(checks, "C3", + "cpu %d: floor(F / 1000) is %" PRIu64 + " kHz but cpu MHz is %" PRIu64 " kHz", + cpu, value / 1000, checks->cpu_khz[cpu]); + if (ok && full) { + error = msr_read(fd, 0x40000023, &value); + if (error != 0 || (value != 1000000000 && value != 200000000) || + (checks->lapic_hz != 0 && value != checks->lapic_hz)) + ok = check_failed(checks, "C3", + "cpu %d: MSR 0x40000023 (LAPIC rate) is %" PRIu64 + " (error %d)", + cpu, value, error); + else + checks->lapic_hz = value; + } + if (ok && full && + ((error = msr_read(fd, 0x40000118, &value)) != 0 || value != 1)) + ok = check_failed(checks, "C3", + "cpu %d: MSR 0x40000118 is %" PRIu64 " (error %d)", + cpu, value, error); + for (size_t index = 0; ok && full && index < ARRAY_SIZE(faulting); + index++) { + error = msr_read(fd, faulting[index], &value); + if (error != -EIO) + ok = check_failed(checks, "C3", + "cpu %d: MSR 0x%08" PRIx32 + " read did not fail with EIO (error %d)", + cpu, faulting[index], error); + } + close(fd); + return ok; +} + +static bool has_word(const char *list, const char *word) +{ + size_t length = strlen(word); + + for (const char *cursor = list; (cursor = strstr(cursor, word)) != NULL; + cursor += length) { + bool starts = cursor == list || cursor[-1] == ' ' || cursor[-1] == '\t'; + char after = cursor[length]; + + if (starts && (after == '\0' || after == ' ' || after == '\n')) + return true; + } + return false; +} + +// Parses /proc/cpuinfo line by line: records each CPU's `cpu MHz` in kHz and, +// when FLAGS is set, checks C5 on the CPUs in SELECTED. The first C5 failure +// ends the check. +struct cpuinfo_parse { + struct checks *checks; + const bool *selected; + bool flags; + bool failed; + int cpu; + bool seen[MAX_CPUS]; +}; + +static void cpuinfo_init(struct cpuinfo_parse *parse, struct checks *checks, + const bool *selected, bool flags) +{ + memset(parse, 0, sizeof(*parse)); + parse->checks = checks; + parse->selected = selected; + parse->flags = flags; + parse->cpu = -1; +} + +static void cpuinfo_line(void *context, char *line) +{ + static const char *const required[] = { + "tsc", "constant_tsc", "nonstop_tsc", "tsc_known_freq", + "tsc_reliable", "rdtscp", "hypervisor", "arat"}; + static const char *const forbidden[] = {"tsc_deadline_timer", "tsc_adjust"}; + struct cpuinfo_parse *parse = context; + char *colon = strchr(line, ':'); + int cpu = parse->cpu; + + if (parse->failed || colon == NULL) + return; + if (strncmp(line, "processor", 9) == 0) { + cpu = atoi(colon + 1); + parse->cpu = cpu >= 0 && cpu < MAX_CPUS ? cpu : -1; + if (parse->cpu >= 0) + parse->seen[cpu] = true; + } else if (cpu >= 0 && strncmp(line, "cpu MHz", 7) == 0) { + char fraction[4] = "000"; + const char *dot = strchr(colon, '.'); + unsigned long long whole = strtoull(colon + 1, NULL, 10); + + for (int digit = 0; dot != NULL && digit < 3 && + dot[1 + digit] >= '0' && dot[1 + digit] <= '9'; + digit++) + fraction[digit] = dot[1 + digit]; + parse->checks->cpu_khz[cpu] = whole * 1000 + strtoull(fraction, NULL, 10); + parse->checks->cpu_khz_known[cpu] = true; + } else if (cpu >= 0 && parse->flags && parse->selected[cpu] && + strncmp(line, "flags", 5) == 0) { + for (size_t index = 0; index < ARRAY_SIZE(required); index++) { + if (!has_word(colon + 1, required[index])) { + parse->failed = true; + check_failed(parse->checks, "C5", "cpu %d lacks the %s flag", + cpu, required[index]); + return; + } + } + for (size_t index = 0; index < ARRAY_SIZE(forbidden); index++) { + if (has_word(colon + 1, forbidden[index])) { + parse->failed = true; + check_failed(parse->checks, "C5", "cpu %d has the %s flag", cpu, + forbidden[index]); + return; + } + } + } +} + +static bool cpuinfo_result(const struct cpuinfo_parse *parse) +{ + if (parse->failed) + return false; + for (int index = 0; parse->flags && index < MAX_CPUS; index++) { + if (parse->selected[index] && !parse->seen[index]) + return check_failed(parse->checks, "C5", + "cpu %d is not in /proc/cpuinfo", index); + } + return true; +} + +static bool check_cpuinfo_text(struct checks *checks, char *text, + const bool *selected, bool flags) +{ + struct cpuinfo_parse parse; + + cpuinfo_init(&parse, checks, selected, flags); + for_each_text_line(text, cpuinfo_line, &parse); + return cpuinfo_result(&parse); +} + +static bool check_cpuinfo(struct checks *checks, const bool *selected, + bool flags) +{ + struct cpuinfo_parse parse; + + cpuinfo_init(&parse, checks, selected, flags); + if (for_each_file_line("/proc/cpuinfo", cpuinfo_line, &parse) != 0) + return check_failed(checks, "C5", "read /proc/cpuinfo: %s", + strerror(errno)); + return cpuinfo_result(&parse); +} + +// C6: `tsc` is current, and only tsc, refined-jiffies, or jiffies is +// available. +static bool check_clocksource_values(struct checks *checks, const char *current, + char *available) +{ + bool tsc = false; + + if (strncmp(current, "tsc", 3) != 0 || + (current[3] != '\0' && current[3] != '\n')) + return check_failed(checks, "C6", "current clocksource is %.*s", + (int)strcspn(current, "\n"), current); + for (char *token = strtok(available, " \n"); token != NULL; + token = strtok(NULL, " \n")) { + if (strcmp(token, "tsc") == 0) + tsc = true; + else if (strcmp(token, "refined-jiffies") != 0 && + strcmp(token, "jiffies") != 0) + return check_failed(checks, "C6", "clocksource %.64s is available", + token); + } + if (!tsc) + return check_failed(checks, "C6", "tsc is not an available clocksource"); + return true; +} + +static bool check_clocksource(struct checks *checks) +{ + char current[64]; + char available[256]; + + if (read_text(CLOCKSOURCE_DIR "/current_clocksource", current, + sizeof(current)) < 0 || + read_text(CLOCKSOURCE_DIR "/available_clocksource", available, + sizeof(available)) < 0) + return check_failed(checks, "C6", "clocksource sysfs: %s", + strerror(errno)); + return check_clocksource_values(checks, current, available); +} + +struct tick_section { + bool seen; + int mode; + int state; + char device[48]; + char handler[48]; +}; + +// Line-by-line /proc/timer_list parser. The file lists every active hrtimer +// before the tick devices, so its size depends on the workload. +struct timer_list_parse { + struct tick_section cpus[MAX_CPUS]; + struct tick_section broadcast; + struct tick_section *current; + int tick_mode; +}; + +static void timer_list_init(struct timer_list_parse *parse) +{ + static const struct tick_section empty = {false, -1, -1, "", ""}; + + for (int index = 0; index < MAX_CPUS; index++) + parse->cpus[index] = empty; + parse->broadcast = empty; + parse->current = NULL; + parse->tick_mode = -1; +} + +static void timer_list_line(void *context, char *line) +{ + struct timer_list_parse *parse = context; + struct tick_section *current = parse->current; + char name[48]; + int value; + + if (sscanf(line, "Tick Device: mode: %d", &value) == 1) { + parse->tick_mode = value; + parse->current = NULL; + } else if (strncmp(line, "Broadcast device", 16) == 0) { + parse->current = &parse->broadcast; + parse->current->seen = true; + parse->current->mode = parse->tick_mode; + } else if (sscanf(line, "Per CPU device: %d", &value) == 1) { + parse->current = + value >= 0 && value < MAX_CPUS ? &parse->cpus[value] : NULL; + if (parse->current != NULL) { + parse->current->seen = true; + parse->current->mode = parse->tick_mode; + } + } else if (current != NULL && + sscanf(line, "Clock Event Device: %47s", name) == 1) { + snprintf(current->device, sizeof(current->device), "%s", name); + } else if (current != NULL && sscanf(line, " mode: %d", &value) == 1) { + current->state = value; + } else if (current != NULL && + sscanf(line, " event_handler: %47s", name) == 1) { + snprintf(current->handler, sizeof(current->handler), "%s", name); + } +} + +// C7 verdict: every CPU in SELECTED ticks with the LAPIC in one-shot mode +// through hrtimer_interrupt, and no broadcast device exists. +static bool timer_list_result(const struct timer_list_parse *parse, + const bool *selected, char *detail, size_t size) +{ + if (parse->broadcast.seen && + strcmp(parse->broadcast.device, "") != 0) { + snprintf(detail, size, "broadcast clock event device is %s", + parse->broadcast.device); + return false; + } + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + const struct tick_section *section = &parse->cpus[cpu]; + + if (!selected[cpu]) + continue; + if (!section->seen || strcmp(section->device, "lapic") != 0 || + strcmp(section->handler, "hrtimer_interrupt") != 0 || + section->mode != 1 || (section->state != 3 && section->state != 4)) { + snprintf(detail, size, + "cpu %d tick device %s handler %s tick mode %d state %d", + cpu, section->seen ? section->device : "missing", + section->handler[0] != '\0' ? section->handler : "none", + section->mode, section->state); + return false; + } + } + return true; +} + +static bool timer_list_verdict(char *text, const bool *selected, char *detail, + size_t size) +{ + static struct timer_list_parse parse; + + timer_list_init(&parse); + for_each_text_line(text, timer_list_line, &parse); + return timer_list_result(&parse, selected, detail, size); +} + +// Polls C7 for at most WAIT_NS, because a CPU switches to one-shot mode at +// its first tick. A zero wait checks once. +static bool check_timer_list(struct checks *checks, int64_t wait_ns) +{ + static struct timer_list_parse parse; + char detail[DETAIL_MAX] = ""; + int64_t deadline = clock_ns(CLOCK_MONOTONIC) + wait_ns; + + for (;;) { + timer_list_init(&parse); + if (for_each_file_line("/proc/timer_list", timer_list_line, &parse) != + 0) + return check_failed(checks, "C7", "read /proc/timer_list: %s", + strerror(errno)); + if (timer_list_result(&parse, checks->online, detail, sizeof(detail))) + return true; + if (clock_ns(CLOCK_MONOTONIC) >= deadline) + return check_failed(checks, "C7", "%s", detail); + nanosleep(&(struct timespec){0, 10 * NSEC_PER_MSEC}, NULL); + } +} + +// C8: no VMBus, no cpufreq, and the production RCU stall settings. +static bool check_platform(struct checks *checks) +{ + struct stat info; + int64_t value = -1; + + if (stat("/sys/bus/vmbus", &info) == 0) + return check_failed(checks, "C8", "/sys/bus/vmbus exists"); + if (stat("/sys/devices/system/cpu/cpufreq/policy0", &info) == 0) + return check_failed(checks, "C8", "cpufreq policy0 exists"); + if (read_int64(RCU_STALL_SUPPRESS, &value) != 0 || value != 0) + return check_failed(checks, "C8", "rcu_cpu_stall_suppress is %" PRId64, + value); + if (read_int64(RCU_STALL_TIMEOUT, &value) != 0 || value != 21) + return check_failed(checks, "C8", "rcu_cpu_stall_timeout is %" PRId64, + value); + return true; +} + +// C9: no clock workaround on the kernel command line. +static bool check_cmdline_text(struct checks *checks, char *cmdline) +{ + static const char *const prefixes[] = {"tsc_early_khz=", "lapic_timer_hz="}; + static const char *const tokens[] = {"notsc", "nolapic", "nolapic_timer", + "tsc=unstable", "hpet=force"}; + + for (char *token = strtok(cmdline, " \t\n"); token != NULL; + token = strtok(NULL, " \t\n")) { + bool bad = strncmp(token, "clocksource=", 12) == 0 && + strcmp(token + 12, "tsc") != 0; + + for (size_t index = 0; index < ARRAY_SIZE(prefixes); index++) + bad = bad || strncmp(token, prefixes[index], + strlen(prefixes[index])) == 0; + for (size_t index = 0; index < ARRAY_SIZE(tokens); index++) + bad = bad || strcmp(token, tokens[index]) == 0; + if (bad) + return check_failed(checks, "C9", "command line contains %.120s", + token); + } + return true; +} + +static bool check_cmdline(struct checks *checks) +{ + char text[4096]; + + if (read_text("/proc/cmdline", text, sizeof(text)) < 0) + return check_failed(checks, "C9", "read /proc/cmdline: %s", + strerror(errno)); + return check_cmdline_text(checks, text); +} + +static pid_t daemon_pid(void) +{ + char path[64]; + char comm[32]; + int64_t pid; + + if (read_int64(DAEMON_PID_PATH, &pid) != 0 || pid <= 1 || pid > INT_MAX || + kill((pid_t)pid, 0) != 0) + return -1; + snprintf(path, sizeof(path), "/proc/%" PRId64 "/comm", pid); + if (read_text(path, comm, sizeof(comm)) < 0 || + strcmp(comm, "nvx-time\n") != 0) + return -1; + return (pid_t)pid; +} + +// C10: the daemon runs, has recorded no violation, and no stall was counted. +static bool check_daemon(struct checks *checks) +{ + struct time_state state; + int64_t stalls = -1; + + if (daemon_pid() < 0) + return check_failed(checks, "C10", "the time daemon is not running"); + if (state_load(&state) != 0) + return check_failed(checks, "C10", "time state is unreadable"); + if (state.violations != 0) + return check_failed(checks, "C10", + "the time daemon recorded %" PRIu64 " violations", + state.violations); + if (read_int64(RCU_STALL_COUNT, &stalls) != 0 || stalls != 0) + return check_failed(checks, "C10", "rcu_stall_count is %" PRId64, + stalls); + return true; +} + +// C11: on the CI debug kernel, the soft-lockup and hung-task detectors run. +static bool check_debug_watchdogs(struct checks *checks) +{ + bool soft = access(SOFT_WATCHDOG, F_OK) == 0; + bool hung = access(HUNG_TASK_TIMEOUT, F_OK) == 0; + int64_t value = -1; + + if (!soft && !hung) + return true; + if (!soft || read_int64(SOFT_WATCHDOG, &value) != 0 || value != 1) + return check_failed(checks, "C11", "soft_watchdog is %" PRId64, value); + if (!hung || read_int64(HUNG_TASK_TIMEOUT, &value) != 0 || value == 0) + return check_failed(checks, "C11", + "hung_task_timeout_secs is %" PRId64, value); + return true; +} + +// C1, C2, and C3 on every CPU in SELECTED, pinned to each in turn. The +// three checks are independent, so a report-only run shows each failure; +// the loop stops at the first CPU that fails. +static bool check_cpus(struct checks *checks, const bool *selected, + bool identity, bool scan_once, bool msrs) +{ + cpu_set_t original; + bool scanned = !scan_once; + bool ok = true; + + if (sched_getaffinity(0, sizeof(original), &original) != 0) + return check_failed(checks, "C1", "sched_getaffinity: %s", + strerror(errno)); + for (int cpu = 0; ok && cpu < MAX_CPUS; cpu++) { + if (!selected[cpu]) + continue; + if (pin_to_cpu(cpu) != 0) { + ok = check_failed(checks, "C1", "cannot run on cpu %d: %s", cpu, + strerror(errno)); + break; + } + if (identity) { + if (!check_identity(checks, cpu, !scanned)) + ok = false; + if (!check_time_bits(checks, cpu)) + ok = false; + scanned = true; + } + if (msrs && !check_msrs(checks, cpu, true)) + ok = false; + } + (void)sched_setaffinity(0, sizeof(original), &original); + return ok; +} + +// --------------------------------------------------------------------------- +// Host time samples and clock adjustment +// --------------------------------------------------------------------------- + +// Latches a sample with selector 0xa7 and reads it from 0xeb. CLOCK_REALTIME +// brackets the selector write, so the host instant lies in [t0, t1]. The +// caller holds the portb lock. +static int take_sample(struct time_sample *sample, char *detail, size_t size) +{ + uint8_t bytes[SAMPLE_SIZE]; + + sample->t0 = clock_ns(CLOCK_REALTIME); + outb(SELECT_TIME_SAMPLE, PORTB_STATUS); + sample->t1 = clock_ns(CLOCK_REALTIME); + portb_read(PORTB_WINDOW, bytes, sizeof(bytes)); + return parse_sample(bytes, sample, detail, size); +} + +static int lock_portb(char *detail, size_t size) +{ + int lock = lock_file(PORTB_LOCK_PATH); + + if (lock < 0) + snprintf(detail, size, "portb lock: %s", strerror(errno)); + return lock; +} + +// Takes up to ATTEMPTS samples of generation G and keeps the first whose +// uncertainty is within BOUND_NS. Returns 0 when one is, 1 when only wider +// samples were seen (the narrowest is returned), and -1 when none was valid +// or the VMM has no time-sample window. A sample from another generation +// stops the search with -2. The caller holds the portb lock. +static int best_sample(int attempts, int64_t bound_ns, uint32_t generation, + struct time_sample *best, int64_t *theta, + int64_t *epsilon, char *detail, size_t size) +{ + unsigned status; + int result = -1; + + *epsilon = INT64_MAX; + if (enable_ports() != 0) { + snprintf(detail, size, "ioperm: %s", strerror(errno)); + return -1; + } + status = inb(PORTB_STATUS); + if ((status & PORTB_TIME_WINDOW) == 0) { + snprintf(detail, size, "portb status 0x%02x has no time-sample window", + status); + return -1; + } + for (int attempt = 0; attempt < attempts; attempt++) { + struct time_sample sample; + int64_t sample_theta; + int64_t sample_epsilon; + + if (take_sample(&sample, detail, size) != 0) + continue; + if (sample.generation != generation) { + snprintf(detail, size, + "time sample generation %" PRIu32 ", expected %" PRIu32, + sample.generation, generation); + return -2; + } + pair_clocks(sample.t0, sample.t1, sample.utc_ns, &sample_theta, + &sample_epsilon); + if (sample_epsilon < *epsilon) { + *best = sample; + *theta = sample_theta; + *epsilon = sample_epsilon; + result = 1; + } + if (sample_epsilon <= bound_ns) + return 0; + snprintf(detail, size, + "time sample uncertainty %" PRId64 " ns exceeds %" PRId64 + " ns", + sample_epsilon, bound_ns); + } + return result; +} + +static int step_clock(int64_t offset_ns) +{ + struct timex tx; + + memset(&tx, 0, sizeof(tx)); + tx.modes = ADJ_SETOFFSET | ADJ_NANO; + offset_to_timeval(offset_ns, &tx.time); + return adjtimex(&tx) < 0 ? -1 : 0; +} + +static int set_frequency(int64_t frequency) +{ + struct timex tx; + + memset(&tx, 0, sizeof(tx)); + tx.modes = ADJ_FREQUENCY | ADJ_STATUS; + tx.freq = (long)frequency; + tx.status = STA_PLL | STA_NANO; + return adjtimex(&tx) < 0 ? -1 : 0; +} + +static int64_t current_frequency(void) +{ + struct timex tx; + + memset(&tx, 0, sizeof(tx)); + return adjtimex(&tx) < 0 ? 0 : (int64_t)tx.freq; +} + +// C12: one sample within 1 ms (three attempts), then a clock step. The boot +// step is not a discontinuity. +static bool check_initial_sample(struct checks *checks, int64_t *theta, + int64_t *epsilon) +{ + struct time_sample sample; + char detail[DETAIL_MAX] = "no valid time sample"; + int result; + int lock; + + lock = lock_portb(detail, sizeof(detail)); + if (lock < 0) + return check_failed(checks, "C12", "%s", detail); + result = best_sample(SAMPLE_ATTEMPTS, BOOT_BOUND_NS, g_generation, &sample, + theta, epsilon, detail, sizeof(detail)); + close(lock); + if (result != 0) + return check_failed(checks, "C12", "%s", detail); + if (step_clock(*theta) != 0) + return check_failed(checks, "C12", "adjtimex step: %s", + strerror(errno)); + return true; +} + +// --------------------------------------------------------------------------- +// Daemon: violation watcher, wall-clock discipline, restore completion +// --------------------------------------------------------------------------- + +struct daemon { + int kmsg; + int signals; + int timer; + int polls; + int64_t last_accepted_ns; + uint64_t tsc_hz; + uint64_t lapic_hz; + pid_t release_pid; + bool release_done; + int release_status; + int64_t release_deadline_ns; + int64_t restore_elapsed_us; +}; + +static void watcher_consume(void *context, const char *message) +{ + const char *code = match_watch_rule(message); + + (void)context; + if (code != NULL) + fail_fatal(STATUS_VIOLATION, code, "watcher", PHASE_RUNTIME, "%.400s", + message); +} + +static void watch_kernel_log(struct daemon *daemon) +{ + if (kmsg_drain(daemon->kmsg, watcher_consume, daemon) == -EPIPE) + fail_fatal(STATUS_VIOLATION, "G_KMSG_OVERRUN", "watcher", + PHASE_RUNTIME, "kernel log records were lost (EPIPE)"); +} + +static int arm_timer(struct daemon *daemon) +{ + struct itimerspec period; + + memset(&period, 0, sizeof(period)); + period.it_value.tv_sec = + daemon->polls < FAST_POLLS ? FAST_PERIOD_S : SLOW_PERIOD_S; + return timerfd_settime(daemon->timer, 0, &period, NULL); +} + +struct poll_outcome { + bool accepted; + bool stepped; + bool synchronized; + int64_t theta; + int64_t epsilon; + int64_t step_realtime_ns; + int64_t frequency; +}; + +static void apply_poll(struct time_state *state, const void *context) +{ + const struct poll_outcome *outcome = context; + + if (outcome->accepted) { + state->samples++; + state->offset_ns = outcome->theta; + state->uncertainty_ns = outcome->epsilon; + state->frequency_ppb = frequency_to_ppb(outcome->frequency); + snprintf(state->last_sample_error, sizeof(state->last_sample_error), + "none"); + if (outcome->stepped) { + state->discontinuities++; + snprintf(state->last_discontinuity, + sizeof(state->last_discontinuity), "step"); + state->last_step_ns = outcome->theta; + state->last_step_realtime_ns = outcome->step_realtime_ns; + } + } else { + state->rejected_samples++; + snprintf(state->last_sample_error, sizeof(state->last_sample_error), + "G_SAMPLE_UNCERTAIN"); + } + state->synchronized = outcome->synchronized; +} + +// One discipline poll: the RCU stall backstop, then up to three samples. A +// step reapplies the frequency and status that the step's NTP reset clears. +// The poll holds the portb lock from its first sample to the published state, +// as a capture does from its request to the restored clock, so a snapshot +// never contains a half-applied poll: a poll blocked behind a capture samples +// after the restore, sees the new generation, and is discarded. +static void discipline_poll(struct daemon *daemon) +{ + struct poll_outcome outcome; + struct time_sample sample; + char detail[DETAIL_MAX] = ""; + bool fast = daemon->polls < FAST_POLLS; + int64_t stalls = 0; + int result; + int lock; + + memset(&outcome, 0, sizeof(outcome)); + if (read_int64(RCU_STALL_COUNT, &stalls) == 0 && stalls != 0) + fail_fatal(STATUS_VIOLATION, "G_RCU_STALL", "watcher", PHASE_RUNTIME, + "rcu_stall_count is %" PRId64, stalls); + daemon->polls++; + lock = lock_portb(detail, sizeof(detail)); + result = lock < 0 ? -1 + : best_sample(SAMPLE_ATTEMPTS, POLL_BOUND_NS, + g_generation, &sample, &outcome.theta, + &outcome.epsilon, detail, sizeof(detail)); + if (result == -2) { + close(lock); + return; + } + if (result == 0) { + struct timex tx; + int64_t frequency = current_frequency(); + + outcome.stepped = plan_correction(outcome.theta, outcome.epsilon, fast, + &tx); + outcome.accepted = adjtimex(&tx) >= 0 && + (!outcome.stepped || set_frequency(frequency) == 0); + outcome.step_realtime_ns = clock_ns(CLOCK_REALTIME); + outcome.frequency = current_frequency(); + if (outcome.accepted) + daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); + } + outcome.synchronized = + daemon->last_accepted_ns != 0 && + clock_ns(CLOCK_MONOTONIC) - daemon->last_accepted_ns <= + SYNCHRONIZED_AGE_NS; + (void)state_apply(apply_poll, &outcome); + if (lock >= 0) + close(lock); + if (!outcome.accepted) { + char line[192]; + + snprintf(line, sizeof(line), + "%s: v=1 phase=runtime status=uncertain " + "code=G_SAMPLE_UNCERTAIN epsilon_ns=%" PRId64 "\n", + abi_prefix(), result == 1 ? outcome.epsilon : -1); + console_write(line); + } +} + +// Waits for a full normal RCU grace period: with rcu_normal set, the +// synchronize_rcu_expedited() in every unmount waits for one that starts +// after the call, on any number of CPUs. Runs in a child of the daemon. +static void release_child(void) +{ + int64_t normal = 0; + bool saved = read_int64(RCU_NORMAL, &normal) == 0 && + write_int64(RCU_NORMAL, 1) == 0; + int status = 1; + + if (saved && unshare(CLONE_NEWNS) == 0 && + mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL) == 0) { + (void)mkdir(RCU_SYNC_PATH, 0700); + if (mount("nvx-rcu-sync", RCU_SYNC_PATH, "tmpfs", + MS_NOSUID | MS_NODEV | MS_NOEXEC, "size=4k") == 0 && + umount2(RCU_SYNC_PATH, 0) == 0) + status = 0; + (void)rmdir(RCU_SYNC_PATH); + } + if (saved && write_int64(RCU_NORMAL, normal) != 0) + status = 1; + _exit(status); +} + +// Runs the grace-period release in a child and waits at most TIMEOUT_S +// seconds for it, for callers without the daemon's event loop. A release +// still pending at the deadline counts as done: the grace period is stuck, +// and the kernel then reports the real stall. +static bool release_with_deadline(int64_t timeout_s) +{ + int64_t deadline = clock_ns(CLOCK_MONOTONIC) + timeout_s * NSEC_PER_SEC; + sigset_t mask; + sigset_t previous; + bool released = false; + pid_t pid; + + sigemptyset(&mask); + sigaddset(&mask, SIGCHLD); + sigprocmask(SIG_BLOCK, &mask, &previous); + pid = fork(); + if (pid == 0) { + sigprocmask(SIG_SETMASK, &previous, NULL); + release_child(); + } + while (pid > 0) { + int status; + pid_t done = waitpid(pid, &status, WNOHANG); + int64_t remaining = deadline - clock_ns(CLOCK_MONOTONIC); + + if (done == pid) { + released = WIFEXITED(status) && WEXITSTATUS(status) == 0; + break; + } + if (done < 0 && errno != EINTR) + break; + if (remaining <= 0) { + released = true; + break; + } + (void)sigtimedwait(&mask, NULL, + &(struct timespec){remaining / NSEC_PER_SEC, + remaining % NSEC_PER_SEC}); + } + sigprocmask(SIG_SETMASK, &previous, NULL); + return released; +} + +static void print_restore_marker(int failures, int cpus, uint64_t tsc_hz, + uint64_t lapic_hz, int64_t elapsed_us) +{ + char line[256]; + + snprintf(line, sizeof(line), + "%s: v=1 phase=restore status=%s cpus=%d tsc_hz=%" PRIu64 + " lapic_hz=%" PRIu64 " generation=%" PRIu32 + " elapsed_us=%" PRId64 "\n", + abi_prefix(), failures == 0 ? "ok" : "fail", cpus, tsc_hz, + lapic_hz, g_generation, elapsed_us); + console_write(line); +} + +// Step 13 after the grace period ended (or the stall timeout passed): put +// the watchdog settings back, run the deferred C7, print the restore marker, +// and restart the discipline at the fast cadence. +static void complete_restore(bool released, uint64_t tsc_hz, uint64_t lapic_hz, + int64_t elapsed_us) +{ + struct checks checks; + char detail[DETAIL_MAX]; + + if (!released) + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "the RCU grace-period release failed"); + if (suppression_restore(detail, sizeof(detail)) != 0) + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "%s", detail); + if (checks_init(&checks, PHASE_RESTORE) != 0) + check_failed(&checks, "C7", "cannot read the online CPU set"); + else + check_timer_list(&checks, TIMER_LIST_WAIT_NS); + print_restore_marker(checks.failures, checks.online_count, tsc_hz, lapic_hz, + elapsed_us); +} + +static void start_restore(struct daemon *daemon) +{ + char text[256]; + unsigned long long generation; + long long elapsed; + int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; + pid_t pid; + + if (daemon->release_pid > 0) + return; + if (read_text(RESTORE_PATH, text, sizeof(text)) < 0 || + sscanf(text, "generation=%llu elapsed_us=%lld", &generation, + &elapsed) != 2) { + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "the restore record is unreadable"); + return; + } + g_generation = (uint32_t)generation; + daemon->restore_elapsed_us = elapsed; + (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); + pid = fork(); + if (pid == 0) + release_child(); + if (pid < 0) { + complete_restore(false, daemon->tsc_hz, daemon->lapic_hz, elapsed); + return; + } + daemon->release_pid = pid; + daemon->release_done = false; + daemon->release_deadline_ns = + clock_ns(CLOCK_MONOTONIC) + timeout_s * NSEC_PER_SEC; +} + +static void reap_children(struct daemon *daemon) +{ + int status; + pid_t pid; + + while ((pid = waitpid(-1, &status, WNOHANG)) > 0) { + if (pid == daemon->release_pid) { + daemon->release_done = true; + daemon->release_status = status; + } + } +} + +static void handle_signals(struct daemon *daemon) +{ + struct signalfd_siginfo info; + + while (read(daemon->signals, &info, sizeof(info)) == sizeof(info)) { + if (info.ssi_signo == SIGTERM) + _exit(0); + if (info.ssi_signo == SIGUSR1) + start_restore(daemon); + else if (info.ssi_signo == SIGCHLD) + reap_children(daemon); + } +} + +// Sets up the daemon's signal and timer descriptors and its OOM exemption. +// Returns 0 or an errno value. +static int daemon_setup(struct daemon *daemon) +{ + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, SIGUSR1); + sigaddset(&mask, SIGTERM); + sigaddset(&mask, SIGCHLD); + if (sigprocmask(SIG_BLOCK, &mask, NULL) != 0) + return errno; + signal(SIGHUP, SIG_IGN); + signal(SIGPIPE, SIG_IGN); + daemon->signals = signalfd(-1, &mask, SFD_CLOEXEC | SFD_NONBLOCK); + if (daemon->signals < 0) + return errno; + daemon->timer = timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC | TFD_NONBLOCK); + if (daemon->timer < 0) + return errno; + if (write_text("/proc/self/oom_score_adj", "-1000") != 0 || + arm_timer(daemon) != 0) + return errno; + return 0; +} + +static void daemon_loop(struct daemon *daemon) +{ + for (;;) { + struct pollfd fds[3] = {{daemon->kmsg, POLLIN, 0}, + {daemon->signals, POLLIN, 0}, + {daemon->timer, POLLIN, 0}}; + int timeout = -1; + + if (daemon->release_pid > 0 && !daemon->release_done) { + int64_t remaining = + daemon->release_deadline_ns - clock_ns(CLOCK_MONOTONIC); + + timeout = remaining <= 0 ? 0 + : (int)((remaining + NSEC_PER_MSEC - 1) / + NSEC_PER_MSEC); + } + if (poll(fds, ARRAY_SIZE(fds), timeout) < 0 && errno != EINTR) { + sleep(1); + continue; + } + if ((fds[0].revents & (POLLIN | POLLERR)) != 0) + watch_kernel_log(daemon); + if ((fds[1].revents & POLLIN) != 0) + handle_signals(daemon); + if ((fds[2].revents & POLLIN) != 0) { + uint64_t expirations; + ssize_t count = read(daemon->timer, &expirations, + sizeof(expirations)); + + (void)count; + discipline_poll(daemon); + arm_timer(daemon); + } + if (daemon->release_pid > 0 && + (daemon->release_done || + clock_ns(CLOCK_MONOTONIC) >= daemon->release_deadline_ns)) { + bool released = !daemon->release_done || + (WIFEXITED(daemon->release_status) && + WEXITSTATUS(daemon->release_status) == 0); + + daemon->release_pid = 0; + complete_restore(released, daemon->tsc_hz, daemon->lapic_hz, + daemon->restore_elapsed_us); + daemon->polls = 0; + daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); + arm_timer(daemon); + } + } +} + +// Forks the daemon, which keeps reading the kernel log from where the boot +// check stopped, waits until the daemon reports its descriptors ready, and +// publishes its PID. +static int start_daemon(int kmsg, uint64_t tsc_hz, uint64_t lapic_hz, + int64_t last_accepted_ns) +{ + char text[32]; + int ready[2]; + int error = 0; + ssize_t count; + pid_t pid; + + if (pipe2(ready, O_CLOEXEC) != 0) + return -1; + pid = fork(); + if (pid < 0) { + error = errno; + close(ready[0]); + close(ready[1]); + errno = error; + return -1; + } + if (pid == 0) { + struct daemon daemon; + int null = open("/dev/null", O_RDWR | O_CLOEXEC); + + close(ready[0]); + memset(&daemon, 0, sizeof(daemon)); + daemon.kmsg = kmsg; + daemon.tsc_hz = tsc_hz; + daemon.lapic_hz = lapic_hz; + daemon.last_accepted_ns = last_accepted_ns; + (void)setsid(); + if (null >= 0) { + (void)dup2(null, STDIN_FILENO); + (void)dup2(null, STDOUT_FILENO); + (void)dup2(null, STDERR_FILENO); + close(null); + } + error = chdir("/") != 0 ? errno : daemon_setup(&daemon); + (void)write_all(ready[1], &error, sizeof(error)); + close(ready[1]); + if (error != 0) + _exit(1); + daemon_loop(&daemon); + _exit(0); + } + close(ready[1]); + do + count = read(ready[0], &error, sizeof(error)); + while (count < 0 && errno == EINTR); + close(ready[0]); + if (count != (ssize_t)sizeof(error) || error != 0) { + (void)kill(pid, SIGKILL); + (void)waitpid(pid, NULL, 0); + errno = count == (ssize_t)sizeof(error) ? error : EIO; + return -1; + } + snprintf(text, sizeof(text), "%d\n", (int)pid); + return replace_file(DAEMON_PID_PATH, DAEMON_PID_PATH ".tmp", text); +} + +// --------------------------------------------------------------------------- +// Commands +// --------------------------------------------------------------------------- + +static void print_marker(const struct checks *checks, int64_t start_ns) +{ + char line[256]; + char failures[32] = ""; + + if (checks->failures != 0) + snprintf(failures, sizeof(failures), " failures=%d", checks->failures); + snprintf(line, sizeof(line), + "%s: v=1 phase=%s status=%s cpus=%d tsc_hz=%" PRIu64 + " lapic_hz=%" PRIu64 " generation=%" PRIu32 + " elapsed_us=%" PRId64 "%s\n", + abi_prefix(), k_phase_names[checks->phase], + checks->failures == 0 ? "ok" : "fail", checks->online_count, + checks->tsc_hz, checks->lapic_hz, g_generation, + (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC, failures); + console_write(line); +} + +static void load_generation(void) +{ + struct time_state state; + + if (state_load(&state) == 0) + g_generation = (uint32_t)state.generation; +} + +static int cmd_boot(void) +{ + int64_t start = clock_ns(CLOCK_MONOTONIC); + struct checks checks; + struct boot_log log; + struct time_state state; + char detail[DETAIL_MAX]; + int64_t theta = 0; + int64_t epsilon = 0; + int64_t accepted_ns = 0; + int kmsg; + + g_generation = 0; + make_runtime_directories(); + if (checks_init(&checks, PHASE_BOOT) != 0) { + check_failed(&checks, "C1", "cannot read the CPU sets"); + return 1; + } + check_cmdline(&checks); + check_platform(&checks); + check_clocksource(&checks); + memset(&log, 0, sizeof(log)); + kmsg = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC); + if (kmsg < 0) { + check_failed(&checks, "C4", "open /dev/kmsg: %s", strerror(errno)); + } else { + log.overrun = kmsg_drain(kmsg, boot_log_consume, &log) == -EPIPE; + if (!boot_log_verdict(&log, detail, sizeof(detail))) + check_failed(&checks, "C4", "%s", detail); + } + check_cpuinfo(&checks, checks.online, true); + check_cpus(&checks, checks.online, true, true, true); + check_timer_list(&checks, TIMER_LIST_WAIT_NS); + check_debug_watchdogs(&checks); + if (check_initial_sample(&checks, &theta, &epsilon)) + accepted_ns = clock_ns(CLOCK_MONOTONIC); + state_init(&state); + state.synchronized = accepted_ns != 0; + state.samples = accepted_ns != 0; + state.offset_ns = accepted_ns != 0 ? theta : 0; + state.uncertainty_ns = accepted_ns != 0 ? epsilon : 0; + state.frequency_ppb = frequency_to_ppb(current_frequency()); + if (rates_store(checks.tsc_hz, checks.lapic_hz) != 0 || + state_store(&state) != 0) + check_failed(&checks, "C10", "cannot publish the time state: %s", + strerror(errno)); + else if (kmsg >= 0 && + start_daemon(kmsg, checks.tsc_hz, checks.lapic_hz, accepted_ns) != + 0) + check_failed(&checks, "C10", "cannot start the time daemon: %s", + strerror(errno)); + print_marker(&checks, start); + return 0; +} + +// Capture checks (C6, C7, and C10); they never wait. +static void run_capture_checks(struct checks *checks) +{ + (void)rates_load(&checks->tsc_hz, &checks->lapic_hz); + check_clocksource(checks); + check_timer_list(checks, 0); + check_daemon(checks); +} + +// Restore checks: C1, C2, and C5 on newly onlined CPUs, C3 for CPU 0, C6, +// and C10. They never wait. +static void run_restore_checks(struct checks *checks, const bool *new_cpus, + int new_count) +{ + uint64_t tsc_hz = 0; + uint64_t lapic_hz = 0; + + if (rates_load(&tsc_hz, &lapic_hz) != 0) { + check_failed(checks, "C3", "the boot rates are unavailable"); + } else { + checks->tsc_hz = tsc_hz; + checks->lapic_hz = lapic_hz; + } + if (new_count > 0) { + check_cpus(checks, new_cpus, true, false, false); + check_cpuinfo(checks, new_cpus, true); + } + check_msrs(checks, 0, false); + check_clocksource(checks); + check_daemon(checks); +} + +static int parse_new_cpus(const char *text, bool *cpus) +{ + if (strcmp(text, "") == 0 || strcmp(text, "none") == 0) { + memset(cpus, 0, sizeof(bool) * MAX_CPUS); + return 0; + } + return parse_cpu_list(text, cpus); +} + +static int cmd_check(int argc, char **argv) +{ + int64_t start = clock_ns(CLOCK_MONOTONIC); + bool new_cpus[MAX_CPUS] = {false}; + int new_count = 0; + enum phase phase = PHASE_RUNTIME; + struct checks checks; + + for (int index = 0; index < argc; index++) { + if (strcmp(argv[index], "--phase") == 0 && index + 1 < argc) { + const char *name = argv[++index]; + + phase = strcmp(name, "capture") == 0 ? PHASE_CAPTURE + : strcmp(name, "restore") == 0 ? PHASE_RESTORE + : PHASE_RUNTIME; + } else if (strcmp(argv[index], "--new-cpus") == 0 && index + 1 < argc) { + new_count = parse_new_cpus(argv[++index], new_cpus); + } else { + new_count = -1; + } + } + if (phase == PHASE_RUNTIME || new_count < 0) { + fprintf(stderr, "usage: nvx-time check --phase capture|restore " + "[--new-cpus LIST]\n"); + return 2; + } + load_generation(); + if (checks_init(&checks, phase) != 0) { + check_failed(&checks, "C6", "cannot read the CPU sets"); + return 1; + } + if (phase == PHASE_CAPTURE) + run_capture_checks(&checks); + else + run_restore_checks(&checks, new_cpus, new_count); + print_marker(&checks, start); + return checks.failures == 0 ? 0 : 1; +} + +static int cmd_pre_capture(void) +{ + int64_t start = clock_ns(CLOCK_MONOTONIC); + struct checks checks; + + load_generation(); + if (checks_init(&checks, PHASE_CAPTURE) != 0) { + check_failed(&checks, "C6", "cannot read the CPU sets"); + return 1; + } + run_capture_checks(&checks); + if (suppression_save() != 0) { + char detail[DETAIL_MAX]; + int error = errno; + + if (error != EEXIST) + (void)suppression_restore(detail, sizeof(detail)); + fprintf(stderr, "nvx-time: cannot suppress the stall detectors: %s\n", + error == EEXIST ? "a capture is already pending" + : strerror(error)); + return 1; + } + print_marker(&checks, start); + return 0; +} + +static int cmd_cancel_capture(void) +{ + char detail[DETAIL_MAX]; + + if (suppression_restore(detail, sizeof(detail)) != 0) { + fprintf(stderr, "nvx-time: %s\n", detail); + return 1; + } + return 0; +} + +struct repair_record { + const struct restore_packet *packet; + int64_t theta; + int64_t epsilon; + int64_t step_realtime_ns; + int64_t frequency; +}; + +static void apply_repair(struct time_state *state, const void *context) +{ + const struct repair_record *repair = context; + + state->generation = repair->packet->generation; + state->discontinuities++; + snprintf(state->last_discontinuity, sizeof(state->last_discontinuity), + "restore"); + state->last_step_ns = repair->theta; + state->last_step_realtime_ns = repair->step_realtime_ns; + state->last_downtime_ns = repair->packet->downtime_ns; + snprintf(state->last_downtime_source, sizeof(state->last_downtime_source), + "%s", + (repair->packet->flags & PACKET_DOWNTIME_UTC) != 0 ? "utc" + : "monotonic"); + state->synchronized = 1; + state->offset_ns = repair->theta; + state->uncertainty_ns = repair->epsilon; + state->frequency_ppb = frequency_to_ppb(repair->frequency); + state->samples++; + snprintf(state->last_sample_error, sizeof(state->last_sample_error), + "none"); +} + +static int repair_failed(const char *code, const char *detail) +{ + fail_fatal(STATUS_REPAIR, code, "repair", PHASE_RESTORE, "%s", detail); + return 1; +} + +// Restore steps 6 to 8: read the packet with four-byte reads, validate it, +// and set the wall clock from its bracketed UTC. The caller holds the portb +// lock. +static int repair_restore(const uint8_t *previous_id, const char *entropy_path, + int64_t start_ns) +{ + static uint8_t body[PACKET_MAX_SIZE]; + static struct restore_packet packet; + uint8_t header[PACKET_HEADER_SIZE]; + struct repair_record repair; + struct time_sample sample; + char detail[DETAIL_MAX]; + char id[GENERATION_ID_SIZE * 2 + 1]; + char text[128]; + int64_t t0; + int64_t t1; + int64_t sample_theta; + int64_t sample_epsilon; + int fd; + + t0 = clock_ns(CLOCK_REALTIME); + outb(SELECT_PACKET, PORTB_STATUS); + t1 = clock_ns(CLOCK_REALTIME); + portb_read(PORTB_DATA, header, sizeof(header)); + if (parse_packet_header(header, &packet, detail, sizeof(detail)) != 0) + return repair_failed("G_REPAIR_PACKET", detail); + if (packet.generation == g_generation) { + puts("0"); + return 0; + } + if ((uint64_t)packet.generation != (uint64_t)g_generation + 1) { + snprintf(detail, sizeof(detail), + "restore packet generation %" PRIu32 + " does not follow recorded generation %" PRIu32, + packet.generation, g_generation); + return repair_failed("G_REPAIR_GENERATION", detail); + } + portb_read(PORTB_DATA, body, packet_body_size(&packet)); + if (parse_packet_body(body, &packet, detail, sizeof(detail)) != 0) + return repair_failed("G_REPAIR_PACKET", detail); + if (memcmp(packet.entropy, previous_id, GENERATION_ID_SIZE) == 0) + return repair_failed("G_REPAIR_GENERATION", + "the VM generation ID did not change"); + g_generation = packet.generation; + pair_clocks(t0, t1, packet.utc_ns, &repair.theta, &repair.epsilon); + if (repair.epsilon > REPAIR_BOUND_NS) { + int result = best_sample(SAMPLE_ATTEMPTS - 1, REPAIR_BOUND_NS, + packet.generation, &sample, &sample_theta, + &sample_epsilon, detail, sizeof(detail)); + + if (result == -2) + return repair_failed("G_REPAIR_GENERATION", detail); + if (result >= 0 && sample_epsilon < repair.epsilon) { + repair.theta = sample_theta; + repair.epsilon = sample_epsilon; + } + if (repair.epsilon > REPAIR_BOUND_NS) { + snprintf(detail, sizeof(detail), + "UTC pairing uncertainty %" PRId64 " ns exceeds 1 ms", + repair.epsilon); + return repair_failed("G_REPAIR_SAMPLE", detail); + } + } + repair.frequency = restore_frequency(packet.rate_deviation); + if (step_clock(repair.theta) != 0 || set_frequency(repair.frequency) != 0) { + snprintf(detail, sizeof(detail), "adjtimex: %s", strerror(errno)); + return repair_failed("G_REPAIR_CLOCK", detail); + } + repair.step_realtime_ns = clock_ns(CLOCK_REALTIME); + repair.packet = &packet; + fd = open(entropy_path, O_WRONLY | O_CREAT | O_TRUNC | O_NOFOLLOW | O_CLOEXEC, + 0600); + if (fd < 0 || write_all(fd, packet.entropy, PACKET_ENTROPY_SIZE) != 0 || + close(fd) != 0) { + snprintf(detail, sizeof(detail), "write %.200s: %s", entropy_path, + strerror(errno)); + return repair_failed("G_REPAIR_PACKET", detail); + } + snprintf(text, sizeof(text), "generation=%" PRIu32 "\nelapsed_us=%" PRId64 "\n", + packet.generation, + (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC); + if (state_apply(apply_repair, &repair) != 0 || + replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text) != 0) { + snprintf(detail, sizeof(detail), "record the restore: %s", + strerror(errno)); + return repair_failed("G_REPAIR_CLOCK", detail); + } + format_hex(packet.entropy, GENERATION_ID_SIZE, id); + printf("1 %u %u %u %s", packet.flags, packet.online_vp_count, + packet.range_count, id); + for (unsigned index = 0; index < packet.range_count; index++) + printf(" %" PRIu64 " %" PRIu64, packet.ranges[index][0], + packet.ranges[index][1]); + putchar('\n'); + return 0; +} + +// Writes the capture request to 0x605. In a restored process the write +// returns with a restore packet available, and steps 6 to 8 run here before +// any other guest work. Prints 0 when no restore happened. The portb lock is +// held from the request to the restored clock, so the snapshot never +// contains a discipline poll that would apply a pre-capture sample after the +// restore. +static int cmd_capture(int argc, char **argv) +{ + uint8_t previous_id[GENERATION_ID_SIZE]; + char detail[DETAIL_MAX]; + char *end; + long request; + int64_t start; + unsigned status; + int result; + int lock; + + if (argc != 3 || (request = strtol(argv[0], &end, 10), *end != '\0') || + (request != 0 && request != 1) || + parse_hex(argv[1], previous_id, sizeof(previous_id)) != 0) { + fprintf(stderr, "usage: nvx-time capture 0|1 GENERATION_ID " + "ENTROPY_FILE\n"); + return 2; + } + if (state_load(&(struct time_state){0}) != 0) { + fprintf(stderr, "nvx-time: the time state is unavailable\n"); + return 1; + } + load_generation(); + if (enable_ports() != 0) { + fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); + return 1; + } + lock = lock_portb(detail, sizeof(detail)); + if (lock < 0) { + fprintf(stderr, "nvx-time: %s\n", detail); + return 1; + } + outb((unsigned char)request, PORT_SNAPSHOT); + start = clock_ns(CLOCK_MONOTONIC); + status = inb(PORTB_STATUS); + if ((status & PORTB_PACKET_AVAILABLE) == 0) { + close(lock); + puts("0"); + return 0; + } + result = repair_restore(previous_id, argv[2], start); + close(lock); + return result; +} + +// Restore steps 11 to 13: checks, the acknowledgement when the packet asked +// for one, then the daemon finishes the restore off the latency path. +static int cmd_restore_finish(int argc, char **argv) +{ + int64_t start = clock_ns(CLOCK_MONOTONIC); + bool new_cpus[MAX_CPUS] = {false}; + unsigned long long generation; + long long elapsed; + struct checks checks; + char text[128]; + int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; + bool ack = false; + int new_count = 0; + pid_t pid; + + for (int index = 0; index < argc; index++) { + if (strcmp(argv[index], "--ack") == 0) + ack = true; + else if (strcmp(argv[index], "--new-cpus") == 0 && index + 1 < argc) + new_count = parse_new_cpus(argv[++index], new_cpus); + else + new_count = -1; + } + if (new_count < 0) { + fprintf(stderr, "usage: nvx-time restore-finish [--ack] " + "[--new-cpus LIST]\n"); + return 2; + } + load_generation(); + if (read_text(RESTORE_PATH, text, sizeof(text)) < 0 || + sscanf(text, "generation=%llu elapsed_us=%lld", &generation, + &elapsed) != 2) + return repair_failed("G_REPAIR_PACKET", "no restore record"); + if (checks_init(&checks, PHASE_RESTORE) != 0) + check_failed(&checks, "C6", "cannot read the CPU sets"); + else + run_restore_checks(&checks, new_cpus, new_count); + elapsed += (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC; + snprintf(text, sizeof(text), "generation=%llu\nelapsed_us=%lld\n", + generation, elapsed); + (void)replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text); + if (ack) { + if (enable_ports() != 0) { + fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); + return 1; + } + outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); + } + pid = daemon_pid(); + if (pid > 0 && kill(pid, SIGUSR1) == 0) + return 0; + if (!g_report_only) { + check_failed(&checks, "C10", "the time daemon is not running"); + return 1; + } + // Report-only runs may lack the daemon; finish the restore here. + (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); + complete_restore(release_with_deadline(timeout_s), checks.tsc_hz, + checks.lapic_hz, elapsed); + return 0; +} + +static int cmd_sample(void) +{ + struct time_sample sample; + char detail[DETAIL_MAX] = ""; + int64_t theta; + int64_t epsilon; + unsigned status; + int result; + int lock; + + if (enable_ports() != 0) { + fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); + return 1; + } + status = inb(PORTB_STATUS); + if ((status & PORTB_TIME_WINDOW) == 0) { + fprintf(stderr, "nvx-time: portb status 0x%02x has no time window\n", + status); + return 1; + } + lock = lock_portb(detail, sizeof(detail)); + result = lock < 0 ? -1 : take_sample(&sample, detail, sizeof(detail)); + if (lock >= 0) + close(lock); + if (result != 0) { + fprintf(stderr, "nvx-time: %s\n", detail); + return 1; + } + pair_clocks(sample.t0, sample.t1, sample.utc_ns, &theta, &epsilon); + printf("flags=0x%02x generation=%" PRIu32 " utc_ns=%" PRIu64 + " t0=%" PRId64 " t1=%" PRId64 " theta_ns=%" PRId64 + " epsilon_ns=%" PRId64 "\n", + sample.flags, sample.generation, sample.utc_ns, sample.t0, + sample.t1, theta, epsilon); + return 0; +} + +static int cmd_generation_id(void) +{ + uint8_t id[GENERATION_ID_SIZE]; + char text[GENERATION_ID_SIZE * 2 + 1]; + unsigned status; + int lock; + + if (enable_ports() != 0) { + fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); + return 1; + } + status = inb(PORTB_STATUS); + if ((status & PORTB_GENERATION_ID_AVAILABLE) == 0) { + fprintf(stderr, "nvx-time: the VM generation ID is unavailable\n"); + return 1; + } + lock = lock_file(PORTB_LOCK_PATH); + if (lock < 0) { + fprintf(stderr, "nvx-time: portb lock: %s\n", strerror(errno)); + return 1; + } + outb(SELECT_GENERATION_ID, PORTB_STATUS); + portb_read(PORTB_DATA, id, sizeof(id)); + close(lock); + format_hex(id, sizeof(id), text); + puts(text); + return 0; +} + +// --------------------------------------------------------------------------- +// Unit-test entry points: pure decoders and decisions over fixture files +// --------------------------------------------------------------------------- + +static int test_bytes(const char *path, uint8_t *buffer, size_t size) +{ + memset(buffer, 0, size); + return read_bytes(path, buffer, size) < 0 ? -1 : 0; +} + +static int test_text(const char *path, char *buffer, size_t size) +{ + return read_text(path, buffer, size) < 0 ? -1 : 0; +} + +static int test_packet(int argc, char **argv) +{ + static uint8_t bytes[PACKET_MAX_SIZE]; + static struct restore_packet packet; + uint8_t previous[GENERATION_ID_SIZE]; + char detail[DETAIL_MAX]; + char id[GENERATION_ID_SIZE * 2 + 1]; + uint64_t recorded; + + if (argc != 3 || test_bytes(argv[0], bytes, sizeof(bytes)) != 0 || + parse_hex(argv[2], previous, sizeof(previous)) != 0) + return 2; + recorded = strtoull(argv[1], NULL, 10); + if (parse_packet_header(bytes, &packet, detail, sizeof(detail)) != 0) { + printf("error G_REPAIR_PACKET %s\n", detail); + return 0; + } + if (packet.generation == recorded) { + printf("stale\n"); + return 0; + } + if ((uint64_t)packet.generation != recorded + 1) { + printf("error G_REPAIR_GENERATION generation %" PRIu32 "\n", + packet.generation); + return 0; + } + if (parse_packet_body(bytes + PACKET_HEADER_SIZE, &packet, detail, + sizeof(detail)) != 0) { + printf("error G_REPAIR_PACKET %s\n", detail); + return 0; + } + if (memcmp(packet.entropy, previous, GENERATION_ID_SIZE) == 0) { + printf("error G_REPAIR_GENERATION generation ID unchanged\n"); + return 0; + } + format_hex(packet.entropy, GENERATION_ID_SIZE, id); + printf("ok flags=%u online=%u ranges=%u generation=%" PRIu32 + " rate_deviation=%" PRId32 " frequency=%" PRId64 + " downtime_ns=%" PRIu64 " utc_ns=%" PRIu64 " generation_id=%s", + packet.flags, packet.online_vp_count, packet.range_count, + packet.generation, packet.rate_deviation, + restore_frequency(packet.rate_deviation), packet.downtime_ns, + packet.utc_ns, id); + for (unsigned index = 0; index < packet.range_count; index++) + printf(" range=%" PRIu64 ":%" PRIu64, packet.ranges[index][0], + packet.ranges[index][1]); + putchar('\n'); + return 0; +} + +static int test_sample(int argc, char **argv) +{ + uint8_t bytes[SAMPLE_SIZE]; + struct time_sample sample; + char detail[DETAIL_MAX]; + + if (argc != 2 || test_bytes(argv[0], bytes, sizeof(bytes)) != 0) + return 2; + if (parse_sample(bytes, &sample, detail, sizeof(detail)) != 0) + printf("error %s\n", detail); + else if (sample.generation != strtoul(argv[1], NULL, 10)) + printf("error generation %" PRIu32 "\n", sample.generation); + else + printf("ok flags=%u generation=%" PRIu32 " utc_ns=%" PRIu64 "\n", + sample.flags, sample.generation, sample.utc_ns); + return 0; +} + +static int test_lines(int argc, char **argv, bool boot) +{ + static char text[TEXT_MAX]; + struct boot_log log; + char detail[DETAIL_MAX]; + + if (argc != 1 || test_text(argv[0], text, sizeof(text)) != 0) + return 2; + memset(&log, 0, sizeof(log)); + for (char *line = strtok(text, "\n"); line != NULL; + line = strtok(NULL, "\n")) { + const char *code = match_watch_rule(line); + + if (boot) + boot_log_add(&log, line); + else + printf("%s\n", code != NULL ? code : "none"); + } + if (boot) { + if (boot_log_verdict(&log, detail, sizeof(detail))) + printf("ok\n"); + else + printf("fail %s\n", detail); + } + return 0; +} + +static int test_timer_list(int argc, char **argv) +{ + static char text[TEXT_MAX]; + bool cpus[MAX_CPUS]; + char detail[DETAIL_MAX]; + + if (argc != 2 || test_text(argv[0], text, sizeof(text)) != 0 || + parse_cpu_list(argv[1], cpus) <= 0) + return 2; + if (timer_list_verdict(text, cpus, detail, sizeof(detail))) + printf("ok\n"); + else + printf("fail %s\n", detail); + return 0; +} + +static int test_checks(const char *name, int argc, char **argv) +{ + static char text[TEXT_MAX]; + struct checks checks; + bool cpus[MAX_CPUS]; + + memset(&checks, 0, sizeof(checks)); + checks.phase = PHASE_BOOT; + if (strcmp(name, "cpuinfo") == 0) { + if (argc != 2 || test_text(argv[0], text, sizeof(text)) != 0 || + parse_cpu_list(argv[1], cpus) <= 0) + return 2; + if (check_cpuinfo_text(&checks, text, cpus, true)) { + printf("ok"); + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + if (checks.cpu_khz_known[cpu]) + printf(" cpu%d_khz=%" PRIu64, cpu, checks.cpu_khz[cpu]); + } + putchar('\n'); + } + } else if (strcmp(name, "cmdline") == 0) { + if (argc != 1) + return 2; + snprintf(text, sizeof(text), "%s", argv[0]); + if (check_cmdline_text(&checks, text)) + printf("ok\n"); + } else if (strcmp(name, "clocksource") == 0) { + if (argc != 2) + return 2; + snprintf(text, sizeof(text), "%s", argv[1]); + if (check_clocksource_values(&checks, argv[0], text)) + printf("ok\n"); + } else { + return 2; + } + return 0; +} + +static int test_correction(int argc, char **argv) +{ + struct timex tx; + bool step; + + if (argc != 3) + return 2; + step = plan_correction(strtoll(argv[0], NULL, 10), strtoll(argv[1], NULL, 10), + strcmp(argv[2], "fast") == 0, &tx); + if (step) + printf("step modes=0x%x sec=%ld nsec=%ld\n", tx.modes, + (long)tx.time.tv_sec, (long)tx.time.tv_usec); + else + printf("slew modes=0x%x offset=%ld status=0x%x constant=%ld " + "maxerror=%ld esterror=%ld\n", + tx.modes, tx.offset, (unsigned)tx.status, tx.constant, + tx.maxerror, tx.esterror); + return 0; +} + +static int test_state(int argc, char **argv) +{ + static char text[4096]; + struct time_state state; + + if (argc != 1 || test_text(argv[0], text, sizeof(text)) != 0) + return 2; + if (state_parse(text, &state) != 0) { + printf("error\n"); + return 0; + } + state_format(&state, text, sizeof(text)); + fputs(text, stdout); + return 0; +} + +static int cmd_test(int argc, char **argv) +{ + const char *name = argc > 0 ? argv[0] : ""; + + g_test = true; + g_report_only = true; + argc--; + argv++; + if (strcmp(name, "packet") == 0) + return test_packet(argc, argv); + if (strcmp(name, "sample") == 0) + return test_sample(argc, argv); + if (strcmp(name, "pair") == 0 && argc == 3) { + int64_t theta; + int64_t epsilon; + + pair_clocks(strtoll(argv[0], NULL, 10), strtoll(argv[1], NULL, 10), + strtoull(argv[2], NULL, 10), &theta, &epsilon); + printf("theta_ns=%" PRId64 " epsilon_ns=%" PRId64 "\n", theta, epsilon); + return 0; + } + if (strcmp(name, "kmsg") == 0) + return test_lines(argc, argv, false); + if (strcmp(name, "boot-log") == 0) + return test_lines(argc, argv, true); + if (strcmp(name, "timer-list") == 0) + return test_timer_list(argc, argv); + if (strcmp(name, "correction") == 0) + return test_correction(argc, argv); + if (strcmp(name, "frequency") == 0 && argc == 1) { + int64_t frequency = restore_frequency((int32_t)strtol(argv[0], NULL, 10)); + + printf("frequency=%" PRId64 " ppb=%" PRId64 "\n", frequency, + frequency_to_ppb(frequency)); + return 0; + } + if (strcmp(name, "event") == 0 && argc == 6) { + char line[EVENT_MAX + 1]; + enum phase phase = PHASE_RUNTIME; + + for (size_t index = 0; index < ARRAY_SIZE(k_phase_names); index++) { + if (strcmp(argv[2], k_phase_names[index]) == 0) + phase = (enum phase)index; + } + g_report_only = false; + format_event(line, argv[0], argv[1], phase, + (uint32_t)strtoul(argv[3], NULL, 10), + strtoll(argv[4], NULL, 10), argv[5]); + fputs(line, stdout); + return 0; + } + if (strcmp(name, "state") == 0) + return test_state(argc, argv); + return test_checks(name, argc, argv); +} + +// --------------------------------------------------------------------------- +// Entry point +// --------------------------------------------------------------------------- + +static bool cmdline_requests_report_only(void) +{ + char text[4096]; + + if (read_text("/proc/cmdline", text, sizeof(text)) < 0) + return false; + for (char *token = strtok(text, " \t\n"); token != NULL; + token = strtok(NULL, " \t\n")) { + if (strcmp(token, "nvx_time_abi=report-only") == 0) + return true; + } + return false; +} + +static int usage(void) +{ + fprintf(stderr, + "usage: nvx-time [--report-only] COMMAND\n" + " boot\n" + " check --phase capture|restore [--new-cpus LIST]\n" + " pre-capture\n" + " capture 0|1 GENERATION_ID ENTROPY_FILE\n" + " cancel-capture\n" + " restore-finish [--ack] [--new-cpus LIST]\n" + " sample\n" + " generation-id\n" + " test NAME ARGUMENTS...\n"); + return 2; +} + +int main(int argc, char **argv) +{ + char *arguments[32]; + int count = 0; + const char *command; + + for (int index = 1; index < argc; index++) { + if (strcmp(argv[index], "--report-only") == 0) + g_report_only = true; + else if (count < (int)ARRAY_SIZE(arguments)) + arguments[count++] = argv[index]; + else + return usage(); + } + if (count == 0) + return usage(); + command = arguments[0]; + if (strcmp(command, "test") == 0) + return cmd_test(count - 1, arguments + 1); + // An inherited SIG_IGN would make the kernel reap the release child and + // the daemon before waitpid() sees them. + signal(SIGCHLD, SIG_DFL); + if (cmdline_requests_report_only()) + g_report_only = true; + if (strcmp(command, "boot") == 0 && count == 1) + return cmd_boot(); + if (strcmp(command, "check") == 0) + return cmd_check(count - 1, arguments + 1); + if (strcmp(command, "pre-capture") == 0 && count == 1) + return cmd_pre_capture(); + if (strcmp(command, "capture") == 0) + return cmd_capture(count - 1, arguments + 1); + if (strcmp(command, "cancel-capture") == 0 && count == 1) + return cmd_cancel_capture(); + if (strcmp(command, "restore-finish") == 0) + return cmd_restore_finish(count - 1, arguments + 1); + if (strcmp(command, "sample") == 0 && count == 1) + return cmd_sample(); + if (strcmp(command, "generation-id") == 0 && count == 1) + return cmd_generation_id(); + return usage(); +} diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 4fca1017e..708468312 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -310,6 +310,7 @@ class InitramfsBuildConstants: "nvx-port-io", "nvx-console-pending", "nvx-managed-agent", + "nvx-time", ) TIME_PROBE_NAME: Final = "nvx-time-probe" TIME_PROBE_CFLAGS: Final = ( diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py new file mode 100644 index 000000000..ba30af682 --- /dev/null +++ b/scripts/test_guest_time.py @@ -0,0 +1,543 @@ +#!/usr/bin/env python3 +"""Spec-derived tests for the guest time component, guest/common/nvx-time.c. + +The fixtures are encoded here from doc/design/time-abi.md, independently of +the C decoders, so the tests check the guest against the specification. +""" + +from __future__ import annotations + +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +SOURCE = REPO_ROOT / "guest" / "common" / "nvx-time.c" + +DOWNTIME_UTC = 0x01 +MEMORY_TARGET = 0x02 +ACK_REQUIRED = 0x04 +TEST_HOOKS = 0x08 +UTC_NS = 1_790_841_600_123_456_789 +ENTROPY = bytes(range(64)) +PREVIOUS_ID = "ff" * 16 +THIRTY_DAYS_NS = 2_592_000 * 1_000_000_000 + + +def encode_packet( + *, + flags: int = 0, + online: int = 0, + ranges: tuple[tuple[int, int], ...] = (), + range_count: int | None = None, + generation: int = 1, + rate_deviation: int = 0, + downtime_ns: int = 0, + utc_ns: int = UTC_NS, + magic: bytes = b"OVR", + version: int = 4, + reserved: int = 0, + entropy: bytes = ENTROPY, +) -> bytes: + """Encode a restore packet v4 (spec: "Restore packet").""" + count = len(ranges) if range_count is None else range_count + header = magic + bytes((version, flags, online, count, reserved)) + header += struct.pack(" bytes: + """Encode a time sample (spec: "Time sample").""" + return struct.pack(" None: + compiler = shutil.which("cc") or shutil.which("gcc") + if compiler is None: + raise unittest.SkipTest("a C compiler is unavailable") + cls.directory = tempfile.TemporaryDirectory() + cls.binary = Path(cls.directory.name) / "nvx-time" + subprocess.run( + [ + compiler, + "-O1", + "-Wall", + "-Wextra", + "-Werror", + "-o", + str(cls.binary), + str(SOURCE), + ], + check=True, + capture_output=True, + timeout=120, + ) + + @classmethod + def tearDownClass(cls) -> None: + cls.directory.cleanup() + + def run_test(self, *arguments: str) -> str: + result = subprocess.run( + [str(self.binary), "test", *arguments], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + return result.stdout + + def fixture(self, name: str, content: bytes | str) -> str: + path = Path(self.directory.name) / name + if isinstance(content, str): + path.write_text(content, encoding="utf-8") + else: + path.write_bytes(content) + return str(path) + + def packet(self, packet: bytes, recorded: int = 0, previous: str = PREVIOUS_ID): + return self.run_test( + "packet", self.fixture("packet.bin", packet), str(recorded), previous + ).strip() + + def test_packet_v4_golden_vector(self): + packet = encode_packet( + flags=DOWNTIME_UTC | ACK_REQUIRED | TEST_HOOKS, + online=4, + generation=3, + rate_deviation=-13_107_200, + downtime_ns=30_000_000_000, + ) + + self.assertEqual(len(packet), 96) + self.assertEqual( + self.packet(packet, recorded=2), + "ok flags=13 online=4 ranges=0 generation=3 " + "rate_deviation=-13107200 frequency=13107200 " + "downtime_ns=30000000000 " + f"utc_ns={UTC_NS} generation_id={ENTROPY[:16].hex()}", + ) + + def test_packet_memory_ranges_follow_the_header(self): + packet = encode_packet( + flags=MEMORY_TARGET, + ranges=((0x8000_0000, 0x0800_0000), (0x1_0000_0000, 0x1000_0000)), + ) + + self.assertEqual(len(packet), 32 + 2 * 16 + 64) + self.assertTrue( + self.packet(packet).endswith( + " range=2147483648:134217728 range=4294967296:268435456" + ) + ) + self.assertIn(" ranges=0 ", self.packet(encode_packet(flags=MEMORY_TARGET))) + + def test_packet_rejections_name_the_repair_code(self): + cases = { + "magic": encode_packet(magic=b"OVX"), + "version 3": encode_packet(version=3), + "reserved flag": encode_packet(flags=0x10), + "reserved byte": encode_packet(reserved=1), + "online count": encode_packet(online=3), + "ranges without target": encode_packet(range_count=1), + "generation zero": encode_packet(generation=0), + "rate above": encode_packet(rate_deviation=16_384_001), + "rate below": encode_packet(rate_deviation=-16_384_001), + "downtime": encode_packet(downtime_ns=THIRTY_DAYS_NS + 1), + "utc zero": encode_packet(utc_ns=0), + "empty range": encode_packet(flags=MEMORY_TARGET, ranges=((0, 0),)), + "range overflow": encode_packet( + flags=MEMORY_TARGET, ranges=((2**64 - 4096, 8192),) + ), + } + for name, packet in cases.items(): + with self.subTest(case=name): + self.assertTrue( + self.packet(packet).startswith("error G_REPAIR_PACKET "), + name, + ) + + def test_packet_bounds_are_inclusive(self): + for packet in ( + encode_packet(rate_deviation=16_384_000), + encode_packet(rate_deviation=-16_384_000), + encode_packet(downtime_ns=THIRTY_DAYS_NS), + encode_packet(online=8), + ): + with self.subTest(packet=packet[:32].hex()): + self.assertTrue(self.packet(packet).startswith("ok ")) + + def test_packet_generation_must_follow_the_recorded_generation(self): + self.assertEqual(self.packet(encode_packet(generation=5), recorded=5), "stale") + self.assertTrue( + self.packet(encode_packet(generation=7), recorded=5).startswith( + "error G_REPAIR_GENERATION " + ) + ) + self.assertTrue( + self.packet( + encode_packet(generation=6), recorded=5, previous=ENTROPY[:16].hex() + ).startswith("error G_REPAIR_GENERATION ") + ) + + def sample(self, sample: bytes, generation: int = 0) -> str: + return self.run_test( + "sample", self.fixture("sample.bin", sample), str(generation) + ).strip() + + def test_time_sample_layout(self): + self.assertEqual(len(encode_sample()), 16) + self.assertEqual( + self.sample(encode_sample(generation=4, flags=TEST_HOOKS), 4), + f"ok flags=8 generation=4 utc_ns={UTC_NS}", + ) + for name, sample in { + "version": encode_sample(version=2), + "flags": encode_sample(flags=0x01), + "reserved": encode_sample(reserved=1), + "utc": encode_sample(utc_ns=0), + "generation": encode_sample(generation=3), + }.items(): + with self.subTest(case=name): + self.assertTrue(self.sample(sample).startswith("error "), name) + + def test_utc_pairing_uses_the_bracket_midpoint(self): + self.assertEqual( + self.run_test("pair", "1000", "3000", "5000").strip(), + "theta_ns=3000 epsilon_ns=1000", + ) + self.assertEqual( + self.run_test("pair", "10000", "10400", "9000").strip(), + "theta_ns=-1200 epsilon_ns=200", + ) + + def test_restore_frequency_compensates_and_clamps(self): + self.assertEqual( + self.run_test("frequency", "13107200").strip(), + "frequency=-13107200 ppb=-200000", + ) + self.assertEqual( + self.run_test("frequency", "-40000000").strip(), + "frequency=32768000 ppb=500000", + ) + self.assertEqual( + self.run_test("frequency", "-65536").strip(), + "frequency=65536 ppb=1000", + ) + + def test_discipline_steps_at_128_ms_and_slews_below(self): + self.assertEqual( + self.run_test("correction", "127999999", "10002", "fast").strip(), + "slew modes=0x203d offset=127999999 status=0x2001 constant=4 " + "maxerror=128011 esterror=11", + ) + self.assertEqual( + self.run_test("correction", "-500", "20", "slow").strip(), + "slew modes=0x203d offset=-500 status=0x2001 constant=6 " + "maxerror=1 esterror=1", + ) + self.assertEqual( + self.run_test("correction", "128000000", "10", "fast").strip(), + "step modes=0x2100 sec=0 nsec=128000000", + ) + self.assertEqual( + self.run_test("correction", "-1128000000", "10", "slow").strip(), + "step modes=0x2100 sec=-2 nsec=872000000", + ) + + def test_watcher_rules_match_linux_6_18_messages(self): + messages = { + "tsc: Marking TSC unstable due to clocksource watchdog": ("G_TSC_UNSTABLE"), + "clocksource: timekeeping watchdog on CPU0: Marking clocksource " + "'tsc-early' as unstable because the skew is too large:": ( + "G_CLOCKSOURCE_UNSTABLE" + ), + "clocksource: Switched to clocksource refined-jiffies": ( + "G_CLOCKSOURCE_SWITCH" + ), + "clocksource: CPUs 1 ahead of CPU 0 for clocksource tsc.": ( + "G_CLOCKSOURCE_SKEW" + ), + "clocksource: CPUs 2-3 behind CPU 0 for clocksource tsc.": ( + "G_CLOCKSOURCE_SKEW" + ), + "TSC synchronization [CPU#0 -> CPU#1]:": "G_TSC_WARP", + "Measured 45 cycles TSC warp between CPUs, turning off TSC clock.": ( + "G_TSC_WARP" + ), + "TSC warped randomly between CPUs": "G_TSC_WARP", + "[Firmware Bug]: TSC ADJUST differs: CPU1 0 --> 100. Restoring": ( + "G_TSC_ADJUST" + ), + "rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:": "G_RCU_STALL", + "rcu: INFO: rcu_preempt self-detected stall on CPU": "G_RCU_STALL", + "rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: " + "{ 1-... }": "G_RCU_STALL", + "rcu: rcu_preempt kthread starved for 2603 jiffies! g345 f0x0": ( + "G_RCU_STARVED" + ), + "rcu: rcu_preempt kthread timer wakeup didn't happen for 2602 " + "jiffies! g345 f0x0": "G_RCU_STARVED", + "watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [sh:123]": ( + "G_SOFT_LOCKUP" + ), + "watchdog: Watchdog detected hard LOCKUP on cpu 1": "G_HARD_LOCKUP", + "INFO: task kworker/0:1:12 blocked for more than 120 seconds.": ( + "G_HUNG_TASK" + ), + "unchecked MSR access error: RDMSR from 0x1ad at rIP: " + "0xffffffff816f11ee (__rdmsr_on_cpu+0x1e/0x30)": "G_UNCHECKED_MSR", + "clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fa3": "none", + "Hyper-V: LAPIC Timer Frequency: 0x1e8480": "none", + "rcu: Hierarchical RCU implementation.": "none", + 'NVX-TIME-ABI-VIOLATION: v=1 code=G_UNCHECKED_MSR detail="unchecked ' + 'MSR access error"': "none", + } + output = self.run_test( + "kmsg", self.fixture("kmsg.txt", "\n".join(messages) + "\n") + ) + self.assertEqual(output.splitlines(), list(messages.values())) + + def boot_log(self, lines: list[str]) -> str: + return self.run_test( + "boot-log", self.fixture("boot.txt", "\n".join(lines) + "\n") + ).strip() + + BOOT_LOG = [ + "Hypervisor detected: Microsoft Hyper-V", + "Hyper-V: privilege flags low 0x8860, high 0x0, ext 0x0, hints 0x0, misc 0x100", + "Hyper-V: LAPIC Timer Frequency: 0x1e8480", + "tsc: Detected 2194.843 MHz processor", + "clocksource: Switched to clocksource tsc-early", + "clocksource: Switched to clocksource tsc", + ] + + def test_boot_log_requires_the_identity_and_a_final_switch_to_tsc(self): + self.assertEqual(self.boot_log(self.BOOT_LOG), "ok") + kvm_rate = [line.replace("0x1e8480", "0x989680") for line in self.BOOT_LOG] + self.assertEqual(self.boot_log(kvm_rate), "ok") + failures = { + "missing identity": self.BOOT_LOG[1:], + "wrong LAPIC rate": [ + line.replace("0x1e8480", "0x1e8481") for line in self.BOOT_LOG + ], + "last switch": [ + *self.BOOT_LOG, + "clocksource: Switched to clocksource refined-jiffies", + ], + "kvm-clock": [*self.BOOT_LOG, "kvm-clock: Using msrs 4b564d01"], + "refinement": [ + *self.BOOT_LOG, + "tsc: Refined TSC clocksource calibration: 2194.843 MHz", + ], + "lapic token": [ + *self.BOOT_LOG, + "APIC timer: using supplied frequency 200000000 Hz", + ], + "watcher record": [ + *self.BOOT_LOG, + "tsc: Marking TSC unstable due to clocksource watchdog", + ], + } + for name, lines in failures.items(): + with self.subTest(case=name): + self.assertTrue(self.boot_log(lines).startswith("fail "), name) + + def timer_list( + self, + sections: list[tuple[int, str, int, str, int]], + broadcast: str = "", + cpus: str = "0-1", + ) -> str: + text = TIMER_LIST_BROADCAST.format(device=broadcast) + for cpu, device, state, handler, mode in sections: + text += TIMER_LIST_CPU.format( + cpu=cpu, device=device, state=state, handler=handler + ).replace("Tick Device: mode: 1", f"Tick Device: mode: {mode}") + return self.run_test( + "timer-list", self.fixture("timer_list.txt", text), cpus + ).strip() + + def test_timer_list_requires_one_shot_lapic_ticks(self): + good = [ + (0, "lapic", 3, "hrtimer_interrupt", 1), + (1, "lapic", 4, "hrtimer_interrupt", 1), + ] + self.assertEqual(self.timer_list(good), "ok") + failures = { + "deadline": [good[0], (1, "lapic-deadline", 3, "hrtimer_interrupt", 1)], + "periodic tick": [good[0], (1, "lapic", 3, "hrtimer_interrupt", 0)], + "periodic state": [good[0], (1, "lapic", 2, "hrtimer_interrupt", 1)], + "handler": [good[0], (1, "lapic", 3, "tick_handle_periodic", 1)], + "missing cpu": [good[0]], + } + for name, sections in failures.items(): + with self.subTest(case=name): + self.assertTrue(self.timer_list(sections).startswith("fail ")) + self.assertTrue(self.timer_list(good, broadcast="pit").startswith("fail ")) + + def test_cpuinfo_flags_and_rate(self): + def cpuinfo(flags: str, cpus: int = 2) -> str: + text = "".join( + CPUINFO_CPU.format(cpu=cpu, flags=flags) for cpu in range(cpus) + ) + return self.run_test( + "cpuinfo", self.fixture("cpuinfo.txt", text), "0-1" + ).strip() + + self.assertEqual(cpuinfo(GOOD_FLAGS), "ok cpu0_khz=2194843 cpu1_khz=2194843") + for name, flags in { + "tsc_reliable": GOOD_FLAGS.replace("tsc_reliable ", ""), + "deadline": GOOD_FLAGS + " tsc_deadline_timer", + "adjust": GOOD_FLAGS + " tsc_adjust", + }.items(): + with self.subTest(case=name): + self.assertIn("status=fail check=C5", cpuinfo(flags)) + self.assertIn("status=fail check=C5", cpuinfo(GOOD_FLAGS, cpus=1)) + + def test_command_line_rejects_clock_workarounds(self): + accepted = "earlycon=xe9 console=hvc0 quiet clocksource=tsc tsc=reliable" + self.assertEqual(self.run_test("cmdline", accepted).strip(), "ok") + for token in ( + "tsc_early_khz=2194843", + "lapic_timer_hz=200000000", + "notsc", + "nolapic", + "nolapic_timer", + "tsc=unstable", + "hpet=force", + "clocksource=kvm-clock", + ): + with self.subTest(token=token): + self.assertIn( + "status=fail check=C9", + self.run_test("cmdline", f"{accepted} {token}"), + ) + + def test_clocksource_must_be_tsc_with_only_jiffies_fallbacks(self): + self.assertEqual(self.run_test("clocksource", "tsc\n", "tsc \n").strip(), "ok") + self.assertEqual( + self.run_test( + "clocksource", "tsc\n", "tsc refined-jiffies jiffies\n" + ).strip(), + "ok", + ) + for current, available in ( + ("tsc-early\n", "tsc-early \n"), + ("tsc\n", "tsc kvm-clock \n"), + ("refined-jiffies\n", "refined-jiffies \n"), + ): + with self.subTest(current=current, available=available): + self.assertIn( + "status=fail check=C6", + self.run_test("clocksource", current, available), + ) + + def test_violation_event_format_escaping_and_limit(self): + line = self.run_test( + "event", + "G_UNCHECKED_MSR", + "watcher", + "runtime", + "3", + "123456789", + 'unchecked "MSR"\\ access\x01', + ) + self.assertEqual( + line, + "NVX-TIME-ABI-VIOLATION: v=1 code=G_UNCHECKED_MSR source=watcher " + "phase=runtime generation=3 boottime_ns=123456789 " + 'detail="unchecked \\"MSR\\"\\\\ access\\x01"\n', + ) + long_line = self.run_test( + "event", "G_HUNG_TASK", "watcher", "runtime", "0", "1", "\x02" * 400 + ) + self.assertLessEqual(len(long_line), 512) + self.assertTrue(long_line.endswith('\\x02"\n')) + + def test_state_file_round_trip(self): + text = ( + "version=1\ngeneration=2\ndiscontinuities=3\nlast_discontinuity=step\n" + "last_step_ns=-200000000\nlast_step_realtime_ns=1790841600000000000\n" + "last_downtime_ns=31000000000\nlast_downtime_source=utc\n" + "synchronized=1\noffset_ns=-42\nuncertainty_ns=1800\n" + "frequency_ppb=-1200\nsamples=17\nrejected_samples=1\n" + "last_sample_error=none\nviolations=0\n" + ) + self.assertEqual(self.run_test("state", self.fixture("state.txt", text)), text) + self.assertEqual( + self.run_test( + "state", self.fixture("state.txt", "unknown=x\nsamples=bad\n") + ).strip(), + "error", + ) + + +if __name__ == "__main__": + unittest.main() From f7c15859fd1b3952f8a7ef4aecd5056b4fa7d9cb Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:11:39 -0700 Subject: [PATCH 011/188] doc: settle the time ABI CPU profiles and answer guest questions Adopt the profiles agent's design, backed by fingerprints of all 18 hosts: per-(generation, backend) profiles in canonical JSON with dense CPUID tables, VMM-owned bit masks, XSAVE layouts, pinned MSRs, and MSHV/WHP feature banks; IDs ...v with the report generation names; stepping ranges; auto selection; and verification classes mapped onto the existing E_* codes. Invariant TSC and ARAT are profile policy, exempt from host-support checks. Also answer the guest agent's questions: the restore packet is consumed when read, the portb lock scope, epsilon_ns=none, the report-only test mode, CPUID access, C1/C3/C10 details, and the power-off path. State that the restore anchor is logged but not bounded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 184 +++++++++++++++++++++++++++-------------- 1 file changed, 120 insertions(+), 64 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6cdc77393..0dc7a917f 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -249,54 +249,87 @@ timestamp read. ## CPU profiles -A CPU profile is the complete guest-visible CPU surface for one CPU -generation. Profiles are pinned in the OpenVMM source tree, derived by -intersecting host fingerprints (the Firecracker `cpu-template-helper` -workflow), and immutable once released: any change creates a new revision ID. - -**Format.** A profile has these fields, in a canonical, deterministic -encoding defined by OpenVMM's `cpu_profile` crate: +A CPU profile is the complete guest-visible CPU surface for one vendor, CPU +generation, and backend. The guest sees only profile values, never host +passthrough, except for a fixed, code-defined set of VMM-owned fields and the +fields the time ABI owns. Profiles are data: derived mechanically by +intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker +`cpu-template-helper` workflow), reviewed, checked in at +`vmm_core/cpu_profile/profiles/.json`, embedded in the OpenVMM binary, +and immutable once released. A normative change creates a new revision; +released revisions are never deleted, so old snapshots stay restorable. +Profiles are per backend because MSHV and WHP gate features through +processor feature banks that a CPUID table cannot express, and cross-backend +restore is rejected anyway. + +**Format.** Schema `openvmm-cpu-profile/v1`, a canonical JSON document (keys +sorted by UTF-8 bytes, no whitespace, numbers as hexadecimal strings): | Field | Content | | --- | --- | -| `id` | `--v`, lowercase | +| `id` | `...v`, each component `[a-z0-9-]+`, for example `intel.icelake-sp.kvm.v1` | | `vendor` | The 12-byte CPUID vendor string | -| `generation` | Allowed `(family, model)` display signatures; stepping is unconstrained | -| `cpuid` | Every basic and extended leaf and subleaf the guest can observe, with exact register values and a mask of topology fields that OpenVMM synthesizes from the machine topology | -| `xcr0`, `xss` | Exposed XSAVE feature masks | -| `xsave_components` | Exact offset, size, and alignment of every enabled component | -| `physical_address_width` | Guest physical address width | -| `msrs` | Pinned MSR values the guest can read, such as `IA32_ARCH_CAPABILITIES` | - -The profile digest is the SHA-256 of the canonical encoding. Profile content -outside the topology mask is host-independent, so the effective guest CPUID -is a pure function of the profile, the topology, and the time ABI. - -**Selection.** A cold boot uses `--cpu-profile `, or `auto`, which maps -the host's `(vendor, family, model)` to exactly one pinned profile. An -unmapped host is rejected (`E_PROFILE_HOST_UNKNOWN`); host CPUID passthrough -does not exist. A restore always uses the profile embedded in the snapshot; -an explicit `--cpu-profile` must name the same profile. - -**Verification.** At partition creation, for cold boot and restore: - -1. The profile satisfies the [CPU time bits](#cpu-time-bits) - (`E_PROFILE_TIME_BITS`). -2. The host's `(vendor, family, model)` is in the profile's generation - (`E_CPU_GENERATION`). -3. The backend supports every feature bit the profile sets, every numeric - limit is within the backend's, the XSAVE layout matches exactly, and every - pinned MSR value can be presented (`E_PROFILE_UNSUPPORTED`, naming the - first missing leaf, register, and bit). -4. On restore only: this OpenVMM pins a profile with the same ID and digest - (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective - CPUID equals the recorded one (`E_CPU_SURFACE`). - -Initial generations are `skylake-sp` (family 6, model 85: the bare-metal -hosts), `icelake-sp` (6/106: the Xeon Platinum 8370C runners), and -`emeraldrapids` (6/207: the 8573C runners). Profile IDs, contents, whether one -profile serves all backends of a generation, and the fate of the -`MYSTERY_MSRS` stubs are TBD(profiles). +| `cpus` | Allowed `(family, model, stepping range)` display signatures; stepping ranges separate model 85's Skylake-SP (0 to 4), Cascade Lake, and Cooper Lake | +| `backend` | `kvm`, `mshv`, or `whp` | +| `cpuid` | A dense table of every leaf and subleaf in `[0, max basic]` and `[0x80000000, max extended]` with a value and a mask per register; mask bit 1 pins the value, mask bit 0 marks a VMM-owned bit | +| `xsave` | `xcr0` and `xss` masks the guest may enable, and the size, offset, and flags of every enabled component | +| `msrs` | Pinned MSR values with masks, such as `IA32_ARCH_CAPABILITIES` and `IA32_PERF_CAPABILITIES` (0, no vPMU) | +| `feature_banks` | MSHV and WHP only: processor feature banks 0 and 1 and the XSAVE feature bank | +| `provenance` | Source fingerprint surface digests (informative) | +| `digest` | `sha256:` and the SHA-256 of the canonical document without `digest`, `description`, and `provenance` | + +VMM-owned bits are a code table, not profile data: APIC IDs and logical +counts, `OSXSAVE`, `OSPKE`, x2APIC, the XSAVE sizes for the current XCR0 and +XSS, and cache-sharing counts; the topology leaves `0xB`, `0x1F`, +`0x8000001E`, and `0x80000026` are absent from profiles. The time ABI owns +`0x40000000..=0x4fffffff`, `0x15`, and `0x16`, which profiles exclude, and +every profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and +ARAT are pinned set and exempt from host-support verification, because +nested Azure hypervisors hide them from fingerprints; host qualification +measures them instead. Profiles also pin policy zeros (VMX and SVM, SGX, PT, +RDT, PCONFIG, and the other features listed by the profiles' generator). +`IA32_UCODE_REV` is not pinned in v1. The effective guest CPUID is a pure +function of the profile, the VM topology, and the time ABI. + +**Selection.** A microVM always has a profile. A cold boot uses +`--cpu-profile `, or `auto` (the default), which selects the highest +revision of the single profile whose backend is the running backend and +whose `cpus` cover the host's vendor, family, model, and stepping as the +VMM's host OS sees them (the L1 view on Azure). No match, or matches in more +than one generation, is `E_PROFILE_HOST_UNKNOWN`, naming the host's +signature and the available IDs; host CPUID passthrough does not exist. A +restore always uses the profile recorded in the snapshot; an explicit +`--cpu-profile` must name the same profile. + +**Verification.** At partition creation, for cold boot and restore, OpenVMM +reports every violation at once, naming the leaf, subleaf, register, and bit: + +1. The profile is valid: schema, digest, density, the VMM-owned mask table, + and the [CPU time bits](#cpu-time-bits) (`E_PROFILE_TIME_BITS`; a catalog + profile that fails is a build defect caught by unit tests). +2. The backend is the profile's (`E_BACKEND_MISMATCH`), and the host's + vendor, family, model, and stepping are in `cpus` (`E_CPU_GENERATION`). +3. The backend supports the profile (`E_PROFILE_UNSUPPORTED`): every set + feature bit is a supported bit, every limit (maximum leaves, address + widths) is within the backend's, every enabled XSAVE component has the + same size, offset, and flags, XCR0 and XSS are subsets, every pinned MSR + value can be presented (an `ARCH_CAPABILITIES` immunity the host lacks is + a violation), and on MSHV and WHP the profile's banks are subsets of the + host's. Time-ABI-owned bits and bits the backend emulates (the + hypervisor bit, KVM's in-kernel x2APIC) are exempt. +4. On restore only: this OpenVMM pins a profile with the same ID and digest, + and the recorded document hashes to it (`E_PROFILE_UNKNOWN`, + `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the + recorded one (`E_CPU_SURFACE`). The effective-CPUID record replaces the + exact-equality CPU contract. + +The initial catalog has eight profiles: `intel.skylake-sp.kvm.v1`, +`intel.skylake-sp.mshv.v1`, `intel.skylake-sp.whp.v1` (family 6, model 85, +steppings 0 to 4: the bare-metal hosts), `intel.icelake-sp.kvm.v1`, +`intel.icelake-sp.mshv.v1`, `intel.icelake-sp.whp.v1` (6/106: the Xeon +Platinum 8370C runners), `intel.emeraldrapids.mshv.v1`, and +`intel.emeraldrapids.whp.v1` (6/207: the 8573C runners; no KVM host exists). +Their contents and the fate of the `MYSTERY_MSRS` stubs are TBD(profiles). ## TSC rate policy and LAPIC rate rule @@ -367,8 +400,10 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this 516 ns on MSHV (dual-socket bare metal, bounded by the probe's round trip), and at most 70 ns on WHP. -The guest warp probe (`nvx-time-probe warp` in the fleet tooling) runs two -tests on every pair of online CPUs, each for at least 100 ms per pair: +The guest warp probe (`nvx-time-probe warp`, built from +`guest/common/nvx-time-probe.c` and installed as `/sbin/nvx-time-probe`) +runs two tests on every pair of online CPUs, each for at least 100 ms per +pair: - `max_backward_ns`: the largest backward TSC step observed when the two CPUs alternately read the TSC under a shared spinlock (the Linux @@ -480,7 +515,9 @@ In the worker, with every VP stopped: (`E_TSC_TARGET_OVERFLOW` if it exceeds 64 bits). The backend writes `TSC_target` to every instantiated VP at that instant. Per-VP TSC values - from saved VP state are superseded and never applied afterwards. + from saved VP state are superseded and never applied afterwards. The + restore anchor's pairing is logged above 10 µs but not bounded: a pairing + error shifts every VP's TSC alike, by at most that amount. 13. Read back: every instantiated VP holds the synchronized value as defined by its backend (`E_TSC_SYNC_READBACK`). 14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every @@ -589,7 +626,10 @@ driver polls `0xea` and reads `0xe9` from a timer thread; a sample on the console stream could interleave with input bytes. The window is not saved state: restore empties it. Guest users of selectors `0xa5`, `0xa6`, and `0xa7` serialize their transactions with an exclusive `flock` on -`/run/nvx/portb.lock`. +`/run/nvx/portb.lock`. The snapshot agent holds it from its capture request +through repair step 8, and a discipline poll holds it from its first sample +to the published state, so a capture may wait for an in-flight poll, which +is bounded by three samples and two `adjtimex` calls. ### Restore packet @@ -631,6 +671,12 @@ packet; the packet is authoritative. `ACK_REQUIRED` replaces the guest's inference of gating from the tier and targets, so an untiered guest never writes `0x605` after an ungated restore. +A VM process exposes at most one packet, and reading its last byte consumes +it: status bits 1 to 4 clear and later `0xa5` writes select nothing. A guest +that is captured again in the same process, including after a rejected +capture, therefore never sees the earlier packet. It also ignores a packet +whose `g` equals its recorded `g`. + ### Time sample Writing `0xa7` to `0xea` latches a 16-byte sample into the window at `0xeb`: @@ -705,12 +751,13 @@ and the managed agent. Init runs the boot check immediately after mounting `/proc`, `/sys`, and `/dev`, before any other guest work, and starts the daemon only if it passes. -CPUID and MSRs are read on every online CPU through `/dev/cpu//cpuid` and -`/dev/cpu//msr`. +CPUID is executed on every online CPU (the checker pins itself to each CPU in +turn), and MSRs are read through `/dev/cpu//msr`. A CPU's VP index is its +Linux CPU number, because CPUs come online as a prefix in APIC-ID order. | ID | Check | Phases | | --- | --- | --- | -| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; `0x40000006` and `0x40000081` follow the out-of-range rule; no base `0x40000100..=0x4000ff00` carries `KVMKVMKVM` | boot, restore (new CPUs) | +| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; the explicit zero leaves `0x40000006..=0x4000000f` and `0x40000080..=0x40000082` are zero; at boot, on one CPU, no base `0x40000100..=0x4000ff00` carries `KVMKVMKVM` | boot, restore (new CPUs) | | `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | | `C3` | MSR `0x40000002` equals the CPU's VP index; `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz; `0x40000023` equals 1,000,000,000 or 200,000,000; `0x40000118` equals 1; reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | | `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | @@ -719,7 +766,7 @@ CPUID and MSRs are read on every online CPU through `/dev/cpu//cpuid` and | `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | | `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | | `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | -| `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0 | capture, restore | +| `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0; at boot, the state file is published and the daemon started | boot, capture, restore | | `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | | `C12` | A time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | @@ -744,6 +791,14 @@ workload starts and is not counted as a discontinuity. The only other line with this prefix is the non-fatal `phase=runtime status=uncertain` line of the [wall-clock discipline](#wall-clock-discipline). +**Report-only mode (test only).** The kernel token `nvx_time_abi=report-only` +makes the checks and the watcher report instead of powering off, so the guest +can be evaluated under an OpenVMM that does not implement the time ABI. It +prints `NVX-TIME-REPORT` and `NVX-TIME-REPORT-VIOLATION` lines with the same +fields, and its marker ends with ` failures=`. These prefixes are reserved +for this mode; production images never set the token, and harnesses never +accept a report-only boot as conformant. + ### Violation watcher The daemon reads `/dev/kmsg` from the start of the buffer and then follows @@ -779,10 +834,10 @@ NVX-TIME-ABI-VIOLATION: v=1 code= source= phas ``` The guest writes it to `/dev/kmsg` at priority 2, to the portb data port -`0xe9` through `/dev/port`, and to the console. Consumers treat identical -lines as one event. The guest then writes the status to the shutdown port -`0x604` through `nvx-exit`, whose first byte becomes the OpenVMM process exit -status. +`0xe9` (through `/dev/port` or `outb`), and to the console. Consumers treat +identical lines as one event. The guest then writes the status to the +shutdown port `0x604` (directly, or through `nvx-exit`); its first byte +becomes the OpenVMM process exit status. ### Snapshot agent @@ -864,7 +919,8 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: `epsilon <= 50 µs`. Otherwise skip the poll, count a rejected sample, set `last_sample_error=G_SAMPLE_UNCERTAIN`, and print `NVX-TIME-ABI: v=1 phase=runtime status=uncertain code=G_SAMPLE_UNCERTAIN - epsilon_ns=` on the console. The next accepted sample resets + epsilon_ns=` on the console, with `epsilon_ns=none` when no + attempt produced a valid sample. The next accepted sample resets `last_sample_error` to `none`. A sample whose `g` differs from the recorded `g` is discarded: a restore happened, and restore repair resets the discipline. @@ -923,13 +979,13 @@ code. | --- | --- | --- | | `E_SNAPSHOT_VERSION` | Manifest version is not 6; the snapshot must be recaptured | Restore | | `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | -| `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | -| `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM | Restore | -| `E_PROFILE_DIGEST` | Embedded, recorded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | -| `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile | Cold boot | -| `E_PROFILE_TIME_BITS` | The profile violates the CPU time bits | Cold boot, restore | -| `E_CPU_GENERATION` | Host CPU generation not in the profile | Cold boot, restore | -| `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, or MSR value of the profile | Cold boot, restore | +| `E_BACKEND_MISMATCH` | Snapshot taken on another backend, or a profile for another backend | Cold boot, restore | +| `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM | Cold boot, restore | +| `E_PROFILE_DIGEST` | Recorded, embedded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | +| `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile, or to profiles of more than one generation | Cold boot | +| `E_PROFILE_TIME_BITS` | The profile is invalid or violates the CPU time bits | Cold boot, restore | +| `E_CPU_GENERATION` | Host CPU vendor, family, model, or stepping not in the profile | Cold boot, restore | +| `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, MSR value, or feature-bank bit of the profile, or the host is not qualified | Cold boot, restore | | `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one | Restore | | `E_IDENTITY_ROUTING` | Backend cannot deliver the identity CPUID or MSRs | Cold boot, restore | | `E_TSC_SYNC_UNSUPPORTED` | Backend lacks the synchronized TSC set | Cold boot, restore | From 18e4d9a2cad0e57881d4deba06249d1bdeb48106 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:26:23 -0700 Subject: [PATCH 012/188] doc: trim the time ABI boot check from the guest measurements The guest agent measured the boot check at 4.5 to 4.8 ms on its first run at one vCPU, over the 2 ms budget. Move the static KVM base scan from C1 into the CI exhaustive check, read the partition-wide C3 MSRs on CPU 0 only, allow one pinned thread per CPU, read the boot log once with SYSLOG_ACTION_READ_ALL after positioning /dev/kmsg at its end, state that the marker is printed synchronously after elapsed_us is taken, and state the budget per vCPU count with the gate as the authority. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 38 +++++++++++++++++++++++++++----------- 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 0dc7a917f..c662ff4c2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -752,14 +752,17 @@ and the managed agent. Init runs the boot check immediately after mounting `/proc`, `/sys`, and `/dev`, before any other guest work, and starts the daemon only if it passes. CPUID is executed on every online CPU (the checker pins itself to each CPU in -turn), and MSRs are read through `/dev/cpu//msr`. A CPU's VP index is its -Linux CPU number, because CPUs come online as a prefix in APIC-ID order. +turn, or runs one thread pinned to each CPU), and MSRs are read through +`/dev/cpu//msr`. A CPU's VP index is its Linux CPU number, because CPUs +come online as a prefix in APIC-ID order. CPUID tables are per VP on KVM, so +`C1` and `C2` run on every CPU; the MSRs other than the VP index are +partition-wide on every backend, so `C3` reads them on CPU 0 only. | ID | Check | Phases | | --- | --- | --- | -| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; the explicit zero leaves `0x40000006..=0x4000000f` and `0x40000080..=0x40000082` are zero; at boot, on one CPU, no base `0x40000100..=0x4000ff00` carries `KVMKVMKVM` | boot, restore (new CPUs) | +| `C1` | Identity leaves `0x40000000..=0x40000005` equal the [identity table](#hypervisor-identity), with `C` equal to the number of possible CPUs; the explicit zero leaves `0x40000006..=0x4000000f` and `0x40000080..=0x40000082` are zero | boot, restore (new CPUs) | | `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | -| `C3` | MSR `0x40000002` equals the CPU's VP index; `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz; `0x40000023` equals 1,000,000,000 or 200,000,000; `0x40000118` equals 1; reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | +| `C3` | MSR `0x40000002` equals the CPU's VP index on every CPU; on CPU 0, `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz, `0x40000023` equals 1,000,000,000 or 200,000,000, `0x40000118` equals 1, and reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | | `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | | `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU (`nonstop_tsc` on WHP: TBD(whp)) | boot, restore (new CPUs) | | `C6` | `current_clocksource` is `tsc`; `available_clocksource` contains only `tsc`, `refined-jiffies`, and `jiffies` | boot, capture, restore | @@ -773,9 +776,12 @@ Linux CPU number, because CPUs come online as a prefix in APIC-ID order. Capture and restore checks never wait. The boot `C7` check and the deferred restore `C7` check may poll for at most 200 ms, because a CPU switches to one-shot mode at its first tick. The CI conformance suite adds an exhaustive -check: every leaf `0x40000006..=0x400000ff`, writes of 0 and 1 to -`0x40000118` and of 2 (which must fail), writes to every read-only -identity MSR (which must fail), and reads of `IA32_TSC_ADJUST` and +check on every CPU: every leaf `0x40000006..=0x400000ff`; no base +`0x40000100..=0x4000ff00` (step `0x100`) carrying `KVMKVMKVM` or another +hypervisor signature (the NVX kernel has no KVM guest support, so the boot +check leaves this static property of the backend to CI); every `C3` MSR; +writes of 0 and 1 to `0x40000118` and of 2 (which must fail); writes to every +read-only identity MSR (which must fail); and reads of `IA32_TSC_ADJUST` and `IA32_TSC_DEADLINE` (which must fail). On success the check prints one line to the console: @@ -784,6 +790,12 @@ On success the check prints one line to the console: NVX-TIME-ABI: v=1 phase= status=ok cpus= tsc_hz= lapic_hz= generation= elapsed_us= ``` +`elapsed_us` covers the checks and, at boot, the daemon start. The line is +printed after it is taken, synchronously and before any workload starts, so +a harness sees the marker before workload output and an early power-off +cannot lose it; its console cost (one port exit per byte) is part of the +cold-boot cost the performance gate measures. + On failure it prints `status=fail check= detail=""` in the same format, emits a violation event with code `G_CONFORMANCE_`, and powers off with status 193. The boot step of `C12` steps the clock before any @@ -801,9 +813,13 @@ accept a report-only boot as conformant. ### Violation watcher -The daemon reads `/dev/kmsg` from the start of the buffer and then follows -it; it sets `oom_score_adj` to -1000. A record matches when its message text -contains every substring of a row: +The boot check opens `/dev/kmsg`, seeks it to its end, and only then reads +the whole kernel log once with `syslog(SYSLOG_ACTION_READ_ALL)` for `C4`. The +daemon inherits that descriptor and follows it, so no record is missed; +records logged in between are seen twice, and identical lines are one event. +A log buffer that wrapped before the boot check fails `C4`, because its +required records are missing. The daemon sets `oom_score_adj` to -1000. A +record matches when its message text contains every substring of a row: | Code | Substrings | | --- | --- | @@ -1073,7 +1089,7 @@ Expected effects: | Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | | No capture-time clocksource waits | Removes up to 5 s of capture latency on MSHV and WHP; outside the restore metrics | | No LAPIC calibration and no `tsc-early` window at cold boot | Unchanged or faster cold boot | -| Boot check and daemon start | Added cold-boot cost; budget 2 ms on KVM and MSHV and 5 ms on WHP, reported as `elapsed_us` | +| Boot check and daemon start | Added cold-boot cost, reported as the first run's `elapsed_us`; budget 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP; the gate is authoritative | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore repair and checks before the acknowledgement | Run in one helper process; the RCU release and deferred checks run after the acknowledgement | From 8480c3bcae3769520105c2e43628a5937733a9cd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:33:03 -0700 Subject: [PATCH 013/188] guest: link the time tools against musl nvx-time and nvx-time-probe were static glibc binaries of 863 KB and 941 KB. The kernel unpacks the whole initramfs before init runs, so on prometheus32 each added about 6 ms to a one-vCPU cold start (paired medians +6.4 ms and +5.3 ms over 18 scenario-rounds). Linked statically against musl with the same flags, they are 104 KB and 108 KB: the initramfs shrinks from 8.58 MB to 7.90 MB compressed, and cold start is 13.9 ms faster than with the glibc builds (paired median, n = 18, range -20.3 to -6.1 ms). The musl builds behave the same in the guest, and the probe's warp test still passes. Install musl-tools in the build image and build both tools with musl-gcc; the other static helpers keep glibc. Native builds need musl-gcc (Debian's musl-tools), which doc/setup.md now lists. The third-party notices record the statically linked musl. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- THIRD_PARTY_NOTICES.md | 1 + doc/build.md | 4 +++- doc/setup.md | 5 ++++- docker/Dockerfile | 1 + scripts/nvx_tools/build.py | 11 ++++++++++- scripts/nvx_tools/build_constants.py | 5 ++++- scripts/test_guest_time.py | 27 +++++++++++++++++++++++++++ scripts/test_nvx_tools.py | 22 +++++++++++++++++++++- 8 files changed, 71 insertions(+), 5 deletions(-) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index b9c510fcd..bc9ab58f9 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -9,6 +9,7 @@ the distribution tooling. | Linux 6.18.38 | Generated under `.cache/linux` | GPL-2.0-only; the upstream archive and patch series are pinned in `SOURCE-MANIFEST.json` and `kernel/patches`. Each build exports its exact config, and `collect-sources` emits the complete patched tree. | | OpenVMM | `openvmm` | MIT; see `openvmm/LICENSE`. Source is a private Git submodule and is available only to authorized users. | | NVX guest/build sources | `guest`, `ubuntu`, `kernel`, `scripts/nvx_tools` | MIT; see `LICENSE`. | +| musl libc (Debian 12 `musl`) | statically linked into `/sbin/nvx-time` and `/sbin/nvx-time-probe` in generated initramfs images | MIT; Copyright © 2005-2020 Rich Felker, et al. The build image's `/usr/share/doc/musl/copyright` carries the full notice, and the source is Debian's `musl` source package. | | Alpine Linux packages | generated initramfs images | Per-package licenses. Every build emits `*.packages.json`; retain that manifest and collect the corresponding aports recipes and upstream sources before distributing an image. | | Azure Linux 3.0 packages | generated Azure Linux initramfs | Per-package licenses, including GPLv2 BusyBox. The base image is digest-pinned in `SOURCE-MANIFEST.json`, every supplemental RPM is SHA-256 pinned in `azurelinux/packages.lock.json`, and every build emits `initramfs-azurelinux.cpio.gz.packages.json`. `collect-sources` does not collect this corresponding source yet, so `--include-source` packages omit the Azure Linux initramfs; publish the matching source before distributing a binary-only package that contains it. | | Ubuntu Base 26.04.1 LTS and supplemental packages | generated Ubuntu initramfs and EROFS images | Per-package licenses. The base archive and every supplemental `.deb` are SHA-256 pinned. Retain the generated Ubuntu manifests and publish the matching `.dsc` plus all referenced source members before distributing an image. Ubuntu artifacts contain Ubuntu userland with the NVX kernel, not the Ubuntu kernel. | diff --git a/doc/build.md b/doc/build.md index 227928606..58302a26f 100644 --- a/doc/build.md +++ b/doc/build.md @@ -151,7 +151,9 @@ checks, the violation watcher, the restore packet v4 and time-sample parsers, the snapshot agent's time steps, and the wall-clock discipline. Its `--report-only` flag (or the `nvx_time_abi=report-only` kernel command-line token) reports violations without powering off, for runs against VMMs that -predate the ABI. The matching kernel enables +predate the ABI. Both time tools are linked statically against musl +(`musl-gcc`): with static glibc each would add about 700 KB to the initramfs, +and every unpacked kilobyte delays cold boot. The matching kernel enables virtio-blk, compressed EROFS, overlayfs, ext4 scratch, memory cgroups, and cgroup BPF. The build fails if `olddefconfig` drops any required option. The APK manifest records the `blkid` and `util-linux` tools used by the bootstrap diff --git a/doc/setup.md b/doc/setup.md index fc66166f1..b8f9003ad 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -35,9 +35,12 @@ For Debian/Ubuntu hosts: sudo apt-get update sudo apt-get install -y \ bc binutils bison build-essential ca-certificates cpio curl erofs-utils flex \ - gzip libelf-dev libssl-dev python3 rsync tar xz-utils zstd + gzip libelf-dev libssl-dev musl-tools python3 rsync tar xz-utils zstd ``` +`musl-tools` provides `musl-gcc`, which native initramfs builds use for the +guest time tools; Docker builds install it in the build image. + ### Linux / KVM Enable Intel VT-x or AMD-V in the host firmware. Linux normally loads the matching KVM kernel diff --git a/docker/Dockerfile b/docker/Dockerfile index 1047a7e6d..88630c59d 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -20,6 +20,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ gzip \ libelf-dev \ libssl-dev \ + musl-tools \ patch \ python3 \ rsync \ diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 772e47277..9f0b51568 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -577,8 +577,9 @@ def _build_static_helper( source: Path, destination: Path, cflags: tuple[str, ...] = InitramfsBuildConstants.STATIC_HELPER_CFLAGS, + compiler_name: str = "cc", ) -> dict[str, str]: - compiler = require_tool("cc") + compiler = require_tool(compiler_name) output = work / source.stem run_checked( [ @@ -841,12 +842,20 @@ def _install_guest_files( common / f"{name}.c", root / "sbin" / name, ) + for name in InitramfsBuildConstants.MUSL_STATIC_HELPERS: + helpers[name] = _build_static_helper( + config.work, + common / f"{name}.c", + root / "sbin" / name, + compiler_name=InitramfsBuildConstants.MUSL_COMPILER, + ) probe = InitramfsBuildConstants.TIME_PROBE_NAME helpers[probe] = _build_static_helper( config.work, common / f"{probe}.c", root / "sbin" / probe, InitramfsBuildConstants.TIME_PROBE_CFLAGS, + compiler_name=InitramfsBuildConstants.MUSL_COMPILER, ) helpers["nvx-device-io"] = _build_device_io_helper( config.work, diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 708468312..5d6a99ab4 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -310,8 +310,11 @@ class InitramfsBuildConstants: "nvx-port-io", "nvx-console-pending", "nvx-managed-agent", - "nvx-time", ) + # Time tools are linked against musl: a static glibc binary adds about + # 700 KB to the initramfs, and every unpacked kilobyte delays cold boot. + MUSL_COMPILER: Final = "musl-gcc" + MUSL_STATIC_HELPERS: Final = ("nvx-time",) TIME_PROBE_NAME: Final = "nvx-time-probe" TIME_PROBE_CFLAGS: Final = ( "-std=gnu11", diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index ba30af682..1f077f8c6 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -184,6 +184,33 @@ def test_packet_v4_golden_vector(self): f"utc_ns={UTC_NS} generation_id={ENTROPY[:16].hex()}", ) + def test_musl_build_matches_the_reference_build(self): + musl = shutil.which("musl-gcc") + if musl is None: + self.skipTest("musl-gcc is unavailable") + binary = Path(self.directory.name) / "nvx-time-musl" + subprocess.run( + [musl, "-static", "-Os", "-s", "-Wall", "-Wextra", "-Werror"] + + ["-o", str(binary), str(SOURCE)], + check=True, + capture_output=True, + timeout=120, + ) + packet = self.fixture( + "musl-packet.bin", encode_packet(flags=ACK_REQUIRED, generation=2) + ) + arguments = ["test", "packet", packet, "1", PREVIOUS_ID] + result = subprocess.run( + [str(binary), *arguments], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout, self.run_test(*arguments[1:])) + def test_packet_memory_ranges_follow_the_header(self): packet = encode_packet( flags=MEMORY_TARGET, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 0572afc44..5ac50e6db 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -6336,8 +6336,11 @@ def compile_helper( output = Path(command[command.index("-o") + 1]) output.write_bytes(f"static-elf {output.name}".encode()) + def find_tool(name: str) -> str: + return name + with ( - patch.object(build, "require_tool", return_value="cc"), + patch.object(build, "require_tool", side_effect=find_tool), patch.object(build, "run_checked", side_effect=compile_helper), ): helpers = build._install_guest_files( @@ -6351,16 +6354,33 @@ def compile_helper( Path(command[-1]).stem: command[1 : command.index("-o")] for command in commands } + compiler_by_source = { + Path(command[-1]).stem: command[0] for command in commands + } self.assertEqual( flags_by_source[probe], list(InitramfsBuildConstants.TIME_PROBE_CFLAGS) ) self.assertIn("-pthread", flags_by_source[probe]) + musl = InitramfsBuildConstants.MUSL_COMPILER + self.assertEqual(compiler_by_source[probe], musl) for name in InitramfsBuildConstants.STATIC_HELPERS: with self.subTest(helper=name): self.assertEqual( flags_by_source[name], list(InitramfsBuildConstants.STATIC_HELPER_CFLAGS), ) + self.assertEqual(compiler_by_source[name], "cc") + for name in InitramfsBuildConstants.MUSL_STATIC_HELPERS: + with self.subTest(helper=name): + self.assertEqual( + flags_by_source[name], + list(InitramfsBuildConstants.STATIC_HELPER_CFLAGS), + ) + self.assertEqual(compiler_by_source[name], musl) + dockerfile = (BuildConstants.REPO_ROOT / "docker" / "Dockerfile").read_text( + encoding="utf-8" + ) + self.assertIn(" musl-tools \\\n", dockerfile) self.assertEqual(helpers[probe]["source_sha256"], common.sha256_file(source)) self.assertEqual( helpers[probe]["binary_sha256"], From 64dda405b266242cf7974dd26560c8f6945d3b93 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:50:06 -0700 Subject: [PATCH 014/188] guest: trim the nvx-time boot check to the revised spec The time ABI spec now trims the boot check after measuring it at 4.5 to 4.8 ms on its first run at one vCPU, against a 2 ms budget, and settles the guest questions. Follow it: - C1 checks the identity leaves and requires the explicit zero leaves 0x40000006..=0x4000000f and 0x40000080..=0x40000082 to read zero. The 255-leaf scan for a KVM signature at other bases moves to the CI exhaustive check; the NVX kernel has no KVM guest support. - C3 reads the VP index on every CPU and the partition-wide MSRs on CPU 0 only, which saves six MSR exits per further CPU. - C4 positions the daemon's /dev/kmsg descriptor at the end of the log and then reads the whole log once with SYSLOG_ACTION_READ_ALL instead of one read() per record. A wrapped buffer fails C4 through its missing records, so the EPIPE path is gone. Only lines that contain a required record, a forbidden record, or the first substring of a watcher row can change the verdict, so a few whole-buffer searches locate them and only those lines are matched. Matching every rule against every line had become the largest boot-check cost with musl, whose string functions are simpler than glibc's. - The uncertain discipline line prints epsilon_ns=none when no attempt produced a valid sample, and a report-only marker always ends with failures=. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-time.c | 282 ++++++++++++++++++++++++++++--------- scripts/test_guest_time.py | 27 ++++ 2 files changed, 246 insertions(+), 63 deletions(-) diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 943b3c137..b0757127a 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -1172,7 +1173,6 @@ struct boot_log { bool hypervisor; bool privileges; bool lapic; - bool overrun; char last_switch[48]; char problem[DETAIL_MAX]; }; @@ -1219,9 +1219,7 @@ static void boot_log_add(struct boot_log *log, const char *message) static bool boot_log_verdict(const struct boot_log *log, char *detail, size_t size) { - if (log->overrun) - snprintf(detail, size, "kernel log records were lost (EPIPE)"); - else if (log->problem[0] != '\0') + if (log->problem[0] != '\0') snprintf(detail, size, "%s", log->problem); else if (!log->hypervisor) snprintf(detail, size, "no 'Hypervisor detected: Microsoft Hyper-V'"); @@ -1271,9 +1269,155 @@ static int kmsg_drain(int fd, kmsg_consumer consume, void *context) } } -static void boot_log_consume(void *context, const char *message) +#define SYSLOG_ACTION_READ_ALL 3 +#define SYSLOG_ACTION_SIZE_BUFFER 10 + +// Returns the message text of one SYSLOG_ACTION_READ_ALL line, without its +// "" prefix and its printk timestamp ("[ 1.234567] "). +static const char *syslog_message(const char *line) +{ + const char *cursor = line; + const char *end = strchr(line, '>'); + const char *scan; + + if (*cursor == '<' && end != NULL) + cursor = end + 1; + if (*cursor != '[') + return cursor; + scan = cursor + 1; + while (*scan == ' ') + scan++; + while (*scan >= '0' && *scan <= '9') + scan++; + if (*scan++ != '.') + return cursor; + while (*scan >= '0' && *scan <= '9') + scan++; + if (*scan != ']') + return cursor; + return scan[1] == ' ' ? scan + 2 : scan + 1; +} + +static void boot_log_line(void *context, char *line) +{ + boot_log_add(context, syslog_message(line)); +} + +struct line_starts { + const char **starts; + size_t count; + size_t capacity; +}; + +// Records the start of every line of BUFFER that contains NEEDLE. +static bool mark_lines(const char *buffer, const char *needle, + struct line_starts *lines) +{ + for (const char *hit = strstr(buffer, needle); hit != NULL; + hit = strstr(hit + 1, needle)) { + const char *start = hit; + + while (start > buffer && start[-1] != '\n') + start--; + if (lines->count == lines->capacity) { + size_t capacity = lines->capacity * 2; + const char **grown = + realloc(lines->starts, capacity * sizeof(*lines->starts)); + + if (grown == NULL) + return false; + lines->starts = grown; + lines->capacity = capacity; + } + lines->starts[lines->count++] = start; + } + return true; +} + +static int compare_starts(const void *left, const void *right) +{ + const char *a = *(const char *const *)left; + const char *b = *(const char *const *)right; + + return a < b ? -1 : a > b; +} + +// C4 over the whole SYSLOG_ACTION_READ_ALL text. Only lines that contain a +// required record, a forbidden record, or the first substring of a watcher +// row can change the verdict, so a few whole-buffer searches find them, and +// boot_log_add() sees just those lines, in log order. Matching every rule +// against every line cost about a millisecond on a guest's boot log. +static void boot_log_scan(char *buffer, struct boot_log *log) +{ + static const char *const anchors[] = { + "Hypervisor detected: Microsoft Hyper-V", + "Hyper-V: privilege flags low 0x8860,", + "Hyper-V: LAPIC Timer Frequency: ", + "Fast TSC calibration", + "Refined TSC clocksource calibration", + "kvm-clock", + "APIC timer: using supplied frequency"}; + struct line_starts lines = {malloc(256 * sizeof(const char *)), 0, 256}; + bool ok = lines.starts != NULL; + + for (size_t index = 0; ok && index < ARRAY_SIZE(anchors); index++) + ok = mark_lines(buffer, anchors[index], &lines); + for (size_t index = 0; ok && index < ARRAY_SIZE(k_watch_rules); index++) { + const struct watch_rule *rule = &k_watch_rules[index]; + + if (rule->all[0] != NULL) { + ok = mark_lines(buffer, rule->all[0], &lines); + continue; + } + for (size_t part = 0; + ok && part < ARRAY_SIZE(rule->any) && rule->any[part]; part++) + ok = mark_lines(buffer, rule->any[part], &lines); + } + if (!ok) { + free(lines.starts); + for_each_text_line(buffer, boot_log_line, log); + return; + } + qsort(lines.starts, lines.count, sizeof(*lines.starts), compare_starts); + for (size_t index = 0; index < lines.count; index++) { + char *start = (char *)lines.starts[index]; + char *end = strchr(start, '\n'); + + if (index > 0 && lines.starts[index - 1] == start) + continue; + if (end != NULL) + *end = '\0'; + boot_log_add(log, syslog_message(start)); + if (end != NULL) + *end = '\n'; + } + free(lines.starts); +} + +// Reads the whole kernel log for C4 with one SYSLOG_ACTION_READ_ALL. The +// caller has already positioned the daemon's /dev/kmsg descriptor at the end +// of the log, so records logged in between are seen twice, never missed. +static int read_boot_log(struct boot_log *log) { - boot_log_add(context, message); + int size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0); + char *buffer; + int count; + int error; + + if (size <= 0) + return -1; + buffer = malloc((size_t)size + 1); + if (buffer == NULL) + return -1; + count = klogctl(SYSLOG_ACTION_READ_ALL, buffer, size); + error = errno; + if (count >= 0) { + buffer[count] = '\0'; + boot_log_scan(buffer, log); + } + free(buffer); + errno = error; + return count < 0 ? -1 : 0; } // --------------------------------------------------------------------------- @@ -1360,17 +1504,15 @@ static int pin_to_cpu(int cpu) return sched_getcpu() == cpu ? 0 : -1; } -static const uint32_t k_kvm_signature[3] = {0x4b4d564b, 0x564b4d56, - 0x0000004d}; - static bool all_zero(const uint32_t value[4]) { return (value[0] | value[1] | value[2] | value[3]) == 0; } -// C1: the identity leaves, the out-of-range rule, and (on one CPU, because it -// costs 255 CPUID exits) the absence of a KVM signature at any base. -static bool check_identity(struct checks *checks, int cpu, bool scan_bases) +// C1: the identity leaves and the explicit zero leaves. The absence of a +// hypervisor signature at other bases is a static backend property that the +// CI exhaustive check covers, so the boot check skips that 255-leaf scan. +static bool check_identity(struct checks *checks, int cpu) { const uint32_t capacity = (uint32_t)checks->possible; const uint32_t expected[6][4] = { @@ -1381,9 +1523,9 @@ static bool check_identity(struct checks *checks, int cpu, bool scan_bases) {0, 0xffffffff, 0, 0}, {capacity, capacity, 0, 0}, }; - static const uint32_t out_of_range[] = {0x40000006, 0x40000081}; + static const uint32_t zero_leaves[][2] = {{0x40000006, 0x4000000f}, + {0x40000080, 0x40000082}}; uint32_t value[4]; - uint32_t highest[4]; for (uint32_t index = 0; index < 6; index++) { cpuid(0x40000000 + index, 0, value); @@ -1397,26 +1539,17 @@ static bool check_identity(struct checks *checks, int cpu, bool scan_bases) value[3], expected[index][0], expected[index][1], expected[index][2], expected[index][3]); } - cpuid(0, 0, value); - cpuid(value[0], 0, highest); - for (size_t index = 0; index < ARRAY_SIZE(out_of_range); index++) { - cpuid(out_of_range[index], 0, value); - if (!all_zero(value) && memcmp(value, highest, sizeof(value)) != 0) - return check_failed(checks, "C1", - "cpu %d leaf 0x%08" PRIx32 - " is neither zero nor out of range", - cpu, out_of_range[index]); - } - if (!scan_bases) - return true; - for (uint32_t base = 0x40000100; base <= 0x4000ff00; base += 0x100) { - cpuid(base, 0, value); - if (value[1] == k_kvm_signature[0] && value[2] == k_kvm_signature[1] && - value[3] == k_kvm_signature[2]) - return check_failed(checks, "C1", - "cpu %d leaf 0x%08" PRIx32 - " carries the KVM signature", - cpu, base); + for (size_t range = 0; range < ARRAY_SIZE(zero_leaves); range++) { + for (uint32_t leaf = zero_leaves[range][0]; leaf <= zero_leaves[range][1]; + leaf++) { + cpuid(leaf, 0, value); + if (!all_zero(value)) + return check_failed( + checks, "C1", + "cpu %d leaf 0x%08" PRIx32 " is %08" PRIx32 ":%08" PRIx32 + ":%08" PRIx32 ":%08" PRIx32 ", expected zero", + cpu, leaf, value[0], value[1], value[2], value[3]); + } } return true; } @@ -1504,8 +1637,8 @@ static int msr_read(int fd, uint32_t msr, uint64_t *value) return count < 0 ? -errno : -EIO; } -// C3: the synthetic MSRs. FULL checks every row; otherwise only the -// declared TSC rate, as the restore phase requires for CPU 0. +// C3: the synthetic MSRs. At boot (FULL), the VP index on every CPU and the +// partition-wide MSRs on CPU 0 only; at restore, CPU 0's declared TSC rate. static bool check_msrs(struct checks *checks, int cpu, bool full) { static const uint32_t faulting[] = {0x40000000, 0x40000001, 0x40000020}; @@ -1526,6 +1659,10 @@ static bool check_msrs(struct checks *checks, int cpu, bool full) "cpu %d: MSR 0x40000002 (VP index) is %" PRIu64 " (error %d)", cpu, value, error); + if (cpu != 0) { + close(fd); + return ok; + } if (ok && (error = msr_read(fd, 0x40000022, &value)) != 0) ok = check_failed(checks, "C3", "cpu %d: MSR 0x40000022 read: %s", cpu, strerror(-error)); @@ -1962,10 +2099,9 @@ static bool check_debug_watchdogs(struct checks *checks) // three checks are independent, so a report-only run shows each failure; // the loop stops at the first CPU that fails. static bool check_cpus(struct checks *checks, const bool *selected, - bool identity, bool scan_once, bool msrs) + bool identity, bool msrs) { cpu_set_t original; - bool scanned = !scan_once; bool ok = true; if (sched_getaffinity(0, sizeof(original), &original) != 0) @@ -1980,11 +2116,10 @@ static bool check_cpus(struct checks *checks, const bool *selected, break; } if (identity) { - if (!check_identity(checks, cpu, !scanned)) + if (!check_identity(checks, cpu)) ok = false; if (!check_time_bits(checks, cpu)) ok = false; - scanned = true; } if (msrs && !check_msrs(checks, cpu, true)) ok = false; @@ -2260,12 +2395,15 @@ static void discipline_poll(struct daemon *daemon) if (lock >= 0) close(lock); if (!outcome.accepted) { + char epsilon[24] = "none"; char line[192]; + if (result == 1) + snprintf(epsilon, sizeof(epsilon), "%" PRId64, outcome.epsilon); snprintf(line, sizeof(line), "%s: v=1 phase=runtime status=uncertain " - "code=G_SAMPLE_UNCERTAIN epsilon_ns=%" PRId64 "\n", - abi_prefix(), result == 1 ? outcome.epsilon : -1); + "code=G_SAMPLE_UNCERTAIN epsilon_ns=%s\n", + abi_prefix(), epsilon); console_write(line); } } @@ -2340,14 +2478,17 @@ static bool release_with_deadline(int64_t timeout_s) static void print_restore_marker(int failures, int cpus, uint64_t tsc_hz, uint64_t lapic_hz, int64_t elapsed_us) { + char suffix[32] = ""; char line[256]; + if (g_report_only) + snprintf(suffix, sizeof(suffix), " failures=%d", failures); snprintf(line, sizeof(line), "%s: v=1 phase=restore status=%s cpus=%d tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 " generation=%" PRIu32 - " elapsed_us=%" PRId64 "\n", + " elapsed_us=%" PRId64 "%s\n", abi_prefix(), failures == 0 ? "ok" : "fail", cpus, tsc_hz, - lapic_hz, g_generation, elapsed_us); + lapic_hz, g_generation, elapsed_us, suffix); console_write(line); } @@ -2581,7 +2722,7 @@ static void print_marker(const struct checks *checks, int64_t start_ns) char line[256]; char failures[32] = ""; - if (checks->failures != 0) + if (g_report_only) snprintf(failures, sizeof(failures), " failures=%d", checks->failures); snprintf(line, sizeof(line), "%s: v=1 phase=%s status=%s cpus=%d tsc_hz=%" PRIu64 @@ -2625,15 +2766,19 @@ static int cmd_boot(void) check_clocksource(&checks); memset(&log, 0, sizeof(log)); kmsg = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC); - if (kmsg < 0) { + if (kmsg < 0 || lseek(kmsg, 0, SEEK_END) < 0) { check_failed(&checks, "C4", "open /dev/kmsg: %s", strerror(errno)); - } else { - log.overrun = kmsg_drain(kmsg, boot_log_consume, &log) == -EPIPE; - if (!boot_log_verdict(&log, detail, sizeof(detail))) - check_failed(&checks, "C4", "%s", detail); + if (kmsg >= 0) + close(kmsg); + kmsg = -1; + } else if (read_boot_log(&log) != 0) { + check_failed(&checks, "C4", "read the kernel log: %s", + strerror(errno)); + } else if (!boot_log_verdict(&log, detail, sizeof(detail))) { + check_failed(&checks, "C4", "%s", detail); } check_cpuinfo(&checks, checks.online, true); - check_cpus(&checks, checks.online, true, true, true); + check_cpus(&checks, checks.online, true, true); check_timer_list(&checks, TIMER_LIST_WAIT_NS); check_debug_watchdogs(&checks); if (check_initial_sample(&checks, &theta, &epsilon)) @@ -2681,7 +2826,7 @@ static void run_restore_checks(struct checks *checks, const bool *new_cpus, checks->lapic_hz = lapic_hz; } if (new_count > 0) { - check_cpus(checks, new_cpus, true, false, false); + check_cpus(checks, new_cpus, true, false); check_cpuinfo(checks, new_cpus, true); } check_msrs(checks, 0, false); @@ -3171,23 +3316,32 @@ static int test_sample(int argc, char **argv) return 0; } -static int test_lines(int argc, char **argv, bool boot) +// Watcher rules per line (KMSG), or the C4 verdict over message lines +// (BOOT_LOG) or SYSLOG_ACTION_READ_ALL text (SYSLOG). +enum line_test { LINES_KMSG, LINES_BOOT_LOG, LINES_SYSLOG }; + +static int test_lines(int argc, char **argv, enum line_test mode) { static char text[TEXT_MAX]; struct boot_log log; char detail[DETAIL_MAX]; + bool boot = mode != LINES_KMSG; if (argc != 1 || test_text(argv[0], text, sizeof(text)) != 0) return 2; memset(&log, 0, sizeof(log)); - for (char *line = strtok(text, "\n"); line != NULL; - line = strtok(NULL, "\n")) { - const char *code = match_watch_rule(line); - - if (boot) - boot_log_add(&log, line); - else - printf("%s\n", code != NULL ? code : "none"); + if (mode == LINES_SYSLOG) { + boot_log_scan(text, &log); + } else { + for (char *line = strtok(text, "\n"); line != NULL; + line = strtok(NULL, "\n")) { + const char *code = match_watch_rule(line); + + if (boot) + boot_log_add(&log, line); + else + printf("%s\n", code != NULL ? code : "none"); + } } if (boot) { if (boot_log_verdict(&log, detail, sizeof(detail))) @@ -3310,9 +3464,11 @@ static int cmd_test(int argc, char **argv) return 0; } if (strcmp(name, "kmsg") == 0) - return test_lines(argc, argv, false); + return test_lines(argc, argv, LINES_KMSG); if (strcmp(name, "boot-log") == 0) - return test_lines(argc, argv, true); + return test_lines(argc, argv, LINES_BOOT_LOG); + if (strcmp(name, "syslog") == 0) + return test_lines(argc, argv, LINES_SYSLOG); if (strcmp(name, "timer-list") == 0) return test_timer_list(argc, argv); if (strcmp(name, "correction") == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 1f077f8c6..2cad54147 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -436,6 +436,33 @@ def test_boot_log_requires_the_identity_and_a_final_switch_to_tsc(self): with self.subTest(case=name): self.assertTrue(self.boot_log(lines).startswith("fail "), name) + def test_boot_log_reads_syslog_read_all_lines(self): + # SYSLOG_ACTION_READ_ALL prefixes every line with "" and, with + # printk timestamps, "[seconds.micros] ". + lines = [ + f"<6>[{index:5d}.{index * 7919 % 1000000:06d}] {line}" + for index, line in enumerate(self.BOOT_LOG) + ] + lines.insert(1, "<6>[ 0.123456] [drm] a bracketed message is kept") + own_event = ( + "<2>[ 3.000000] NVX-TIME-REPORT-VIOLATION: v=1 code=G_CONFORMANCE_C4 " + 'detail="forbidden record: kvm-clock: Using msrs"' + ) + + def syslog(text_lines: list[str]) -> str: + return self.run_test( + "syslog", self.fixture("syslog.txt", "\n".join(text_lines) + "\n") + ).strip() + + self.assertEqual(syslog(lines), "ok") + self.assertEqual(syslog([*lines, own_event]), "ok") + self.assertTrue( + syslog([*lines, "<6>[ 4.000000] kvm-clock: Using msrs"]).startswith( + "fail forbidden record: kvm-clock" + ) + ) + self.assertTrue(syslog(lines[2:]).startswith("fail no 'Hypervisor")) + def timer_list( self, sections: list[tuple[int, str, int, str, int]], From c301cbd255b508a086b77f1a5d0748545ad288a3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 05:51:31 -0700 Subject: [PATCH 015/188] doc: settle the time ABI CPU profiles and WHP cells from the landed work The profiles agent landed one CPU profile per generation, shared by every backend (intel.skylake-sp.v1, intel.icelake-sp.v1, and intel.emeraldrapids.v1), which settles the TBD(profiles) choice of one profile per generation or per backend. Describe the landed format, selection without a backend, and verification, keep E_BACKEND_MISMATCH for snapshots only, and name the canonical effective-CPUID encoding. Fill the WHP obligations from the WHP report's guest view: CPUID exits for the identity and time-bit leaves, banks derived from the profile, UnhandledMsrs exits with the offloaded APIC, the fixed 200 MHz LAPIC, and the invariant-TSC bit set by the CPUID override, so C5 and the time bits apply unchanged on WHP. Record the MSHV capture-anchor method. Hosts without a host-visible invariant TSC qualify on H4 and H6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 173 ++++++++++++++++++++++------------------- 1 file changed, 94 insertions(+), 79 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c662ff4c2..88be22983 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -8,7 +8,8 @@ MSHV, and WHP. It is authoritative for the `time-abi-v1` integration branches of `microsoft/nvx` and `nanvix/openvmm`. When implemented, it replaces the time rules in [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) and the clock tokens in [Cold boot](cold-boot.md#effective-command-line). -Entries marked `TBD()` wait for that Phase 1 spike's report. +Entries marked `TBD()` wait for that agent's backend implementation or +measurements. Notation: @@ -131,7 +132,7 @@ Every CPU profile fixes these bits. A profile that violates them is rejected. | `0xa` | EAX, EBX, ECX, EDX | 0 (no PMU) | | `0x15`, `0x16` | EAX, EBX, ECX, EDX | 0, when within the maximum basic leaf | | `0x80000001` | EDX[27] RDTSCP | 1 | -| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only); on WHP, whether EDX[8] must be set is TBD(whp) | +| `0x80000007` | EAX, EBX, ECX, EDX | 0, 0, 0, `0x00000100` (invariant TSC only) | Leaves `0x15` and `0x16` are zeroed rather than synthesized. Linux takes both rates from the frequency MSRs, and zero leaves keep the profile's CPUID @@ -142,9 +143,9 @@ CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag and spares each AP a delay-loop calibration (about 150 ms per AP on Azure MSHV). The bit is policy, not a fingerprint feature: profiles set it even where a nested hypervisor hides it from host fingerprints (Azure's MSHV and WHP L1 -partitions), and host qualification backs it with measured invariance. Until -the WHP report settles whether WHP can and must expose it, a WHP profile may -clear it (see [Host qualification](#host-qualification)). +partitions), every backend exposes it (WHP through its CPUID override), and +host qualification backs it with measured invariance (see +[Host qualification](#host-qualification)). ### Rates @@ -223,16 +224,16 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight | CPUID result list or CPUID exits for the identity and explicit zero leaves — TBD(whp) | -| Profile CPUID and time bits | `KVM_SET_CPUID2` | Processor feature banks plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks plus CPUID result list or exits — TBD(whp) | -| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | Unhandled-MSR exits enabled for the offloaded APIC; no `hv1_emulator` — TBD(whp) | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), plus CPUID exits for leaves 1, 6, 7, `0xA`, `0x15`, `0x16`, and `0x80000007` | +| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | -| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC only, 200 MHz; the emulated APIC is rejected — TBD(whp) | -| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits cleared and CPUID bits cleared — TBD(whp) | -| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC are rejected (`azure-azlinux-2`) | Azure WHP hosts report no invariant TSC to the L1 partition; hosts qualify on the measured warp and rate stability, and whether the CPUID bit must be set is TBD(whp) | +| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | +| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; the hypervisor raises #GP for `IA32_TSC_ADJUST` | +| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC fail qualification (`azure-azlinux-2`) | Set by the CPUID override: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`), so those hosts qualify on the warp probe and rate stability (the guest TSC matched the declared rate against host QPC within 0.001 ppm over 118 s) | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads (0.1 to 0.6 µs) | TBD(mshv) | TBD(whp) | +| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads (0.1 to 0.6 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | TBD(whp) | | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and thaw at the first VP run (56 to 312 µs for 1 to 8 VPs) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | @@ -249,87 +250,99 @@ timestamp read. ## CPU profiles -A CPU profile is the complete guest-visible CPU surface for one vendor, CPU -generation, and backend. The guest sees only profile values, never host -passthrough, except for a fixed, code-defined set of VMM-owned fields and the -fields the time ABI owns. Profiles are data: derived mechanically by +A CPU profile is the complete guest-visible CPU surface of one vendor and CPU +generation, shared by every backend. The guest sees only profile values, never +host passthrough, except for a fixed, code-defined set of VMM-owned fields and +the fields the time ABI owns. Profiles are data: derived mechanically by intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker -`cpu-template-helper` workflow), reviewed, checked in at -`vmm_core/cpu_profile/profiles/.json`, embedded in the OpenVMM binary, -and immutable once released. A normative change creates a new revision; -released revisions are never deleted, so old snapshots stay restorable. -Profiles are per backend because MSHV and WHP gate features through -processor feature banks that a CPUID table cannot express, and cross-backend -restore is rejected anyway. - -**Format.** Schema `openvmm-cpu-profile/v1`, a canonical JSON document (keys -sorted by UTF-8 bytes, no whitespace, numbers as hexadecimal strings): +`cpu-template-helper` workflow) per register, first within each backend and +then across backends, reviewed, checked in at +`vmm_core/cpu_profile/profiles/.json`, embedded in the OpenVMM binary, and +immutable once released. A normative change creates a new revision; released +revisions are never deleted, so old snapshots stay restorable. One surface +serves all three backends: MSHV and WHP derive their processor feature banks +from the profile's CPUID (`cpu_profile::hv_banks`) instead of from the host's, +and each backend verifies at partition creation that it supports the profile. +A shared profile gives the same guest behavior on every backend; it does not +make snapshots portable, because cross-backend restore is rejected +(`E_BACKEND_MISMATCH`). + +**Format.** Schema `openvmm-cpu-profile/v1`. The canonical encoding is +compact canonical JSON (object keys sorted by their UTF-8 bytes, no +whitespace, numbers as hexadecimal strings); the profile digest is the +SHA-256 of that encoding, and decoding accepts only canonical bytes, so a +profile has exactly one encoding and one digest. Pinned files use the same +document in pretty form. | Field | Content | | --- | --- | -| `id` | `...v`, each component `[a-z0-9-]+`, for example `intel.icelake-sp.kvm.v1` | +| `schema` | `openvmm-cpu-profile/v1` | +| `id` | `..v`, each component `[a-z0-9-]+`, for example `intel.icelake-sp.v1` | +| `description` | Free text | | `vendor` | The 12-byte CPUID vendor string | -| `cpus` | Allowed `(family, model, stepping range)` display signatures; stepping ranges separate model 85's Skylake-SP (0 to 4), Cascade Lake, and Cooper Lake | -| `backend` | `kvm`, `mshv`, or `whp` | -| `cpuid` | A dense table of every leaf and subleaf in `[0, max basic]` and `[0x80000000, max extended]` with a value and a mask per register; mask bit 1 pins the value, mask bit 0 marks a VMM-owned bit | -| `xsave` | `xcr0` and `xss` masks the guest may enable, and the size, offset, and flags of every enabled component | -| `msrs` | Pinned MSR values with masks, such as `IA32_ARCH_CAPABILITIES` and `IA32_PERF_CAPABILITIES` (0, no vPMU) | -| `feature_banks` | MSHV and WHP only: processor feature banks 0 and 1 and the XSAVE feature bank | -| `provenance` | Source fingerprint surface digests (informative) | -| `digest` | `sha256:` and the SHA-256 of the canonical document without `digest`, `description`, and `provenance` | - -VMM-owned bits are a code table, not profile data: APIC IDs and logical -counts, `OSXSAVE`, `OSPKE`, x2APIC, the XSAVE sizes for the current XCR0 and -XSS, and cache-sharing counts; the topology leaves `0xB`, `0x1F`, -`0x8000001E`, and `0x80000026` are absent from profiles. The time ABI owns -`0x40000000..=0x4fffffff`, `0x15`, and `0x16`, which profiles exclude, and -every profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and -ARAT are pinned set and exempt from host-support verification, because -nested Azure hypervisors hide them from fingerprints; host qualification -measures them instead. Profiles also pin policy zeros (VMX and SVM, SGX, PT, -RDT, PCONFIG, and the other features listed by the profiles' generator). -`IA32_UCODE_REV` is not pinned in v1. The effective guest CPUID is a pure -function of the profile, the VM topology, and the time ABI. +| `generation` | The generation `name` (`skylake-sp`, `icelake-sp`, or `emeraldrapids`) and its `cpus`: `(family, model, stepping range)` display signatures; stepping ranges separate model 85's Skylake-SP (0 to 4), Cascade Lake, and Cooper Lake | +| `cpuid` | A dense table of every leaf and subleaf in `[0, max basic]` and `[0x80000000, max extended]` with a value and a mask per register; mask bit 1 pins the value, mask bit 0 marks a VMM-owned or runtime-owned bit | +| `xcr0`, `xss`, `xsave_components` | The XSAVE features the guest may enable, and the size, offset, and flags of every enabled component | +| `physical_address_width` | The guest physical address width | +| `msrs` | Pinned MSR values with masks; v1 pins `IA32_ARCH_CAPABILITIES` only | +| `provenance` | The derivation method and the source fingerprints' backends, host counts, and surface digests (informative) | + +VMM-owned bits are a code table, not profile data: `CPUID.1:EBX[31:16]` +(logical count and initial APIC ID), x2APIC (`CPUID.1:ECX[21]`), the core +and cache-sharing counts in `CPUID.4:EAX[31:14]`, and the topology leaves +`0xB` and `0x1F`, which profiles omit. Runtime-owned bits mirror control +state: `OSXSAVE`, `OSPKE`, and the XSAVE sizes for the current XCR0 and XSS. +The time ABI owns `0x40000000..=0x4fffffff`, `0x15`, and `0x16`, and every +profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and ARAT are +pinned set and, with the hypervisor bit, exempt from host-support +verification, because nested Azure hypervisors hide them from fingerprints; +host qualification measures them instead. Profiles also pin policy zeros +(VMX and SVM, SGX, PT, RDT, PCONFIG, and the other features listed by the +profiles' derivation policy). `IA32_UCODE_REV` is not pinned in v1. The +effective guest CPUID is a pure function of the profile, the VM topology, +and the time ABI's identity leaves. **Selection.** A microVM always has a profile. A cold boot uses `--cpu-profile `, or `auto` (the default), which selects the highest -revision of the single profile whose backend is the running backend and -whose `cpus` cover the host's vendor, family, model, and stepping as the -VMM's host OS sees them (the L1 view on Azure). No match, or matches in more -than one generation, is `E_PROFILE_HOST_UNKNOWN`, naming the host's -signature and the available IDs; host CPUID passthrough does not exist. A -restore always uses the profile recorded in the snapshot; an explicit -`--cpu-profile` must name the same profile. +revision of the single profile whose generation covers the host's vendor, +family, model, and stepping as the VMM's host OS sees them (the L1 view on +Azure). No match, or matches in more than one generation, is +`E_PROFILE_HOST_UNKNOWN`, naming the host's signature and the available IDs; +host CPUID passthrough does not exist. A restore always uses the profile +recorded in the snapshot; an explicit `--cpu-profile` must name the same +profile (`E_PROFILE_UNKNOWN`). **Verification.** At partition creation, for cold boot and restore, OpenVMM reports every violation at once, naming the leaf, subleaf, register, and bit: -1. The profile is valid: schema, digest, density, the VMM-owned mask table, - and the [CPU time bits](#cpu-time-bits) (`E_PROFILE_TIME_BITS`; a catalog - profile that fails is a build defect caught by unit tests). -2. The backend is the profile's (`E_BACKEND_MISMATCH`), and the host's - vendor, family, model, and stepping are in `cpus` (`E_CPU_GENERATION`). +1. The profile is valid: schema, canonical encoding, density, the VMM-owned + and runtime-owned mask tables, and the [CPU time bits](#cpu-time-bits) + (`E_PROFILE_TIME_BITS`; a catalog profile that fails is a build defect + caught by unit tests). +2. The host's vendor, family, model, and stepping are in the profile's + generation (`E_CPU_GENERATION`). 3. The backend supports the profile (`E_PROFILE_UNSUPPORTED`): every set feature bit is a supported bit, every limit (maximum leaves, address widths) is within the backend's, every enabled XSAVE component has the - same size, offset, and flags, XCR0 and XSS are subsets, every pinned MSR - value can be presented (an `ARCH_CAPABILITIES` immunity the host lacks is - a violation), and on MSHV and WHP the profile's banks are subsets of the - host's. Time-ABI-owned bits and bits the backend emulates (the - hypervisor bit, KVM's in-kernel x2APIC) are exempt. + same size, offset, and flags, XCR0 and XSS are subsets, and every pinned + MSR value can be presented (an `ARCH_CAPABILITIES` immunity the host lacks + is a violation). The time policy bits are exempt. 4. On restore only: this OpenVMM pins a profile with the same ID and digest, and the recorded document hashes to it (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the recorded one (`E_CPU_SURFACE`). The effective-CPUID record replaces the exact-equality CPU contract. -The initial catalog has eight profiles: `intel.skylake-sp.kvm.v1`, -`intel.skylake-sp.mshv.v1`, `intel.skylake-sp.whp.v1` (family 6, model 85, -steppings 0 to 4: the bare-metal hosts), `intel.icelake-sp.kvm.v1`, -`intel.icelake-sp.mshv.v1`, `intel.icelake-sp.whp.v1` (6/106: the Xeon -Platinum 8370C runners), `intel.emeraldrapids.mshv.v1`, and -`intel.emeraldrapids.whp.v1` (6/207: the 8573C runners; no KVM host exists). -Their contents and the fate of the `MYSTERY_MSRS` stubs are TBD(profiles). +The catalog has three profiles, derived from the fingerprints of three +bare-metal hosts (one per backend) and fifteen Azure hosts: + +| ID | Generation | Hosts | Source backends | +| --- | --- | --- | --- | +| `intel.skylake-sp.v1` | 6/85, steppings 0 to 4 | Bare-metal prometheus hosts | KVM, MSHV, WHP | +| `intel.icelake-sp.v1` | 6/106 | Xeon Platinum 8370C runners | KVM, MSHV, WHP | +| `intel.emeraldrapids.v1` | 6/207 | Xeon Platinum 8573C runners | MSHV, WHP (no KVM host exists) | + +The fate of the `MYSTERY_MSRS` stubs is TBD(profiles). ## TSC rate policy and LAPIC rate rule @@ -591,7 +604,7 @@ reused. It adds two required fields: | 1 | `id` | `string` | Profile ID | | 2 | `sha256` | `bytes` | Profile digest, 32 bytes | | 3 | `profile` | `bytes` | Canonical profile encoding, at most 1 MiB | -| 4 | `effective_cpuid` | `bytes` | Canonical encoding of the effective guest CPUID: profile, topology, and identity leaves | +| 4 | `effective_cpuid` | `bytes` | The effective guest CPUID in its canonical encoding, `openvmm-effective-cpuid/v1` (compact canonical JSON): the profile's pinned values, the topology fields, and the identity leaves | | 5 | `effective_cpuid_sha256` | `bytes` | Digest of `effective_cpuid`, 32 bytes | | 6 | `capture_cpu_signature` | `u32` | CPUID.1:EAX of the capture host, for diagnostics | @@ -764,7 +777,7 @@ partition-wide on every backend, so `C3` reads them on CPU 0 only. | `C2` | The [CPU time bits](#cpu-time-bits) | boot, restore (new CPUs) | | `C3` | MSR `0x40000002` equals the CPU's VP index on every CPU; on CPU 0, `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz, `0x40000023` equals 1,000,000,000 or 200,000,000, `0x40000118` equals 1, and reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | | `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | -| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU (`nonstop_tsc` on WHP: TBD(whp)) | boot, restore (new CPUs) | +| `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU | boot, restore (new CPUs) | | `C6` | `current_clocksource` is `tsc`; `available_clocksource` contains only `tsc`, `refined-jiffies`, and `jiffies` | boot, capture, restore | | `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | | `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | @@ -995,7 +1008,7 @@ code. | --- | --- | --- | | `E_SNAPSHOT_VERSION` | Manifest version is not 6; the snapshot must be recaptured | Restore | | `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | -| `E_BACKEND_MISMATCH` | Snapshot taken on another backend, or a profile for another backend | Cold boot, restore | +| `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | | `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM | Cold boot, restore | | `E_PROFILE_DIGEST` | Recorded, embedded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | | `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile, or to profiles of more than one generation | Cold boot | @@ -1059,9 +1072,11 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | Qualification gates on these measured properties and on the profile's -features. On WHP in particular, the warp probe (`H6`) and rate stability -(`H4`) are the evidence for TSC invariance; whether WHP must also expose the -CPUID invariant-TSC bit is TBD(whp). +features. Hosts whose OS sees no invariant TSC (Azure WHP and nested MSHV +partitions) still expose the CPUID invariant-TSC bit through the profile, so +the warp probe (`H6`) and rate stability (`H4`) are the evidence for TSC +invariance there; a host that fails them, such as `azure-azlinux-2`, is not +qualified. Generation names used in logs, reports, and job summaries are `skylake-sp` (family 6, model 85), `icelake-sp` (6/106), and `emeraldrapids` (6/207). @@ -1127,7 +1142,7 @@ as the fleet restore matrix. | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | | Across VMs of one generation | `azure-kvm-1` to `-2`; `azure-azlinux-3` to `-4`; `azure-windows-1` to `-2`; `azure-windows-3` to `-4` | Restored | | Across generations | `azure-windows-1` (8370C) to `-3` (8573C) | `E_CPU_GENERATION` | -| Host without invariant TSC | `azure-azlinux-2` | Cold boot and restore rejected: `E_PROFILE_UNSUPPORTED` | +| Host without invariant TSC at the host OS | `azure-azlinux-2` (out of CI rotation) | Qualified only if `H4` and `H6` pass; otherwise `nvx.py doctor` fails | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | From bb75e931b501874a953ae586ebc3c123ac018fa3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 07:38:15 -0700 Subject: [PATCH 016/188] guest: fail the boot check on a writable and executable kernel mapping CONFIG_DEBUG_WX audits the kernel page tables before init runs and logs "Found insecure W+X mapping" for a mapping that is both writable and executable. On Linux 6.18 without STRICT_MODULE_RWX, the ITS mitigation's thunk pages are such a mapping, as every time ABI boot on MSHV and WHP showed. Treat the record as a hard boot failure, separate from the clock checks: check K1 emits G_KERNEL_WX with source=conformance and powers off with status 193; report-only mode reports it and continues. The boot-log scan anchors the record, ignores nvx-time's own lines, and keeps it out of C4, so a W+X kernel is reported even when C4 passes. The check ID and the code are provisional until the spec adds them to its boot-check and failure-code tables. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 4 ++- guest/common/nvx-time.c | 68 ++++++++++++++++++++++++++++---------- scripts/test_guest_time.py | 51 ++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 18 deletions(-) diff --git a/doc/build.md b/doc/build.md index 58302a26f..cdebf6579 100644 --- a/doc/build.md +++ b/doc/build.md @@ -148,7 +148,9 @@ hypervisor identity, and runs the cross-vCPU warp test that the skew bound. `nvx-time` implements the time ABI's [guest obligations](design/time-abi.md#guest-obligations): the conformance checks, the violation watcher, the restore packet v4 and time-sample parsers, -the snapshot agent's time steps, and the wall-clock discipline. Its +the snapshot agent's time steps, and the wall-clock discipline. Its boot check +also fails when the kernel's boot-time W+X audit reports a writable and +executable mapping. The `--report-only` flag (or the `nvx_time_abi=report-only` kernel command-line token) reports violations without powering off, for runs against VMMs that predate the ABI. Both time tools are linked statically against musl diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index b0757127a..60e27284a 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -1175,8 +1175,13 @@ struct boot_log { bool lapic; char last_switch[48]; char problem[DETAIL_MAX]; + char wx[DETAIL_MAX]; }; +// CONFIG_DEBUG_WX audits the kernel page tables before init runs and logs +// this for a mapping that is both writable and executable. +static const char k_wx_record[] = "Found insecure W+X mapping"; + static void boot_log_add(struct boot_log *log, const char *message) { static const char *const forbidden[] = { @@ -1186,6 +1191,9 @@ static void boot_log_add(struct boot_log *log, const char *message) const char *name = strstr(message, switched); const char *code; + if (log->wx[0] == '\0' && strncmp(message, "NVX-TIME-", 9) != 0 && + strstr(message, k_wx_record) != NULL) + snprintf(log->wx, sizeof(log->wx), "%.300s", message); if (strstr(message, "Hypervisor detected: Microsoft Hyper-V") != NULL) log->hypervisor = true; if (strstr(message, "Hyper-V: privilege flags low 0x8860,") != NULL) @@ -1342,11 +1350,12 @@ static int compare_starts(const void *left, const void *right) return a < b ? -1 : a > b; } -// C4 over the whole SYSLOG_ACTION_READ_ALL text. Only lines that contain a -// required record, a forbidden record, or the first substring of a watcher -// row can change the verdict, so a few whole-buffer searches find them, and -// boot_log_add() sees just those lines, in log order. Matching every rule -// against every line cost about a millisecond on a guest's boot log. +// C4 and K1 over the whole SYSLOG_ACTION_READ_ALL text. Only lines that +// contain a required record, a forbidden record, the W+X record, or the first +// substring of a watcher row can change the verdict, so a few whole-buffer +// searches find them, and boot_log_add() sees just those lines, in log order. +// Matching every rule against every line cost about a millisecond on a +// guest's boot log. static void boot_log_scan(char *buffer, struct boot_log *log) { static const char *const anchors[] = { @@ -1356,7 +1365,8 @@ static void boot_log_scan(char *buffer, struct boot_log *log) "Fast TSC calibration", "Refined TSC clocksource calibration", "kvm-clock", - "APIC timer: using supplied frequency"}; + "APIC timer: using supplied frequency", + k_wx_record}; struct line_starts lines = {malloc(256 * sizeof(const char *)), 0, 256}; bool ok = lines.starts != NULL; @@ -1436,19 +1446,18 @@ struct checks { uint64_t lapic_hz; }; -static bool check_failed(struct checks *checks, const char *id, - const char *format, ...) - __attribute__((format(printf, 3, 4))); +static bool check_failed_code(struct checks *checks, const char *id, + const char *code, const char *format, ...) + __attribute__((format(printf, 4, 5))); -// Prints the failure line, emits G_CONFORMANCE_, and powers off with -// status 193. In report-only mode it returns false and the run continues. -static bool check_failed(struct checks *checks, const char *id, - const char *format, ...) +// Prints the failure line, emits CODE, and powers off with status 193. In +// report-only mode it returns false and the run continues. +static bool check_failed_code(struct checks *checks, const char *id, + const char *code, const char *format, ...) { char detail[DETAIL_MAX]; char escaped[DETAIL_MAX]; char line[EVENT_MAX + 64]; - char code[32]; va_list arguments; va_start(arguments, format); @@ -1459,7 +1468,6 @@ static bool check_failed(struct checks *checks, const char *id, "%s: v=1 phase=%s status=fail check=%s detail=\"%s\"\n", abi_prefix(), k_phase_names[checks->phase], id, escaped); console_write(line); - snprintf(code, sizeof(code), "G_CONFORMANCE_%s", id); checks->failures++; emit_event(code, "conformance", checks->phase, detail); if (!g_report_only) @@ -1467,6 +1475,25 @@ static bool check_failed(struct checks *checks, const char *id, return false; } +static bool check_failed(struct checks *checks, const char *id, + const char *format, ...) + __attribute__((format(printf, 3, 4))); + +// check_failed_code() with the conformance code G_CONFORMANCE_. +static bool check_failed(struct checks *checks, const char *id, + const char *format, ...) +{ + char detail[DETAIL_MAX]; + char code[32]; + va_list arguments; + + va_start(arguments, format); + vsnprintf(detail, sizeof(detail), format, arguments); + va_end(arguments); + snprintf(code, sizeof(code), "G_CONFORMANCE_%s", id); + return check_failed_code(checks, id, code, "%s", detail); +} + static int checks_init(struct checks *checks, enum phase phase) { char text[256]; @@ -2774,8 +2801,13 @@ static int cmd_boot(void) } else if (read_boot_log(&log) != 0) { check_failed(&checks, "C4", "read the kernel log: %s", strerror(errno)); - } else if (!boot_log_verdict(&log, detail, sizeof(detail))) { - check_failed(&checks, "C4", "%s", detail); + } else { + if (!boot_log_verdict(&log, detail, sizeof(detail))) + check_failed(&checks, "C4", "%s", detail); + // K1 is kernel hardening, not time: a writable and executable kernel + // mapping, such as unprotected ITS thunk pages. + if (log.wx[0] != '\0') + check_failed_code(&checks, "K1", "G_KERNEL_WX", "%s", log.wx); } check_cpuinfo(&checks, checks.online, true); check_cpus(&checks, checks.online, true, true); @@ -3348,6 +3380,8 @@ static int test_lines(int argc, char **argv, enum line_test mode) printf("ok\n"); else printf("fail %s\n", detail); + if (log.wx[0] != '\0') + printf("wx %s\n", log.wx); } return 0; } diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 2cad54147..130c2d43a 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -463,6 +463,57 @@ def syslog(text_lines: list[str]) -> str: ) self.assertTrue(syslog(lines[2:]).startswith("fail no 'Hypervisor")) + def test_boot_log_reports_a_kernel_wx_mapping_apart_from_c4(self): + # Linux 6.18's CONFIG_DEBUG_WX audit, as logged when the ITS + # mitigation's thunk pages are left writable and executable. + warning = [ + "<4>[ 0.412345] ------------[ cut here ]------------", + "<4>[ 0.412346] x86/mm: Found insecure W+X mapping at address " + "0xffffffffc0000000", + "<4>[ 0.412400] WARNING: CPU: 0 PID: 1 at " + "arch/x86/mm/dump_pagetables.c:246 note_wx+0x5a/0x70", + "<6>[ 0.412500] x86/mm: Checked W+X mappings: FAILED, 4 W+X pages " + "found.", + ] + lines = [f"<6>[ 0.100000] {line}" for line in self.BOOT_LOG] + + def syslog(text_lines: list[str]) -> list[str]: + return ( + self.run_test( + "syslog", + self.fixture("syslog.txt", "\n".join(text_lines) + "\n"), + ) + .strip() + .splitlines() + ) + + self.assertEqual( + syslog([*lines[:2], *warning, *lines[2:]]), + [ + "ok", + "wx x86/mm: Found insecure W+X mapping at address 0xffffffffc0000000", + ], + ) + self.assertEqual( + syslog( + [ + *lines, + "<6>[ 0.412500] x86/mm: Checked W+X mappings: passed, " + "no W+X pages found.", + ] + ), + ["ok"], + ) + failed = syslog([*warning, *lines[1:]]) + self.assertEqual(len(failed), 2) + self.assertTrue(failed[0].startswith("fail no 'Hypervisor")) + self.assertTrue(failed[1].startswith("wx x86/mm: Found insecure W+X")) + own_event = ( + "<2>[ 3.000000] NVX-TIME-REPORT-VIOLATION: v=1 code=G_KERNEL_WX " + 'detail="x86/mm: Found insecure W+X mapping at address 0x0"' + ) + self.assertEqual(syslog([*lines, own_event]), ["ok"]) + def timer_list( self, sections: list[tuple[int, str, int, str, int]], From 81ab71beddd4db0d1ea0de0d71d10b07206af1d6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 07:38:32 -0700 Subject: [PATCH 017/188] kernel: keep runtime code read-only with STRICT_MODULE_RWX The time ABI's hypervisor bit makes Linux 6.18 assume ITS vulnerability unless ARCH_CAPABILITIES reports ITS_NO, which MSHV can't expose. Unless Spectre v2 uses retpolines, the ITS mitigation patches indirect branches with thunks in pages from execmem, and only execmem_restore_rox() makes them read-only again. Without CONFIG_MODULES, x86 doesn't select ARCH_HAS_EXECMEM_ROX (it needs STRICT_MODULE_RWX), so that call is a no-op and four thunk pages stay writable and executable: the boot logs "Found insecure W+X mapping at address 0xffffffffc0000000". No 6.18.y stable or mainline change fixes the !MODULES case. Enable MODULES with STRICT_MODULE_RWX, which selects ARCH_HAS_EXECMEM_ROX, and lock module loading down: - no option may be =m; the build fails if one is; - MODPROBE_PATH is empty, so request_module() never starts a helper; - TRIM_UNUSED_KSYMS drops every symbol export, so a module would have nothing to link against. It also keeps out about 200 KB of export tables that would push the kernel's data past the next 2 MiB boundary (1.8 MiB more memory touched at boot); - init writes 1 to /proc/sys/kernel/modules_disabled right after mounting the pseudo-filesystems, and fails closed if it can't. The build and the release source check require these settings, along with MITIGATION_ITS and DEBUG_WX. The mitigation itself is unchanged. The loaded image grows by about 16 KB. Where the mitigation patches indirect branches, its thunks now come from a 2 MiB read-only executable cache block. Validated with today's OpenVMM on KVM (Skylake on prometheus32, also with indirect_target_selection=force; Ice Lake on azure-kvm-5) and on the MSHV spike (prometheus30). In each case the mitigation stays active, the W+X audit passes, and insmod fails with EPERM. Cold start is unchanged within noise: the paired median is +3.5 ms on prometheus32 and -0.7 ms on azure-kvm-5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 20 ++++++++++++++++ doc/guests/ubuntu-guest.md | 2 +- guest/common/init | 11 +++++++++ kernel/config-microvm | 16 ++++++++++++- scripts/nvx_tools/build.py | 19 ++++++++++++++++ scripts/nvx_tools/build_constants.py | 15 ++++++++++++ scripts/nvx_tools/release.py | 1 + scripts/test_nvx_tools.py | 34 ++++++++++++++++++++++++++++ 8 files changed, 116 insertions(+), 2 deletions(-) diff --git a/doc/build.md b/doc/build.md index cdebf6579..41bd6b3b4 100644 --- a/doc/build.md +++ b/doc/build.md @@ -186,6 +186,26 @@ the soft-lockup or hung-task detectors. Changing an archive hash or patch invalidates both source and object caches; changing the input configuration invalidates the object cache. +The kernel keeps the code it generates at run time read-only. On x86, only +`CONFIG_STRICT_MODULE_RWX` makes such memory read-only and executable +(`CONFIG_ARCH_HAS_EXECMEM_ROX`), and it requires `CONFIG_MODULES`. Without it, +the thunk pages of the Indirect Target Selection (ITS) mitigation stay writable +and executable when the mitigation patches indirect branches at boot, which it +does unless Spectre v2 uses retpolines. The kernel therefore supports modules +but never builds or ships one: + +- the build fails if any option is set to `m`; +- `CONFIG_MODPROBE_PATH` is empty, so the kernel never starts a module helper; +- `CONFIG_TRIM_UNUSED_KSYMS` drops every symbol export, about 200 KB that would + otherwise push the kernel's data past the next 2 MiB boundary; +- init writes 1 to `/proc/sys/kernel/modules_disabled` before anything else + runs. + +When the mitigation patches indirect branches, the read-only executable cache +takes one 2 MiB block of guest memory. `CONFIG_DEBUG_WX` audits the kernel page +tables at boot, and the `nvx-time` boot check fails if the audit reports a +writable and executable mapping. + ## CI debug kernel CI proves that snapshot restores never trip the kernel watchdogs with a debug diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 68a38ecf7..6d8821f21 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -116,7 +116,7 @@ ext4/EROFS root: - the xe9 and shared-status patches required by the microVM ABI. The kernel intentionally omits functionality found in a general Ubuntu kernel, -including loadable modules, PCI, IPv6, user namespaces, fanotify, and +including module loading, PCI, IPv6, user namespaces, fanotify, and SquashFS. Ubuntu support therefore means Ubuntu userland under the NVX kernel policy, not compatibility with every Ubuntu workload or host-integration feature. diff --git a/guest/common/init b/guest/common/init index f01157053..294646f43 100644 --- a/guest/common/init +++ b/guest/common/init @@ -4,6 +4,17 @@ mount -t proc none /proc mount -t sysfs none /sys mount -t devtmpfs none /dev 2>/dev/null || mdev -s +# The kernel supports modules only so that STRICT_MODULE_RWX keeps runtime +# code, such as the ITS mitigation thunks, read-only. No module is shipped, so +# loading is switched off for good before any other process runs. +if [ -e /proc/sys/kernel/modules_disabled ] && + ! echo 1 >/proc/sys/kernel/modules_disabled; then + echo "NVX-INIT-ERROR: cannot disable kernel module loading" >&2 + /sbin/nvx-exit 1 + while :; do + sleep 3600 + done +fi mount -t tmpfs none /tmp export PATH=/usr/sbin:/usr/bin:/sbin:/bin HOME=/root TERM=linux ifconfig lo 127.0.0.1 up diff --git a/kernel/config-microvm b/kernel/config-microvm index 9c5debed6..6fdbbfa5b 100644 --- a/kernel/config-microvm +++ b/kernel/config-microvm @@ -689,6 +689,7 @@ CONFIG_RANDOMIZE_KSTACK_OFFSET=y CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y CONFIG_STRICT_KERNEL_RWX=y CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y +CONFIG_STRICT_MODULE_RWX=y CONFIG_HAVE_ARCH_PREL32_RELOCATIONS=y # CONFIG_LOCK_EVENT_COUNTS is not set CONFIG_ARCH_HAS_MEM_ENCRYPT=y @@ -725,7 +726,19 @@ CONFIG_ARCH_HAS_CPU_ATTACK_VECTORS=y # end of General architecture-dependent options CONFIG_RT_MUTEXES=y -# CONFIG_MODULES is not set +CONFIG_MODULES=y +# CONFIG_MODULE_DEBUG is not set +# CONFIG_MODULE_FORCE_LOAD is not set +# CONFIG_MODULE_UNLOAD is not set +# CONFIG_MODVERSIONS is not set +# CONFIG_MODULE_SRCVERSION_ALL is not set +# CONFIG_MODULE_SIG is not set +# CONFIG_MODULE_COMPRESS is not set +# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set +CONFIG_MODPROBE_PATH="" +CONFIG_TRIM_UNUSED_KSYMS=y +CONFIG_UNUSED_KSYMS_WHITELIST="" +CONFIG_MODULES_TREE_LOOKUP=y CONFIG_BLOCK=y CONFIG_BLOCK_LEGACY_AUTOLOAD=y CONFIG_BLK_RQ_ALLOC_TIME=y @@ -871,6 +884,7 @@ CONFIG_ARCH_SUPPORTS_PER_VMA_LOCK=y CONFIG_PER_VMA_LOCK=y CONFIG_LOCK_MM_AND_FIND_VMA=y CONFIG_EXECMEM=y +CONFIG_ARCH_HAS_EXECMEM_ROX=y CONFIG_ARCH_SUPPORTS_PT_RECLAIM=y CONFIG_PT_RECLAIM=y diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 9f0b51568..9063505c7 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -127,6 +127,24 @@ def _assert_time_abi_kernel_config(path: Path) -> None: ) +def _assert_hardening_kernel_config(path: Path) -> None: + _assert_kernel_config( + path, + KernelBuildConstants.REQUIRED_HARDENING_CONFIG, + "kernel configuration does not keep runtime code read-only: ", + ) + modules = [ + line + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith("CONFIG_") and line.endswith("=m") + ] + if modules: + raise ScriptError( + "kernel configuration builds loadable modules, which are never " + "shipped: " + ", ".join(modules) + ) + + def _assert_watchdog_kernel_config(path: Path, *, debug: bool) -> None: if debug: _assert_kernel_config( @@ -155,6 +173,7 @@ def assert_required_kernel_config(path: Path, *, debug: bool = False) -> None: _assert_sandbox_kernel_config(path) _assert_shared_status_kernel_config(path) _assert_time_abi_kernel_config(path) + _assert_hardening_kernel_config(path) _assert_watchdog_kernel_config(path, debug=debug) diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 5d6a99ab4..0d5168561 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -85,6 +85,21 @@ class KernelBuildConstants: "# CONFIG_HYPERV is not set", "# CONFIG_CPU_FREQ is not set", ) + # x86 makes runtime-generated code, such as the ITS mitigation thunks, + # read-only only through STRICT_MODULE_RWX (ARCH_HAS_EXECMEM_ROX), which + # needs MODULES. No module is ever built or shipped: there is no autoload + # helper, no symbol is exported, and init disables module loading before + # anything else runs. + REQUIRED_HARDENING_CONFIG: Final = ( + "CONFIG_STRICT_KERNEL_RWX=y", + "CONFIG_MODULES=y", + "CONFIG_STRICT_MODULE_RWX=y", + "CONFIG_ARCH_HAS_EXECMEM_ROX=y", + 'CONFIG_MODPROBE_PATH=""', + "CONFIG_TRIM_UNUSED_KSYMS=y", + "CONFIG_MITIGATION_ITS=y", + "CONFIG_DEBUG_WX=y", + ) DEBUG_WATCHDOG_CONFIG: Final = ( "CONFIG_SOFTLOCKUP_DETECTOR=y", "CONFIG_DETECT_HUNG_TASK=y", diff --git a/scripts/nvx_tools/release.py b/scripts/nvx_tools/release.py index b26c8e3e4..96a68cb93 100644 --- a/scripts/nvx_tools/release.py +++ b/scripts/nvx_tools/release.py @@ -1650,6 +1650,7 @@ def verify_source_tree() -> None: "CONFIG_FUSE_FS=y", *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + *KernelBuildConstants.REQUIRED_HARDENING_CONFIG, ): if setting not in config.splitlines(): raise ScriptError(f"{config_path} is missing {setting}") diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 5ac50e6db..a087ab8fc 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -279,6 +279,7 @@ def _write_release_fixture( *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + *KernelBuildConstants.REQUIRED_HARDENING_CONFIG, ) ) + "\n", @@ -4643,6 +4644,7 @@ def test_kernel_build_rejects_input_config_mutation_during_build(self): *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + *KernelBuildConstants.REQUIRED_HARDENING_CONFIG, ) ) + "\n", @@ -6050,6 +6052,7 @@ def test_shared_status_kernel_config_is_required(self): def test_checked_in_config_meets_time_abi_kernel_contract(self): config = BuildConstants.REPO_ROOT / KernelBuildConstants.INPUT_CONFIG build._assert_time_abi_kernel_config(config) + build._assert_hardening_kernel_config(config) build._assert_watchdog_kernel_config(config, debug=False) configured = config.read_text(encoding="utf-8").splitlines() for removed in ( @@ -6081,6 +6084,36 @@ def test_time_abi_kernel_config_requires_every_setting(self): with self.assertRaisesRegex(common.ScriptError, missing): build._assert_time_abi_kernel_config(config) + def test_hardening_kernel_config_requires_every_setting(self): + with tempfile.TemporaryDirectory() as temporary: + config = Path(temporary) / ".config" + for missing in KernelBuildConstants.REQUIRED_HARDENING_CONFIG: + with self.subTest(missing=missing): + config.write_text( + "\n".join( + setting + for setting in KernelBuildConstants.REQUIRED_HARDENING_CONFIG + if setting != missing + ) + + "\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(common.ScriptError, re.escape(missing)): + build._assert_hardening_kernel_config(config) + + def test_hardening_kernel_config_rejects_loadable_modules(self): + with tempfile.TemporaryDirectory() as temporary: + config = Path(temporary) / ".config" + config.write_text( + "\n".join( + (*KernelBuildConstants.REQUIRED_HARDENING_CONFIG, "CONFIG_FOO=m") + ) + + "\n", + encoding="utf-8", + ) + with self.assertRaisesRegex(common.ScriptError, "CONFIG_FOO=m"): + build._assert_hardening_kernel_config(config) + def test_watchdog_kernel_config_separates_production_and_debug(self): with tempfile.TemporaryDirectory() as temporary: config = Path(temporary) / ".config" @@ -6180,6 +6213,7 @@ def test_debug_kernel_build_applies_fragment_and_records_provenance(self): *KernelBuildConstants.REQUIRED_SHARED_STATUS_CONFIG, *KernelBuildConstants.REQUIRED_SANDBOX_CONFIG, *KernelBuildConstants.REQUIRED_TIME_ABI_CONFIG, + *KernelBuildConstants.REQUIRED_HARDENING_CONFIG, "# CONFIG_DEBUG_KERNEL is not set", ) ) From 0d25471b8dd6a985b97d3d614e2cee149dd77770 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:22:16 -0700 Subject: [PATCH 018/188] ci: classify time ABI guest failures in the microVM harness Under time ABI v1 the guest reports a failed conformance check or a runtime clock violation itself: it prints an NVX-TIME-ABI or NVX-TIME-ABI-VIOLATION line and powers off with status 193, 194, or 195. The harness saw only a marker timeout or "OpenVMM exited with status 194", so a clock failure looked like any other crash. A TimeAbiMonitor now scans every launch's console output, in the benchmark runners, OpenvmmProcess, and TCP consoles. A violation event or a failed check fails the scenario at once with the guest's code and detail, and an exit status of 193, 194, or 195 is named as a conformance, runtime-violation, or restore-repair power-off together with the event that preceded it. Output that a process left in the reader queue is drained before its exit status is classified, because the guest writes its event just before it powers off. The new nvx_tools.time_abi module also parses the boot and restore markers, validates the boot marker against the backend's LAPIC rate, checks warp probe summaries against the 1 us bound, and names the CPU generations, for the scenarios and host qualification that follow. Today's guests print none of these lines and never use these statuses, so nothing changes until the guest implements the time ABI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-nvx/action.yml | 4 +- doc/ci.md | 7 + scripts/nvx_tools/benchmark.py | 84 ++++- scripts/nvx_tools/microvm_tests.py | 3 + scripts/nvx_tools/openvmm_process.py | 90 ++++- scripts/nvx_tools/time_abi.py | 379 +++++++++++++++++++ scripts/test_time_abi.py | 466 ++++++++++++++++++++++++ 7 files changed, 1012 insertions(+), 21 deletions(-) create mode 100644 scripts/nvx_tools/time_abi.py create mode 100644 scripts/test_time_abi.py diff --git a/.github/actions/validate-nvx/action.yml b/.github/actions/validate-nvx/action.yml index 24f580f5b..e453afe2e 100644 --- a/.github/actions/validate-nvx/action.yml +++ b/.github/actions/validate-nvx/action.yml @@ -10,7 +10,7 @@ runs: run: | set -euo pipefail python3 -m compileall -q scripts - python3 -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py -v + python3 -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py scripts/test_time_abi.py -v python3 scripts/test_adversarial.py -v python3 .github/skills/nvx-host-connect/scripts/test_hosts.py -v python3 scripts/nvx.py --help @@ -28,7 +28,7 @@ runs: $ErrorActionPreference = "Stop" python -m compileall -q scripts if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - python -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py -v + python -m unittest scripts/test_performance.py scripts/test_nvx_tools.py scripts/test_microvm_tests.py scripts/test_development_release.py scripts/test_egress_policy.py scripts/test_control_session.py scripts/test_managed_agent.py scripts/test_guest_time.py scripts/test_time_abi.py -v if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\test_adversarial.py -v if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/doc/ci.md b/doc/ci.md index ff7fe74bf..4f7206181 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -22,6 +22,13 @@ the NVX Linux kernel plus the selected Alpine or Ubuntu initramfs and exercises Linux, SMP, virtio, sandbox, and snapshot behavior through the public OpenVMM CLI. Alpine-control-only scenarios remain explicit and are rejected for the Ubuntu initramfs. Failure logs from the NVX layer are uploaded per backend. +Every harness launch, in the tests and the benchmarks, scans the OpenVMM +console for the guest's [time ABI](design/time-abi.md) output. An +`NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line +fails the scenario at once with the guest's code and detail. An OpenVMM exit +status of 193, 194, or 195 is reported as the guest's time ABI conformance, +runtime-violation, or restore-repair power-off, together with the event that +preceded it, instead of as a generic exit status. The restore-processor scenario also rejects Linux TSC instability diagnostics, even if the requested CPUs came online, so clock skew cannot silently pass by falling back to a different clocksource. After the 1/2/4/8-CPU restores, it diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index 184be1c6f..64166fc8e 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -42,6 +42,7 @@ OpenVMMBuildConstants, ) from .common import bytes_to_mib, sha256_file +from .time_abi import TimeAbiMonitor BOOT_MARKER = b"ALPINE-MICROVM-BOOT-OK" RESTORE_MARKER = b"OPENVMM-SNAPSHOT-RESTORE-OK" @@ -1252,6 +1253,41 @@ def wait_for_process_exit(process: subprocess.Popen[bytes], timeout: float) -> i return process.wait(timeout=timeout) +def drain_exited_output( + output: SeparatedOutput, + monitor: TimeAbiMonitor, + timeout: float = OUTPUT_DRAIN_TIMEOUT_SECONDS, +) -> None: + """Consume the output an exited OpenVMM process left in the reader queues. + + A time ABI power-off writes its violation event to the guest console just + before the exit, so the event must reach the monitor before the exit + status is classified. + """ + deadline = time.monotonic() + timeout + while not output.closed: + try: + stream, chunk = output.get(timeout=max(0.0, deadline - time.monotonic())) + except queue.Empty: + return + if stream != "console": + continue + if chunk is None: + monitor.finish() + else: + monitor.feed(chunk) + + +def exit_status_after_eof( + process: subprocess.Popen[bytes], timeout: float = 5.0 +) -> int | None: + """Return the exit status of a process whose output reached EOF.""" + try: + return process.wait(timeout=timeout) + except subprocess.TimeoutExpired: + return None + + def contains_output_line(output: bytes | bytearray, marker: bytes) -> bool: return any(line.removesuffix(b"\r") == marker for line in output.split(b"\n")) @@ -1745,15 +1781,17 @@ def measure_once( Peak RSS is None when OpenVMM exited before it could be sampled at the marker, which a prequeued guest exit makes possible. A completed output - line starting with ``failure_marker`` stops the launch immediately. Both - markers match only the guest console, and profile records come only from - OpenVMM's stderr. A launch that keeps a profile or a log reads both - streams to their end after teardown, within ``timeout``, and fails if they - do not end. + line starting with ``failure_marker`` stops the launch immediately, and so + does a time ABI violation event or failed conformance check. Both markers + and the time ABI lines match only the guest console, and profile records + come only from OpenVMM's stderr. A launch that keeps a profile or a log + reads both streams to their end after teardown, within ``timeout``, and + fails if they do not end. """ started = time.perf_counter_ns() interaction = InteractiveProcess(command, environment, separate_stderr=True) process = interaction.process + monitor = TimeAbiMonitor(command) profile = ( SnapshotProfileCollector(process.pid, started) if snapshot_profile else None ) @@ -1791,6 +1829,8 @@ def finish_output( stream, chunk = output.get(timeout=min(remaining, 0.25)) except queue.Empty: if process.poll() is not None: + drain_exited_output(output, monitor) + monitor.check_exit(process.returncode) raise RuntimeError( f"OpenVMM exited with status {process.returncode}" ) from None @@ -1800,7 +1840,11 @@ def finish_output( profile.feed(chunk) continue if chunk is None: - raise RuntimeError(f"OpenVMM exited with status {process.poll()}") + monitor.finish() + returncode = exit_status_after_eof(process) + monitor.check_exit(returncode) + raise RuntimeError(f"OpenVMM exited with status {returncode}") + monitor.feed(chunk) console = output.console if failure_marker is not None: failure = completed_output_line_with_prefix(console, failure_marker) @@ -1854,11 +1898,15 @@ def finish_output( process_exited = time.perf_counter_ns() teardown_ms = (process_exited - teardown_started) / 1_000_000 wall_ms = (process_exited - started) / 1_000_000 - finish_output(marker_reached, readiness_counters) if teardown_mode == "guest-exit" and returncode != 0: + # The monitor needs the console's last lines before + # finish_output() would drain them. + drain_exited_output(output, monitor) + monitor.check_exit(returncode) raise RuntimeError( f"OpenVMM exited with status {returncode} during teardown" ) + finish_output(marker_reached, readiness_counters) return elapsed_ms, peak_bytes, teardown_ms, wall_ms except GuestFailureReported: raise @@ -2420,6 +2468,7 @@ def run_guest_script( contain_process_tree=contain_process_tree, ) process = interaction.process + monitor = TimeAbiMonitor(command) if windows_cpus is not None: set_windows_affinity(process.pid, windows_cpus) @@ -2452,6 +2501,7 @@ def run_guest_script( if chunk is None: break output.extend(chunk) + monitor.feed(chunk) peak_bytes = _try_peak_rss(process, peak_bytes) if not input_sent and boot_marker in output: interaction.write_input(script.encode("utf-8")) @@ -2465,7 +2515,9 @@ def run_guest_script( time.monotonic() + TEARDOWN_TIMEOUT_SECONDS, ) + monitor.finish() returncode = process.wait() + monitor.check_exit(returncode) if teardown_mode == "guest-exit" and returncode != 0: raise RuntimeError(f"OpenVMM exited with status {returncode}") if not input_sent: @@ -2528,6 +2580,7 @@ def capture_automatic_snapshot( environment["OPENVMM_LOG"] = "off" interaction = InteractiveProcess(command, environment) process = interaction.process + monitor = TimeAbiMonitor(command) if windows_cpus is not None: set_windows_affinity(process.pid, windows_cpus) @@ -2549,8 +2602,11 @@ def capture_automatic_snapshot( if chunk is None: break output.extend(chunk) + monitor.feed(chunk) + monitor.finish() returncode = process.wait() + monitor.check_exit(returncode) if returncode != 0: raise RuntimeError(f"snapshot source exited with status {returncode}") for marker in required_markers: @@ -2612,6 +2668,7 @@ def capture_device_restore_snapshot( environment["OPENVMM_LOG"] = "off" interaction = InteractiveProcess(command, environment) process = interaction.process + monitor = TimeAbiMonitor(command) if windows_cpus is not None: set_windows_affinity(process.pid, windows_cpus) chunks: queue.Queue[bytes | None] = queue.Queue() @@ -2636,7 +2693,10 @@ def capture_device_restore_snapshot( if chunk is None: break output.extend(chunk) + monitor.feed(chunk) + monitor.finish() returncode = process.wait() + monitor.check_exit(returncode) if returncode != 0: raise RuntimeError( f"{device}/{mode} snapshot source exited with status {returncode}" @@ -2678,6 +2738,7 @@ def run_device_restore_sample( started_ns = time.perf_counter_ns() interaction = InteractiveProcess(command, environment) process = interaction.process + monitor = TimeAbiMonitor(command) if windows_cpus is not None: set_windows_affinity(process.pid, windows_cpus) chunks: queue.Queue[bytes | None] = queue.Queue() @@ -2746,11 +2807,14 @@ def observe_chunk(chunk: bytes, observed_ns: int) -> None: break observed_ns = time.perf_counter_ns() output.extend(chunk) + monitor.feed(chunk) observe_chunk(chunk, observed_ns) if pending: observe_line(bytes(pending), time.perf_counter_ns()) pending.clear() + monitor.finish() returncode = process.wait() + monitor.check_exit(returncode) if returncode != 0: raise RuntimeError( f"{device}/{mode} restore exited with status {returncode}" @@ -4667,6 +4731,7 @@ def capture_snapshot( process_started_ns = time.perf_counter_ns() interaction = InteractiveProcess(command, environment, separate_stderr=True) process = interaction.process + monitor = TimeAbiMonitor(command) profile = SnapshotProfileCollector( process.pid, process_started_ns, @@ -4723,6 +4788,8 @@ def request_snapshot() -> None: peak_bytes = _try_peak_rss(process, peak_bytes) observe_snapshot_publication() if process.poll() is not None and snapshot_published_ns is None: + drain_exited_output(output, monitor) + monitor.check_exit(process.returncode) raise RuntimeError( f"OpenVMM exited with status {process.returncode}" ) from None @@ -4736,6 +4803,7 @@ def request_snapshot() -> None: if stream == "stderr": profile.feed(chunk) continue + monitor.feed(chunk) console = output.console if ( snapshot_requested @@ -4767,8 +4835,10 @@ def request_snapshot() -> None: if snapshot_requested and contains_output_line(console, RESTORE_MARKER): raise RuntimeError("source guest continued past the snapshot boundary") + monitor.finish() returncode = process.wait() source_exited_ns = time.perf_counter_ns() + monitor.check_exit(returncode) if snapshot_published_ns is None and snapshot_path.is_dir(): snapshot_published_ns = source_exited_ns if not snapshot_requested: diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index c8e199c8a..5f9478dcb 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -59,6 +59,7 @@ from .guests import GUEST_NAMES, GuestDescriptor, guest_descriptor from .managed_exec_tests import run_managed_exec_configuration from .openvmm_process import OpenvmmProcess, TcpConsole +from .time_abi import describe_exit_status MICROVM_TEST_SCENARIOS = ( "console-exit", @@ -962,8 +963,10 @@ def run_managed_lifecycle( ) result = process.wait(timeout=timeout) if result != 0: + reason = describe_exit_status(result) raise RuntimeError( f"managed OpenVMM process exited with status {result}" + + (f": {reason}; see {log_path}" if reason else "") ) report = _read_outcome_report(report_path) _preserve_outcome_report( diff --git a/scripts/nvx_tools/openvmm_process.py b/scripts/nvx_tools/openvmm_process.py index a5e41ef27..2d97d90cf 100644 --- a/scripts/nvx_tools/openvmm_process.py +++ b/scripts/nvx_tools/openvmm_process.py @@ -15,6 +15,7 @@ from .benchmark import InteractiveProcess, terminate from .common import remaining_timeout +from .time_abi import TimeAbiFailure, TimeAbiMonitor def _line_marker_end( @@ -84,6 +85,7 @@ def __init__( self._search_offset = 0 self._log_path = log_path self._finished = False + self._monitor = TimeAbiMonitor(command) @property def process(self): @@ -93,12 +95,58 @@ def process(self): def output(self) -> bytes: return bytes(self._output) + @property + def time_abi(self) -> TimeAbiMonitor: + return self._monitor + def send_bytes(self, data: bytes) -> None: self._interaction.write_input(data) def send_line(self, line: str) -> None: self.send_bytes(f"{line}\n".encode()) + def _consume(self, chunk: bytes) -> None: + self._output.extend(chunk) + try: + self._monitor.feed(chunk) + except TimeAbiFailure as error: + self._fail(error) + + def _exited(self, before: str) -> Exception: + try: + returncode: int | None = self.process.wait(timeout=5) + except subprocess.TimeoutExpired: + returncode = None + try: + self._monitor.finish() + self._monitor.check_exit(returncode) + except TimeAbiFailure as error: + return error + return RuntimeError(f"OpenVMM exited with status {returncode} before {before}") + + def wait_for_time_abi(self, phase: str, timeout: float) -> dict[str, str]: + """Wait for a passing ``NVX-TIME-ABI`` marker of a conformance phase.""" + deadline = time.monotonic() + timeout + while True: + if phase == "boot" and self._monitor.boot is not None: + return self._monitor.boot + if phase == "restore" and self._monitor.restores: + return self._monitor.restores[-1] + remaining = deadline - time.monotonic() + if remaining <= 0: + self._fail( + TimeoutError( + f"time ABI {phase} marker was not observed within {timeout:g}s" + ) + ) + try: + chunk = self._chunks.get(timeout=min(remaining, 0.1)) + except queue.Empty: + continue + if chunk is None: + self._fail(self._exited(f"the time ABI {phase} marker")) + self._consume(chunk) + def wait_for(self, marker: bytes, timeout: float) -> None: if not marker: raise ValueError("OpenVMM process marker cannot be empty") @@ -138,13 +186,9 @@ def _wait_for_marker( continue if chunk is None: self._fail( - RuntimeError( - "OpenVMM exited with status " - f"{self.process.poll()} before " - f"{exit_label + ' ' if exit_label else ''}{marker!r}" - ) + self._exited(f"{exit_label + ' ' if exit_label else ''}{marker!r}") ) - self._output.extend(chunk) + self._consume(chunk) def wait(self, timeout: float) -> OpenvmmProcessResult: deadline = time.monotonic() + timeout @@ -162,12 +206,17 @@ def wait(self, timeout: float) -> OpenvmmProcessResult: continue if chunk is None: break - self._output.extend(chunk) + self._consume(chunk) remaining = remaining_timeout(deadline) try: returncode = self.process.wait(timeout=remaining) except subprocess.TimeoutExpired: self._fail(TimeoutError(f"OpenVMM did not exit within {timeout:g}s")) + try: + self._monitor.finish() + self._monitor.check_exit(returncode) + except TimeAbiFailure as error: + self._fail(error) self._finished = True self._write_log() return OpenvmmProcessResult(returncode, bytes(self._output)) @@ -213,13 +262,25 @@ def __exit__( class TcpConsole: - def __init__(self, connection: socket.socket) -> None: + def __init__( + self, + connection: socket.socket, + monitor: TimeAbiMonitor | None = None, + ) -> None: self._connection = connection self._output = bytearray() self._search_offset = 0 + self._monitor = monitor @classmethod - def connect(cls, address: tuple[str, int], timeout: float) -> TcpConsole: + def connect( + cls, + address: tuple[str, int], + timeout: float, + *, + monitor: TimeAbiMonitor | None = None, + ) -> TcpConsole: + """Connect to a virtio console; ``monitor`` checks its time ABI lines.""" deadline = time.monotonic() + timeout while True: try: @@ -229,13 +290,18 @@ def connect(cls, address: tuple[str, int], timeout: float) -> TcpConsole: except BaseException: connection.close() raise - return cls(connection) + return cls(connection, monitor) except OSError as error: if time.monotonic() >= deadline: raise TimeoutError( f"failed to connect to virtio console at {address}" ) from error + def _consume(self, chunk: bytes) -> None: + self._output.extend(chunk) + if self._monitor is not None: + self._monitor.feed(chunk) + def send_bytes(self, data: bytes) -> None: self._connection.sendall(data) @@ -265,7 +331,7 @@ def wait_for(self, marker: bytes, timeout: float) -> None: continue if not chunk: raise RuntimeError(f"TCP console closed before marker {marker!r}") - self._output.extend(chunk) + self._consume(chunk) def wait_for_line(self, marker: bytes, timeout: float) -> None: if not marker or b"\n" in marker or b"\r" in marker: @@ -288,7 +354,7 @@ def wait_for_line(self, marker: bytes, timeout: float) -> None: continue if not chunk: raise RuntimeError(f"TCP console closed before line marker {marker!r}") - self._output.extend(chunk) + self._consume(chunk) def finish(self) -> bytes: self._connection.settimeout(0.1) diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py new file mode 100644 index 000000000..9e2742f6f --- /dev/null +++ b/scripts/nvx_tools/time_abi.py @@ -0,0 +1,379 @@ +"""Harness checks of the NVX time ABI v1 guest obligations. + +doc/design/time-abi.md defines what a conforming guest prints and how it fails: +the ``NVX-TIME-ABI`` marker of each conformance check, the +``NVX-TIME-ABI-VIOLATION`` event, and the power-off statuses 193 (conformance), +194 (runtime violation), and 195 (restore repair). It also fixes the warp +probe's 1 us skew bound and the CPU generation names used in reports. +""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence + +MARKER_PREFIX = "NVX-TIME-ABI: " +VIOLATION_PREFIX = "NVX-TIME-ABI-VIOLATION: " +REPORT_ONLY_PREFIX = "NVX-TIME-REPORT" +ABI_VERSION = "1" +STATUS_CLASSES: Mapping[int, str] = { + 193: "conformance", + 194: "runtime violation", + 195: "restore repair", +} +LAPIC_HZ: Mapping[str, int] = { + "kvm": 1_000_000_000, + "mshv": 200_000_000, + "whp": 200_000_000, +} +MIN_TSC_HZ = 500_000_000 +MAX_TSC_HZ = 10_000_000_000 +WARP_BOUND_NS = 1000 +WARP_PROBE_PATH = "/sbin/nvx-time-probe" +WARP_SUMMARY_PREFIX = "NVX-TIME-PROBE warp " +WARP_DETAIL_PREFIX = "NVX-TIME-PROBE warp-detail " +CPU_GENERATIONS: Mapping[tuple[str, int, int], str] = { + ("GenuineIntel", 6, 85): "skylake-sp", + ("GenuineIntel", 6, 106): "icelake-sp", + ("GenuineIntel", 6, 207): "emeraldrapids", +} +# Guest boot markers that init prints after the time ABI boot check. +GUEST_BOOT_MARKERS = ("ALPINE-MICROVM-BOOT-OK", "NVX-GUEST-BOOT-OK:") +_MAX_PENDING_LINE = 64 * 1024 + + +class TimeAbiFailure(RuntimeError): + """Raised when a guest or its console output violates the time ABI.""" + + +def cpu_generation(vendor: str, family: int, model: int) -> str | None: + """Return the time ABI generation name of a CPU, or None if unknown.""" + return CPU_GENERATIONS.get((vendor, family, model)) + + +def parse_fields(text: str) -> dict[str, str]: + """Parse space-separated ``key=value`` fields; values may be quoted. + + Quoted values use the guest's escapes: ``\\"``, ``\\\\``, and ``\\xNN``. + """ + fields: dict[str, str] = {} + index = 0 + length = len(text) + while index < length: + if text[index] == " ": + index += 1 + continue + separator = text.find("=", index) + if separator < 0: + raise ValueError(f"field without a value: {text[index:]!r}") + key = text[index:separator] + if not key or " " in key or '"' in key: + raise ValueError(f"invalid field name {key!r}") + index = separator + 1 + if index < length and text[index] == '"': + index += 1 + value: list[str] = [] + while True: + if index >= length: + raise ValueError(f"unterminated quoted value for {key!r}") + character = text[index] + if character == '"': + index += 1 + break + if character != "\\": + value.append(character) + index += 1 + continue + escape = text[index + 1 : index + 2] + if escape in ('"', "\\"): + value.append(escape) + index += 2 + elif escape == "x": + digits = text[index + 2 : index + 4] + if len(digits) != 2: + raise ValueError(f"truncated escape in {key!r}") + value.append(chr(int(digits, 16))) + index += 4 + else: + raise ValueError(f"invalid escape in {key!r}") + parsed = "".join(value) + else: + end = text.find(" ", index) + if end < 0: + end = length + parsed = text[index:end] + index = end + if key in fields: + raise ValueError(f"duplicate field {key!r}") + fields[key] = parsed + return fields + + +def _clean_line(line: str) -> str: + return line.removesuffix("\r").removesuffix("\n").removesuffix("\r") + + +def parse_marker(line: str) -> dict[str, str] | None: + """Parse an ``NVX-TIME-ABI:`` marker line, or return None for other lines.""" + line = _clean_line(line) + if not line.startswith(MARKER_PREFIX): + return None + fields = parse_fields(line.removeprefix(MARKER_PREFIX)) + for name in ("v", "phase", "status"): + if name not in fields: + raise ValueError(f"time ABI marker lacks {name!r}: {line!r}") + return fields + + +def parse_violation(line: str) -> dict[str, str] | None: + """Parse an ``NVX-TIME-ABI-VIOLATION:`` event anywhere in a console line.""" + line = _clean_line(line) + start = line.find(VIOLATION_PREFIX) + if start < 0: + return None + fields = parse_fields(line[start + len(VIOLATION_PREFIX) :]) + if "code" not in fields: + raise ValueError(f"time ABI violation lacks a code: {line!r}") + return fields + + +def describe_exit_status(returncode: int | None) -> str | None: + """Describe an OpenVMM exit status that a time ABI guest failure produced.""" + if returncode is None or returncode not in STATUS_CLASSES: + return None + return f"guest powered off with time ABI status {returncode} ({STATUS_CLASSES[returncode]})" + + +def _command_option(command: Sequence[str], option: str) -> str | None: + for index, argument in enumerate(command[:-1]): + if argument == option: + return command[index + 1] + return None + + +def is_cold_boot_command(command: Sequence[str]) -> bool: + """Whether an OpenVMM command boots a kernel rather than restoring.""" + return "--kernel" in command and "--restore-snapshot" not in command + + +def validate_boot_marker( + fields: Mapping[str, str], + *, + backend: str | None, + online_cpus: int | None = None, +) -> None: + """Check the boot marker's fields against the backend's declared rates.""" + problems: list[str] = [] + if fields.get("v") != ABI_VERSION: + problems.append(f"version {fields.get('v')!r} is not {ABI_VERSION}") + if fields.get("phase") != "boot" or fields.get("status") != "ok": + problems.append("it is not a passing boot check") + try: + tsc_hz = int(fields.get("tsc_hz", "")) + if not MIN_TSC_HZ <= tsc_hz <= MAX_TSC_HZ: + problems.append(f"tsc_hz={tsc_hz} is outside 500 MHz to 10 GHz") + except ValueError: + problems.append(f"tsc_hz={fields.get('tsc_hz')!r} is not an integer") + lapic = fields.get("lapic_hz") + expected_lapic = LAPIC_HZ.get(backend) if backend is not None else None + if expected_lapic is not None and lapic != str(expected_lapic): + problems.append(f"lapic_hz={lapic} is not the {backend} rate {expected_lapic}") + if fields.get("generation") != "0": + problems.append(f"generation={fields.get('generation')} is not 0 at cold boot") + if online_cpus is not None and fields.get("cpus") != str(online_cpus): + problems.append(f"cpus={fields.get('cpus')} is not {online_cpus}") + if problems: + raise TimeAbiFailure( + "guest time ABI boot marker is invalid: " + "; ".join(problems) + ) + + +class TimeAbiMonitor: + """Scan OpenVMM console output for time ABI markers and violations. + + ``feed`` raises TimeAbiFailure as soon as a completed line carries a + violation event or a failed conformance check, so a scenario fails fast + with the guest's own explanation instead of a marker timeout. + """ + + def __init__(self, command: Sequence[str] = ()) -> None: + self.backend = _command_option(command, "--hypervisor") + self.cold_boot = is_cold_boot_command(command) + self.boot: dict[str, str] | None = None + self.restores: list[dict[str, str]] = [] + self.uncertain: list[dict[str, str]] = [] + self.violation: str | None = None + self.report_only = False + self.guest_booted = False + self._pending = bytearray() + + def feed(self, chunk: bytes | bytearray) -> None: + self._pending.extend(chunk) + while True: + newline = self._pending.find(b"\n") + if newline < 0: + break + line = bytes(self._pending[:newline]) + del self._pending[: newline + 1] + self._line(line.decode("utf-8", "replace")) + if len(self._pending) > _MAX_PENDING_LINE: + del self._pending[: len(self._pending) - 4096] + + def finish(self) -> None: + """Scan an unterminated final line once the output reached EOF.""" + if self._pending: + line = bytes(self._pending) + self._pending.clear() + self._line(line.decode("utf-8", "replace")) + + def _line(self, line: str) -> None: + line = _clean_line(line) + if VIOLATION_PREFIX in line: + try: + event = parse_violation(line) + except ValueError: + event = None + self.violation = line[line.find(VIOLATION_PREFIX) :] + code = event["code"] if event is not None else "unparsed" + raise TimeAbiFailure( + f"guest reported time ABI violation {code}: {self.violation}" + ) + if line.startswith(REPORT_ONLY_PREFIX): + self.report_only = True + return + if any(line.startswith(marker) for marker in GUEST_BOOT_MARKERS) or any( + f" {marker}" in line for marker in GUEST_BOOT_MARKERS + ): + self.guest_booted = True + if not line.startswith(MARKER_PREFIX): + return + try: + fields = parse_marker(line) + except ValueError as error: + raise TimeAbiFailure(f"malformed time ABI marker: {error}") from error + assert fields is not None + status = fields["status"] + if status == "fail": + raise TimeAbiFailure( + f"guest time ABI {fields['phase']} check {fields.get('check', '?')} " + f"failed: {fields.get('detail', line)}" + ) + if status == "uncertain": + self.uncertain.append(fields) + elif status != "ok": + raise TimeAbiFailure(f"unknown time ABI marker status: {line}") + elif fields["phase"] == "boot": + if self.boot is None: + self.boot = fields + elif fields["phase"] == "restore": + self.restores.append(fields) + + def check_exit(self, returncode: int | None) -> None: + """Fail on an OpenVMM exit status produced by a time ABI power-off.""" + description = describe_exit_status(returncode) + if description is None: + return + event = self.violation or "no NVX-TIME-ABI-VIOLATION event was observed" + raise TimeAbiFailure(f"{description}: {event}") + + def require_boot( + self, + context: str, + *, + online_cpus: int | None = None, + ) -> dict[str, str]: + """Return the validated boot marker of a cold boot, or fail clearly.""" + if self.boot is None: + reason = ( + "the guest ran nvx-time in report-only mode" + if self.report_only + else "the guest image or OpenVMM does not implement time ABI v1" + ) + raise TimeAbiFailure( + f"guest did not print the NVX-TIME-ABI boot marker before {context}; " + f"{reason}" + ) + validate_boot_marker( + self.boot, + backend=self.backend, + online_cpus=online_cpus, + ) + return self.boot + + def require_boot_if_booted(self) -> None: + """Require the boot marker once a cold-booted guest reached its shell.""" + if self.cold_boot and self.guest_booted: + self.require_boot("the guest boot marker") + + +def warp_probe_command(*, cpus: str | None = None) -> str: + """Return the guest command line that runs the warp probe.""" + command = f"{WARP_PROBE_PATH} warp --bound-ns {WARP_BOUND_NS}" + return command if cpus is None else f"{command} --cpus {cpus}" + + +def check_warp_probe( + text: str, + *, + cpus: int, + context: str, +) -> list[dict[str, str]]: + """Validate every warp probe summary in console output. + + The spec bounds both the backward step and the ping-pong offset by 1 us + and forbids stalled pairs; an inconclusive measurement proves nothing. + """ + summaries: list[dict[str, str]] = [] + details: list[dict[str, str]] = [] + for raw in text.splitlines(): + line = _clean_line(raw) + try: + if line.startswith(WARP_DETAIL_PREFIX): + details.append(parse_fields(line.removeprefix(WARP_DETAIL_PREFIX))) + elif line.startswith(WARP_SUMMARY_PREFIX): + summaries.append(parse_fields(line.removeprefix(WARP_SUMMARY_PREFIX))) + except ValueError as error: + raise TimeAbiFailure( + f"{context}: malformed warp probe output: {error}" + ) from error + if not summaries: + raise TimeAbiFailure(f"{context}: the guest warp probe printed no summary") + if len(details) != len(summaries): + raise TimeAbiFailure( + f"{context}: the guest warp probe printed {len(summaries)} summaries " + f"and {len(details)} detail lines" + ) + expected_pairs = cpus * (cpus - 1) // 2 + results: list[dict[str, str]] = [] + for summary, detail in zip(summaries, details, strict=True): + merged = {**summary, **detail} + problems: list[str] = [] + try: + backward = int(merged["max_backward_ns"]) + offset = int(merged["max_abs_offset_ns"]) + pairs = int(merged["pairs"]) + stalled = int(merged["stalled_pairs"]) + except (KeyError, ValueError) as error: + raise TimeAbiFailure( + f"{context}: incomplete warp probe output: {error}" + ) from error + if pairs != expected_pairs: + problems.append(f"measured {pairs} CPU pairs instead of {expected_pairs}") + if backward > WARP_BOUND_NS: + problems.append(f"max_backward_ns={backward} exceeds {WARP_BOUND_NS}") + if offset > WARP_BOUND_NS: + problems.append(f"max_abs_offset_ns={offset} exceeds {WARP_BOUND_NS}") + if stalled: + problems.append(f"{stalled} CPU pair(s) stalled") + if merged.get("conclusive") != "1": + problems.append( + "the measurement is inconclusive " + f"(max_uncertainty_ns={merged.get('max_uncertainty_ns')})" + ) + if merged.get("verdict") != "PASS": + problems.append(f"verdict={merged.get('verdict')}") + if problems: + raise TimeAbiFailure( + f"{context}: cross-vCPU TSC skew check failed: " + "; ".join(problems) + ) + results.append(merged) + return results diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py new file mode 100644 index 000000000..ffde21510 --- /dev/null +++ b/scripts/test_time_abi.py @@ -0,0 +1,466 @@ +"""Unit tests for the harness-side NVX time ABI checks and host qualification.""" + +from __future__ import annotations + +import queue +import socket +import sys +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from nvx_tools import benchmark, openvmm_process, time_abi # noqa: E402 +from nvx_tools.time_abi import TimeAbiFailure, TimeAbiMonitor # noqa: E402 + +BOOT_LINE = ( + "NVX-TIME-ABI: v=1 phase=boot status=ok cpus=4 tsc_hz=2194804000 " + "lapic_hz=1000000000 generation=0 elapsed_us=1873\n" +) +KVM_BOOT = ["openvmm", "--hypervisor", "kvm", "--kernel", "vmlinux"] +MSHV_RESTORE = ["openvmm", "--hypervisor", "mshv", "--restore-snapshot", "snap"] + + +def warp_output( + *, + pairs: int = 6, + backward: int = 0, + offset: int = 15, + verdict: str = "PASS", + stalled: int = 0, + conclusive: int = 1, +) -> str: + return ( + "pair 0-1: warp_iterations=881154 warps=0 max_backward_ns=0\n" + f"NVX-TIME-PROBE warp max_backward_cycles=0 max_backward_ns={backward} " + f"max_abs_offset_ns={offset} pairs={pairs} verdict={verdict} bound_ns=1000\r\n" + "NVX-TIME-PROBE warp-detail max_abs_offset_cycles=33.0 max_uncertainty_ns=127 " + "max_skew_bound_ns=130 total_warps=0 inconsistent_pairs=0 " + f"stalled_pairs={stalled} cpus=0-3 duration_ms=100 tsc_hz=2194804000 " + f"tsc_hz_source=kmsg-detected-processor conclusive={conclusive}\n" + ) + + +class FieldParsingTests(unittest.TestCase): + def test_parses_plain_and_quoted_fields_with_guest_escapes(self): + fields = time_abi.parse_fields( + 'v=1 code=G_TSC_WARP detail="say \\"hi\\" \\\\ \\x07 end" phase=boot' + ) + self.assertEqual( + fields, + { + "v": "1", + "code": "G_TSC_WARP", + "detail": 'say "hi" \\ \x07 end', + "phase": "boot", + }, + ) + + def test_rejects_malformed_fields(self): + for text in ( + "novalue", + "=x", + 'detail="unterminated', + 'detail="bad \\q escape"', + 'detail="\\x4"', + "v=1 v=2", + ): + with self.subTest(text=text), self.assertRaises(ValueError): + time_abi.parse_fields(text) + + def test_parses_markers_and_violations(self): + marker = time_abi.parse_marker(BOOT_LINE) + self.assertIsNotNone(marker) + assert marker is not None + self.assertEqual(marker["phase"], "boot") + self.assertEqual(marker["tsc_hz"], "2194804000") + self.assertIsNone(time_abi.parse_marker("NVX-TIME-REPORT: v=1 phase=boot")) + with self.assertRaises(ValueError): + time_abi.parse_marker("NVX-TIME-ABI: v=1 status=ok") + violation = time_abi.parse_violation( + "noise NVX-TIME-ABI-VIOLATION: v=1 code=G_RCU_STALL source=watcher " + 'phase=runtime generation=1 boottime_ns=5 detail="rcu: INFO: x"\r' + ) + self.assertIsNotNone(violation) + assert violation is not None + self.assertEqual(violation["code"], "G_RCU_STALL") + self.assertEqual(violation["detail"], "rcu: INFO: x") + + def test_describes_only_time_abi_exit_statuses(self): + self.assertIn("conformance", time_abi.describe_exit_status(193) or "") + self.assertIn("runtime violation", time_abi.describe_exit_status(194) or "") + self.assertIn("restore repair", time_abi.describe_exit_status(195) or "") + for status in (None, 0, 1, 37, 192, 196, 255): + self.assertIsNone(time_abi.describe_exit_status(status)) + + def test_maps_cpu_generations(self): + self.assertEqual(time_abi.cpu_generation("GenuineIntel", 6, 85), "skylake-sp") + self.assertEqual(time_abi.cpu_generation("GenuineIntel", 6, 106), "icelake-sp") + self.assertEqual( + time_abi.cpu_generation("GenuineIntel", 6, 207), "emeraldrapids" + ) + self.assertIsNone(time_abi.cpu_generation("GenuineIntel", 6, 143)) + self.assertIsNone(time_abi.cpu_generation("AuthenticAMD", 25, 1)) + + +class MonitorTests(unittest.TestCase): + def test_classifies_commands(self): + monitor = TimeAbiMonitor(KVM_BOOT) + self.assertEqual(monitor.backend, "kvm") + self.assertTrue(monitor.cold_boot) + restore = TimeAbiMonitor(MSHV_RESTORE) + self.assertEqual(restore.backend, "mshv") + self.assertFalse(restore.cold_boot) + + def test_records_markers_across_chunk_boundaries(self): + monitor = TimeAbiMonitor(KVM_BOOT) + data = ( + "[ 0.1] early\n" + + BOOT_LINE + + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + + "NVX-TIME-ABI: v=1 phase=runtime status=uncertain " + + "code=G_SAMPLE_UNCERTAIN epsilon_ns=-1\n" + + "NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 tsc_hz=2194804000 " + + "lapic_hz=1000000000 generation=1 elapsed_us=12\n" + ).encode() + for index in range(0, len(data), 7): + monitor.feed(data[index : index + 7]) + self.assertTrue(monitor.guest_booted) + self.assertEqual(monitor.require_boot("test", online_cpus=4)["cpus"], "4") + self.assertEqual(len(monitor.uncertain), 1) + self.assertEqual(monitor.restores[0]["generation"], "1") + monitor.require_boot_if_booted() + + def test_fails_fast_on_violation_events_and_failed_checks(self): + monitor = TimeAbiMonitor(KVM_BOOT) + with self.assertRaisesRegex(TimeAbiFailure, "violation G_TSC_UNSTABLE"): + monitor.feed( + b"NVX-TIME-ABI-VIOLATION: v=1 code=G_TSC_UNSTABLE source=watcher " + b'phase=runtime generation=0 boottime_ns=1 detail="Marking"\n' + ) + self.assertIn("G_TSC_UNSTABLE", monitor.violation or "") + with self.assertRaisesRegex(TimeAbiFailure, "boot check C9 failed: token"): + TimeAbiMonitor(KVM_BOOT).feed( + b'NVX-TIME-ABI: v=1 phase=boot status=fail check=C9 detail="token"\n' + ) + with self.assertRaisesRegex(TimeAbiFailure, "malformed"): + TimeAbiMonitor(KVM_BOOT).feed(b"NVX-TIME-ABI: v=1 phase=boot\n") + with self.assertRaisesRegex(TimeAbiFailure, "unknown time ABI marker status"): + TimeAbiMonitor(KVM_BOOT).feed( + b"NVX-TIME-ABI: v=1 phase=boot status=maybe\n" + ) + + def test_ignores_report_only_lines_but_reports_them_when_the_marker_is_missing( + self, + ): + monitor = TimeAbiMonitor(KVM_BOOT) + monitor.feed( + b"NVX-TIME-REPORT-VIOLATION: v=1 code=G_TSC_WARP source=watcher " + b'phase=boot generation=0 boottime_ns=1 detail="x"\n' + b"NVX-TIME-REPORT: v=1 phase=boot status=fail cpus=1 failures=3\n" + b"NVX-GUEST-BOOT-OK: alpine\n" + ) + self.assertTrue(monitor.report_only) + with self.assertRaisesRegex(TimeAbiFailure, "report-only"): + monitor.require_boot_if_booted() + + def test_requires_the_boot_marker_only_for_cold_boots_that_reached_the_shell( + self, + ): + monitor = TimeAbiMonitor(KVM_BOOT) + monitor.require_boot_if_booted() + monitor.feed(b"NVX-GUEST-BOOT-OK: alpine\n") + with self.assertRaisesRegex(TimeAbiFailure, "does not implement time ABI"): + monitor.require_boot_if_booted() + restore = TimeAbiMonitor(MSHV_RESTORE) + restore.feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") + restore.require_boot_if_booted() + + def test_validates_boot_marker_fields_against_the_backend(self): + cases = ( + ("v=1", "v=2", "version"), + ("lapic_hz=1000000000", "lapic_hz=200000000", "kvm rate"), + ("tsc_hz=2194804000", "tsc_hz=400000000", "outside 500 MHz"), + ("tsc_hz=2194804000", "tsc_hz=fast", "not an integer"), + ("generation=0", "generation=3", "not 0 at cold boot"), + ("cpus=4", "cpus=2", "cpus=2 is not 4"), + ) + for valid, invalid, message in cases: + fields = time_abi.parse_marker(BOOT_LINE.replace(valid, invalid)) + assert fields is not None + with self.subTest(invalid=invalid): + with self.assertRaisesRegex(TimeAbiFailure, message): + time_abi.validate_boot_marker(fields, backend="kvm", online_cpus=4) + mshv = time_abi.parse_marker( + BOOT_LINE.replace("lapic_hz=1000000000", "lapic_hz=200000000") + ) + assert mshv is not None + time_abi.validate_boot_marker(mshv, backend="mshv") + + def test_classifies_time_abi_exit_statuses_with_the_violation_event(self): + monitor = TimeAbiMonitor(MSHV_RESTORE) + monitor.check_exit(0) + monitor.check_exit(None) + with self.assertRaisesRegex( + TimeAbiFailure, "status 195 \\(restore repair\\): no NVX-TIME-ABI" + ): + monitor.check_exit(195) + try: + monitor.feed( + b"NVX-TIME-ABI-VIOLATION: v=1 code=G_REPAIR_SAMPLE source=repair " + b'phase=restore generation=1 boottime_ns=9 detail="epsilon"' + ) + monitor.finish() + except TimeAbiFailure: + pass + with self.assertRaisesRegex(TimeAbiFailure, "status 195.*G_REPAIR_SAMPLE"): + monitor.check_exit(195) + + def test_bounds_an_unterminated_line(self): + monitor = TimeAbiMonitor(KVM_BOOT) + monitor.feed(b"x" * (time_abi._MAX_PENDING_LINE + 10)) # pyright: ignore[reportPrivateUsage] + monitor.feed(b"\n" + BOOT_LINE.encode()) + self.assertIsNotNone(monitor.boot) + + +class WarpProbeTests(unittest.TestCase): + def test_accepts_conclusive_passing_measurements(self): + results = time_abi.check_warp_probe(warp_output(), cpus=4, context="test") + self.assertEqual(results[0]["max_abs_offset_ns"], "15") + single = time_abi.check_warp_probe( + warp_output(pairs=0, offset=0), cpus=1, context="one vCPU" + ) + self.assertEqual(single[0]["pairs"], "0") + self.assertEqual( + time_abi.warp_probe_command(), + "/sbin/nvx-time-probe warp --bound-ns 1000", + ) + self.assertTrue(time_abi.warp_probe_command(cpus="0-3").endswith("--cpus 0-3")) + + def test_rejects_skew_stalls_and_inconclusive_measurements(self): + cases = ( + (warp_output(backward=1001), "max_backward_ns=1001"), + (warp_output(offset=1500, verdict="FAIL"), "max_abs_offset_ns=1500"), + (warp_output(stalled=1, verdict="FAIL"), "1 CPU pair"), + (warp_output(conclusive=0), "inconclusive"), + (warp_output(pairs=3), "measured 3 CPU pairs instead of 6"), + (warp_output(verdict="FAIL"), "verdict=FAIL"), + ("nothing here\n", "printed no summary"), + ( + warp_output().split("NVX-TIME-PROBE warp-detail")[0], + "1 summaries and 0 detail", + ), + ( + "NVX-TIME-PROBE warp pairs=6\nNVX-TIME-PROBE warp-detail x=1\n", + "incomplete", + ), + ('NVX-TIME-PROBE warp detail="\n', "malformed"), + ) + for text, message in cases: + with self.subTest(message=message): + with self.assertRaisesRegex(TimeAbiFailure, message): + time_abi.check_warp_probe(text, cpus=4, context="restore 4") + + +VIOLATION = ( + b"NVX-TIME-ABI-VIOLATION: v=1 code=G_RCU_STALL source=watcher phase=runtime " + b'generation=1 boottime_ns=99 detail="rcu: INFO: rcu_preempt self-detected stall"\n' +) + + +class FakeProcess: + pid = 123 + + def __init__(self, status: int) -> None: + self.status = status + self.exited = False + self.returncode: int | None = None + + def poll(self) -> int | None: + if self.exited: + self.returncode = self.status + return self.returncode + + def wait(self, timeout: float | None = None) -> int: + del timeout + self.exited = True + self.returncode = self.status + return self.status + + def terminate(self) -> None: + self.exited = True + + def kill(self) -> None: + self.exited = True + + +class FakeInteraction: + def __init__(self, chunks: list[bytes], status: int) -> None: + self.process = FakeProcess(status) + self.chunks = chunks + self.writes: list[bytes] = [] + + def read_output(self, chunks: queue.Queue[bytes | None]) -> None: + for chunk in self.chunks: + chunks.put(chunk) + self.process.exited = True + chunks.put(None) + + def read_stderr(self, chunks: queue.Queue[bytes | None]) -> None: + chunks.put(None) + + def write_input(self, data: bytes) -> None: + self.writes.append(data) + + def close(self) -> None: + pass + + +class RunnerWiringTests(unittest.TestCase): + def _measure(self, chunks: list[bytes], status: int, command: list[str]): + interaction = FakeInteraction(chunks, status) + with ( + patch.object(benchmark, "InteractiveProcess", return_value=interaction), + patch.object(benchmark, "live_peak_rss_bytes", return_value=1024), + ): + return benchmark.measure_once( + command, + environment={}, + timeout=5, + marker=benchmark.RESTORE_MARKER, + marker_must_be_line=True, + guest_exit_prequeued=True, + ) + + def test_measure_once_fails_fast_on_a_violation_event(self): + with self.assertRaisesRegex(RuntimeError, "violation G_RCU_STALL") as raised: + self._measure( + [b"restored\n", VIOLATION, benchmark.RESTORE_MARKER + b"\n"], + 194, + MSHV_RESTORE, + ) + self.assertIn("--- OpenVMM output ---", str(raised.exception)) + + def test_measure_once_classifies_a_time_abi_power_off(self): + with self.assertRaisesRegex( + RuntimeError, "status 195 \\(restore repair\\): no NVX-TIME-ABI" + ): + self._measure([b"restoring\n"], 195, MSHV_RESTORE) + with self.assertRaisesRegex(RuntimeError, "OpenVMM exited with status 1"): + self._measure([b"restoring\n"], 1, MSHV_RESTORE) + + def test_measure_once_classifies_a_power_off_during_teardown(self): + interaction = FakeInteraction([benchmark.RESTORE_MARKER + b"\n"], 194) + with ( + patch.object(benchmark, "InteractiveProcess", return_value=interaction), + patch.object(benchmark, "live_peak_rss_bytes", return_value=1024), + patch.object(benchmark, "wait_for_process_exit", return_value=194), + ): + with self.assertRaisesRegex(RuntimeError, "status 194 \\(runtime"): + benchmark.measure_once( + MSHV_RESTORE, + environment={}, + timeout=5, + marker=benchmark.RESTORE_MARKER, + marker_must_be_line=True, + guest_exit_prequeued=True, + ) + + def test_run_guest_script_classifies_a_conformance_power_off(self): + interaction = FakeInteraction( + [ + b'NVX-TIME-ABI: v=1 phase=boot status=fail check=C4 detail="kvm-clock"', + ], + 193, + ) + with patch.object(benchmark, "InteractiveProcess", return_value=interaction): + with self.assertRaisesRegex( + RuntimeError, "boot check C4 failed: kvm-clock" + ): + benchmark.run_guest_script( + KVM_BOOT, "true\n", b"DONE", timeout=5, boot_marker=b"BOOT" + ) + + def test_capture_snapshot_classifies_a_failed_capture_check(self): + interaction = FakeInteraction([b"booting\n"], 193) + with tempfile.TemporaryDirectory() as temporary: + with patch.object( + benchmark, "InteractiveProcess", return_value=interaction + ): + with self.assertRaisesRegex(RuntimeError, "status 193 \\(conformance"): + benchmark.capture_snapshot( + KVM_BOOT, + Path(temporary) / "snapshot", + backend="kvm", + timeout=5, + ) + + def test_openvmm_process_reports_violations_and_time_abi_statuses(self): + with tempfile.TemporaryDirectory() as temporary: + log_path = Path(temporary) / "process.log" + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([b"FIRST\n", VIOLATION], 194), + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + process.wait_for(b"FIRST", 1) + with self.assertRaisesRegex(RuntimeError, "G_RCU_STALL"): + process.wait_for(b"NEVER", 1) + self.assertIn(b"G_RCU_STALL", log_path.read_bytes()) + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([b"working\n"], 194), + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + with self.assertRaisesRegex(RuntimeError, "status 194"): + process.wait(1) + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([b"working\n"], 195), + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + with self.assertRaisesRegex(RuntimeError, "status 195"): + process.wait_for_line(b"NEVER", 1) + + def test_openvmm_process_waits_for_time_abi_phase_markers(self): + restore = ( + b"NVX-TIME-ABI: v=1 phase=restore status=ok cpus=2 tsc_hz=2300000000 " + b"lapic_hz=200000000 generation=1 elapsed_us=310\n" + ) + with tempfile.TemporaryDirectory() as temporary: + log_path = Path(temporary) / "process.log" + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([b"x\n", restore], 0), + ): + with openvmm_process.OpenvmmProcess(MSHV_RESTORE, log_path) as process: + marker = process.wait_for_time_abi("restore", 1) + self.assertEqual(marker["generation"], "1") + self.assertEqual(process.wait(1).returncode, 0) + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([b"x\n"], 0), + ): + with openvmm_process.OpenvmmProcess(MSHV_RESTORE, log_path) as process: + with self.assertRaisesRegex(RuntimeError, "restore marker"): + process.wait_for_time_abi("restore", 1) + + def test_tcp_console_monitor_reports_violations(self): + connection, peer = socket.socketpair() + console = openvmm_process.TcpConsole(connection, TimeAbiMonitor(KVM_BOOT)) + peer.sendall(VIOLATION) + with self.assertRaisesRegex(TimeAbiFailure, "G_RCU_STALL"): + console.wait_for(b"NEVER", 1.0) + console.close() + peer.close() + + +if __name__ == "__main__": + unittest.main() From 8de77c62ad13b0fdc55f003ce9c7730780aef193 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:38:02 -0700 Subject: [PATCH 019/188] scripts: add nvx.py doctor host qualification Time ABI v1 replaces the runner's nonstop_tsc check with host qualification: checks H1 to H7 in doc/design/time-abi.md, each printed as one NVX-DOCTOR line, failing closed. Nothing implemented them, and the cross-CPU skew and rate checks need TSC reads that Python cannot make. nvx.py doctor --backend runs the checks in spec order: - H1: the backend device, or WHvGetCapability on Windows. - H2: the CPU signature, microcode, and OS build, mapped to the skylake-sp, icelake-sp, or emeraldrapids generation. An unknown CPU fails with E_PROFILE_HOST_UNKNOWN, and a Linux host without constant_tsc and nonstop_tsc with E_PROFILE_UNSUPPORTED. A WHP host's invariant-TSC bit is reported only, while the spec leaves it TBD(whp). - H3: openvmm --x-time-abi-verify, whose profile and rates feed H4. - H4: two 1 s TSC rate measurements against host monotonic time, within 1 ppm of each other and 100 ppm of the backend rate, and a tsc host clocksource on KVM. MSHV roots read time through the hypervisor's TSC page, so hyperv_clocksource_tsc_page is accepted there. - H5: a pinned-thread ping-pong over every host CPU pair, at most 1 us. - H6: a microVM with up to 8 vCPUs must print a valid NVX-TIME-ABI boot marker and pass the guest warp probe. - H7: no STA_UNSYNC from adjtimex, or a synchronized w32tm source. H2, H4, and H5 use host_time_probe.rs, a dependency-free Rust probe that the doctor builds with rustc once per source version, into the runner tool cache in CI. --summary appends a Markdown table with the generation, profile, rates, and skew metrics for job summaries. The command is inert until CI calls it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-nvx/action.yml | 3 + doc/ci.md | 28 + scripts/nvx.py | 7 + scripts/nvx_tools/doctor.py | 819 ++++++++++++++++++++++++ scripts/nvx_tools/host_time_probe.rs | 516 +++++++++++++++ scripts/nvx_tools/time_abi.py | 48 +- scripts/test_time_abi.py | 486 +++++++++++++- 7 files changed, 1893 insertions(+), 14 deletions(-) create mode 100644 scripts/nvx_tools/doctor.py create mode 100644 scripts/nvx_tools/host_time_probe.rs diff --git a/.github/actions/validate-nvx/action.yml b/.github/actions/validate-nvx/action.yml index e453afe2e..364367cf7 100644 --- a/.github/actions/validate-nvx/action.yml +++ b/.github/actions/validate-nvx/action.yml @@ -20,6 +20,7 @@ runs: python3 scripts/nvx.py test-aci-edge-sandboxes --help python3 scripts/nvx.py test-adversarial --help python3 scripts/nvx.py benchmark --help + python3 scripts/nvx.py doctor --help - name: Validate NVX CLI on Windows if: runner.os == 'Windows' @@ -48,3 +49,5 @@ runs: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\nvx.py benchmark --help if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + python scripts\nvx.py doctor --help + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/doc/ci.md b/doc/ci.md index 4f7206181..1ef1fb076 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -156,6 +156,34 @@ pull requests run the GitHub-hosted validation jobs but do not execute code on the Azure runner fleet. A maintainer must stage an external contribution on a trusted repository branch before running the backend matrices. +## Host qualification + +`python3 scripts/nvx.py doctor --backend ` qualifies a host for +the [time ABI](design/time-abi.md#host-qualification). It runs checks H1 to H7 +in order, prints one `NVX-DOCTOR: check= status= detail="..."` +line per check, and exits with status 1 if any check fails. `--checks` selects +a subset, and `--summary` appends a Markdown table with the CPU generation, +profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. A +failure that matches a time ABI failure code starts its detail with the code in +brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. + +| Check | Implementation | +| --- | --- | +| H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | +| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A Linux host must report `constant_tsc` and `nonstop_tsc` (`E_PROFILE_UNSUPPORTED`), so `azure-azlinux-2` stays unqualified. A Windows host's invariant-TSC CPUID bit is only reported: Azure WHP hosts cannot see one, and the spec qualifies them on the measured warp and rate stability | +| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names; OpenVMM's interim `interim.host..v1` profile is accepted until it selects catalog profiles. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | +| H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A KVM host's clocksource must be `tsc`; an MSHV root's may also be `hyperv_clocksource_tsc_page` | +| H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | +| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker, and `nvx-time-probe warp` stays within 1,000 ns | +| H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows | + +H2, H4, and H5 use a dependency-free host probe, +[`host_time_probe.rs`](../scripts/nvx_tools/host_time_probe.rs), which the +doctor builds with `rustc` once per source version into +`$RUNNER_TOOL_CACHE/nvx-host-time-probe` (or `build/host-time-probe` outside +CI). H3 and H6 need the OpenVMM binary and the guest artifacts; `--openvmm`, +`--kernel`, and `--initrd` override their default build paths. + ## Rust crate The required `aci-edge-sandboxes` job checks the [`aci_edge_sandboxes` crate](../aci_edge_sandboxes/README.md) on diff --git a/scripts/nvx.py b/scripts/nvx.py index 5b623d0e5..96a56a85d 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -70,6 +70,7 @@ from nvx_tools.create_linux_source_archive import ( configure_parser as configure_linux_source_archive_parser, ) +from nvx_tools.doctor import configure_parser as configure_doctor_parser from nvx_tools.egress_policy import compile_policy_file from nvx_tools.guests import GUEST_NAMES, guest_descriptor from nvx_tools.microvm_tests import configure_parser as configure_microvm_test_parser @@ -818,6 +819,12 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) configure_microvm_test_parser(microvm_tests) + doctor = subparsers.add_parser( + "doctor", + help="qualify this host for the NVX time ABI", + ) + configure_doctor_parser(doctor) + aci_edge_sandboxes_tests = subparsers.add_parser( "test-aci-edge-sandboxes", help="run the aci_edge_sandboxes crate lifecycle test on a real hypervisor", diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py new file mode 100644 index 000000000..96c89772c --- /dev/null +++ b/scripts/nvx_tools/doctor.py @@ -0,0 +1,819 @@ +"""Host qualification for the NVX time ABI (``nvx.py doctor``). + +doc/design/time-abi.md ("Host qualification") defines checks H1 to H7. Each +check prints one ``NVX-DOCTOR: check= status= detail="..."`` +line, and qualification fails closed if any selected check fails. A failure +that matches a time ABI failure code starts its detail with that code in +brackets, as OpenVMM errors do. +""" + +from __future__ import annotations + +import argparse +import ctypes +import hashlib +import os +import platform +import re +import shutil +import subprocess +import sys +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass, field +from pathlib import Path + +from .benchmark import run_guest_script, workload_boot_command +from .build_constants import AlpineBuildConstants, BuildConstants, KernelBuildConstants +from .ci import OPENVMM_TEST_BACKENDS +from .common import ScriptError, artifact_path, openvmm_binary_path +from .time_abi import ( + LAPIC_HZ, + MAX_TSC_HZ, + MIN_TSC_HZ, + WARP_BOUND_NS, + TimeAbiFailure, + TimeAbiMonitor, + check_warp_probe, + cpu_generation, + parse_fields, + warp_probe_command, +) + +CHECK_IDS = ("H1", "H2", "H3", "H4", "H5", "H6", "H7") +CHECK_TITLES: Mapping[str, str] = { + "H1": "backend device or API", + "H2": "CPU fingerprint and generation", + "H3": "OpenVMM time ABI preflight", + "H4": "host TSC rate stability", + "H5": "host cross-CPU TSC skew", + "H6": "guest warp probe", + "H7": "host UTC synchronization", +} +DOCTOR_PREFIX = "NVX-DOCTOR: " +PROBE_SOURCE = Path(__file__).with_name("host_time_probe.rs") +PROBE_PREFIX = "NVX-HOST-TIME-PROBE " +VERIFY_PREFIX = "NVX-TIME-ABI-VERIFY:" +RATE_MEASURE_MS = 1000 +RATE_AGREEMENT_PPM = 1.0 +RATE_TOLERANCE_PPM = 100.0 +SKEW_PAIR_DURATION_MS = 5 +GUEST_VCPU_COUNTS = (8, 4, 2, 1) +GUEST_MEMORY_MIB = 128 +GUEST_WARP_MARKER = b"NVX-DOCTOR-WARP-DONE" +# Azure WHP hosts cannot see an invariant TSC; the spec qualifies them on the +# measured warp (H6) and rate stability (H4), so the bit is only reported. +# A Linux host whose root sees no invariant TSC (azure-azlinux-2) stays +# unqualified, which the spec's restore matrix expects. +WHP_REQUIRES_INVARIANT_TSC = False +LINUX_INVARIANT_TSC_FLAGS = ("constant_tsc", "nonstop_tsc") +# KVM rewrites per-vCPU TSC offsets on a host whose TSC Linux distrusts. An +# MSHV root's Linux reads time through the hypervisor's TSC page instead. +HOST_CLOCKSOURCES: Mapping[str, tuple[str, ...]] = { + "kvm": ("tsc",), + "mshv": ("tsc", "hyperv_clocksource_tsc_page"), +} +CLOCKSOURCE_PATH = Path( + "/sys/devices/system/clocksource/clocksource0/current_clocksource" +) +CPUINFO_PATH = Path("/proc/cpuinfo") +ADJTIMEX_TIME_ERROR = 5 +ADJTIMEX_STA_UNSYNC = 0x0040 +WHP_CAPABILITY_HYPERVISOR_PRESENT = 0 + + +@dataclass +class CheckResult: + check: str + passed: bool + detail: str + + def line(self) -> str: + status = "pass" if self.passed else "fail" + detail = self.detail.replace("\\", "\\\\").replace('"', '\\"') + return f'{DOCTOR_PREFIX}check={self.check} status={status} detail="{detail}"' + + +@dataclass +class DoctorContext: + backend: str + openvmm: Path + kernel: Path + initrd: Path + openvmm_args: tuple[str, ...] + probe_directory: Path + timeout: float + facts: dict[str, str] = field(default_factory=dict[str, str]) + probe: Path | None = None + + +def host_is_windows() -> bool: + return os.name == "nt" + + +def host_clocksource() -> str: + return CLOCKSOURCE_PATH.read_text(encoding="utf-8").strip() + + +def _escape_markdown(text: str) -> str: + return text.replace("|", "\\|").replace("\n", " ") + + +def probe_path(directory: Path) -> Path: + """Return the cached host probe binary for the current probe source.""" + digest = hashlib.sha256(PROBE_SOURCE.read_bytes()).hexdigest()[:16] + suffix = ".exe" if os.name == "nt" else "" + return directory / f"nvx-host-time-probe-{digest}{suffix}" + + +def build_probe(directory: Path) -> Path: + """Build the host probe once per source version with rustc.""" + target = probe_path(directory) + if target.is_file(): + return target + rustc = shutil.which("rustc") + if rustc is None: + raise ScriptError("rustc is required to build the host time probe") + directory.mkdir(parents=True, exist_ok=True) + temporary = target.with_name(f".{target.stem}-{os.getpid()}{target.suffix}") + completed = subprocess.run( + [ + rustc, + "--edition=2021", + "-C", + "opt-level=2", + "-C", + "debuginfo=0", + "-o", + os.fspath(temporary), + os.fspath(PROBE_SOURCE), + ], + capture_output=True, + text=True, + check=False, + ) + if completed.returncode != 0: + temporary.unlink(missing_ok=True) + raise ScriptError( + f"rustc failed to build the host time probe:\n{completed.stderr.strip()}" + ) + try: + os.replace(temporary, target) + except OSError: + # Another job built the same version first and may be running it. + temporary.unlink(missing_ok=True) + if not target.is_file(): + raise + temporary.with_suffix(".pdb").unlink(missing_ok=True) + return target + + +def run_probe( + context: DoctorContext, *arguments: str +) -> list[tuple[str, dict[str, str]]]: + """Run the host probe and parse its ``NVX-HOST-TIME-PROBE`` records.""" + if context.probe is None: + context.probe = build_probe(context.probe_directory) + completed = subprocess.run( + [os.fspath(context.probe), *arguments], + capture_output=True, + text=True, + timeout=context.timeout, + check=False, + ) + if completed.returncode != 0: + message = completed.stderr.strip() or f"exit status {completed.returncode}" + raise ScriptError(f"host time probe {arguments[0]} failed: {message}") + records: list[tuple[str, dict[str, str]]] = [] + for line in completed.stdout.splitlines(): + if line.startswith(PROBE_PREFIX): + kind, _, fields = line.removeprefix(PROBE_PREFIX).partition(" ") + records.append((kind, parse_fields(fields))) + return records + + +def _probe_record( + records: Sequence[tuple[str, dict[str, str]]], kind: str +) -> dict[str, str]: + for record_kind, fields in records: + if record_kind == kind: + return fields + raise ScriptError(f"host time probe printed no {kind} record") + + +def _linux_cpuinfo(path: Path = CPUINFO_PATH) -> dict[str, str]: + """Return the first processor's /proc/cpuinfo fields.""" + fields: dict[str, str] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + if fields: + break + continue + name, separator, value = line.partition(":") + if separator: + fields.setdefault(name.strip(), value.strip()) + return fields + + +def _windows_registry_value(key: str, name: str) -> object: + if sys.platform != "win32": + raise ScriptError("the Windows registry is unavailable") + import winreg + + with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, key) as handle: + value, _ = winreg.QueryValueEx(handle, name) + return value + + +def host_cpu(context: DoctorContext) -> dict[str, str]: + """Collect the host CPU identity and the host OS's view of its TSC.""" + if host_is_windows(): + match = re.search( + r"Family (\d+) Model (\d+) Stepping (\d+), (\S+)", platform.processor() + ) + if match is None: + raise ScriptError(f"cannot parse the processor {platform.processor()!r}") + family, model, stepping, vendor = match.groups() + processor_key = r"HARDWARE\DESCRIPTION\System\CentralProcessor\0" + info = { + "vendor": vendor, + "family": family, + "model": model, + "stepping": stepping, + "brand": str( + _windows_registry_value(processor_key, "ProcessorNameString") + ).strip(), + } + revision = _windows_registry_value(processor_key, "Update Revision") + if isinstance(revision, bytes) and len(revision) in (4, 8): + info["microcode"] = hex(int.from_bytes(revision[-4:], "little")) + build = _windows_registry_value( + r"SOFTWARE\Microsoft\Windows NT\CurrentVersion", "UBR" + ) + info["os"] = f"Windows {platform.version()}.{build}" + try: + probed = _probe_record(run_probe(context, "cpu"), "cpu") + info["invariant_tsc"] = "yes" if probed["invariant_tsc"] == "1" else "no" + except (ScriptError, KeyError, subprocess.TimeoutExpired) as error: + info["invariant_tsc"] = f"unknown ({error})" + return info + cpuinfo = _linux_cpuinfo() + flags = cpuinfo.get("flags", "").split() + missing = [flag for flag in LINUX_INVARIANT_TSC_FLAGS if flag not in flags] + return { + "vendor": cpuinfo.get("vendor_id", "unknown"), + "family": cpuinfo.get("cpu family", "?"), + "model": cpuinfo.get("model", "?"), + "stepping": cpuinfo.get("stepping", "?"), + "microcode": cpuinfo.get("microcode", "unknown"), + "brand": cpuinfo.get("model name", "unknown"), + "os": f"Linux {platform.release()}", + "invariant_tsc": "yes" if not missing else f"no (missing {' '.join(missing)})", + } + + +def _whp_hypervisor_present() -> bool: + if sys.platform != "win32": + raise ScriptError("WHP qualification requires Windows") + try: + library = ctypes.WinDLL("WinHvPlatform.dll") + except OSError as error: + raise ScriptError(f"WinHvPlatform.dll is unavailable: {error}") from error + present = ctypes.c_int32(0) + written = ctypes.c_uint32(0) + result = int( + library.WHvGetCapability( + WHP_CAPABILITY_HYPERVISOR_PRESENT, + ctypes.byref(present), + ctypes.sizeof(present), + ctypes.byref(written), + ) + ) + if result != 0: + raise ScriptError( + f"WHvGetCapability failed with HRESULT 0x{result & 0xFFFFFFFF:08x}" + ) + return bool(present.value) + + +def check_backend(context: DoctorContext) -> CheckResult: + if context.backend == "whp": + if not host_is_windows(): + return CheckResult("H1", False, "WHP qualification requires Windows") + if not _whp_hypervisor_present(): + return CheckResult( + "H1", False, "the Windows Hypervisor Platform reports no hypervisor" + ) + return CheckResult("H1", True, "the Windows Hypervisor Platform is present") + if host_is_windows(): + return CheckResult( + "H1", False, f"{context.backend} qualification requires Linux" + ) + device = Path("/dev") / context.backend + if not device.exists(): + return CheckResult("H1", False, f"{device} does not exist") + if not os.access(device, os.R_OK | os.W_OK): + return CheckResult("H1", False, f"{device} is not readable and writable") + if context.backend == "kvm" and Path("/dev/mshv").exists(): + return CheckResult( + "H1", False, "/dev/mshv exists, so OpenVMM would select MSHV, not KVM" + ) + return CheckResult("H1", True, f"{device} is readable and writable") + + +def check_cpu(context: DoctorContext) -> CheckResult: + info = host_cpu(context) + signature = f"{info['vendor']} {info['family']}/{info['model']}/{info['stepping']}" + try: + generation = cpu_generation( + info["vendor"], + int(info["family"]), + int(info["model"]), + int(info["stepping"]), + ) + except ValueError: + generation = None + profile = generation.profile_id if generation else None + name = generation.name if generation else "unknown" + context.facts.update( + { + "generation": name, + "profile": profile or "none", + "cpu": signature, + "brand": info["brand"], + "microcode": info.get("microcode", "unknown"), + "os": info["os"], + "invariant_tsc": info["invariant_tsc"], + } + ) + detail = ( + f"generation={name} profile={profile or 'none'} cpu={signature} " + f"microcode={info.get('microcode', 'unknown')} os={info['os']} " + f"invariant_tsc={info['invariant_tsc']} brand={info['brand']}" + ) + if generation is None: + return CheckResult( + "H2", + False, + f"[E_PROFILE_HOST_UNKNOWN] {signature} is not a time ABI generation " + "(skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids " + f"6/207); {detail}", + ) + required = context.backend != "whp" or WHP_REQUIRES_INVARIANT_TSC + if required and info["invariant_tsc"] != "yes": + return CheckResult( + "H2", + False, + "[E_PROFILE_UNSUPPORTED] the host does not expose an invariant TSC, " + f"which every CPU profile requires; {detail}", + ) + return CheckResult("H2", True, detail) + + +def _guest_vcpus() -> int: + host_cpus = os.cpu_count() or 1 + return next(count for count in GUEST_VCPU_COUNTS if count <= host_cpus) + + +def check_openvmm_preflight(context: DoctorContext) -> CheckResult: + for path, description in ( + (context.openvmm, "OpenVMM"), + (context.kernel, "guest kernel"), + (context.initrd, "guest initramfs"), + ): + if not path.is_file(): + return CheckResult("H3", False, f"{description} was not found at {path}") + command = [ + os.fspath(context.openvmm), + "--machine", + "microvm", + "--processors", + str(_guest_vcpus()), + "--hypervisor", + context.backend, + "--kernel", + os.fspath(context.kernel), + "--initrd", + os.fspath(context.initrd), + # Before the flip, pass --openvmm-arg=--x-time-abi-v1. + *context.openvmm_args, + "--x-time-abi-verify", + ] + environment = os.environ.copy() + environment["OPENVMM_LOG"] = "off" + completed = subprocess.run( + command, + capture_output=True, + text=True, + timeout=context.timeout, + env=environment, + check=False, + ) + output = f"{completed.stdout}\n{completed.stderr}" + verify = next( + ( + line.strip() + for line in output.splitlines() + if line.strip().startswith(VERIFY_PREFIX) + ), + None, + ) + if verify is None: + last = next( + (line.strip() for line in reversed(output.splitlines()) if line.strip()), + "no output", + ) + if "x-time-abi" in output and "unexpected argument" in output: + last = "this OpenVMM predates the time ABI's --x-time-abi-verify mode" + return CheckResult( + "H3", + False, + f"OpenVMM verification exited {completed.returncode} without a " + f"verification line: {last}", + ) + fields = parse_fields(verify.removeprefix(VERIFY_PREFIX).strip()) + if fields.get("status") != "ok" or completed.returncode != 0: + code = fields.get("code", "none") + prefix = f"[{code}] " if code.startswith("E_") else "" + return CheckResult( + "H3", + False, + f"{prefix}OpenVMM verification failed (exit {completed.returncode}): " + f"{fields.get('detail', verify)}", + ) + problems: list[str] = [] + if fields.get("backend") != context.backend: + problems.append(f"OpenVMM verified backend {fields.get('backend')}") + for name in ("tsc_hz", "native_tsc_hz"): + try: + rate = int(fields.get(name, "")) + if not MIN_TSC_HZ <= rate <= MAX_TSC_HZ: + problems.append(f"[E_TSC_RATE_IMPLAUSIBLE] {name}={rate}") + except ValueError: + problems.append(f"{name}={fields.get(name)!r} is not an integer") + expected_lapic = LAPIC_HZ[context.backend] + if fields.get("lapic_hz") != str(expected_lapic): + problems.append( + f"[E_LAPIC_RATE_MISMATCH] lapic_hz={fields.get('lapic_hz')} is not " + f"{expected_lapic}" + ) + expected_profile = context.facts.get("profile", "none") + selected_profile = fields.get("cpu_profile", "") + interim = f"interim.host.{context.backend}.v1" + if expected_profile != "none" and selected_profile != interim: + # Later revisions of the same generation are fine. + lineage = expected_profile.rpartition(".v")[0] + ".v" + if not selected_profile.startswith(lineage): + problems.append( + f"OpenVMM selected CPU profile {selected_profile}, but H2 maps the " + f"host to {expected_profile}" + ) + detail = " ".join(f"{name}={value}" for name, value in fields.items()) + if selected_profile == interim: + # OpenVMM uses an interim host profile until it selects catalog + # profiles; core-design, "Changes". + detail += f"; interim CPU profile, expected {expected_profile} later" + context.facts.update( + { + "tsc_hz": fields.get("tsc_hz", ""), + "native_tsc_hz": fields.get("native_tsc_hz", ""), + "lapic_hz": fields.get("lapic_hz", ""), + "profile": selected_profile, + "route": fields.get("msr_route", ""), + "sync": fields.get("sync", ""), + } + ) + if problems: + return CheckResult("H3", False, "; ".join(problems) + f"; {detail}") + return CheckResult("H3", True, detail) + + +def check_rate(context: DoctorContext) -> CheckResult: + clocksource = "" + if context.backend in HOST_CLOCKSOURCES: + clocksource = host_clocksource() + context.facts["host_clocksource"] = clocksource + records = run_probe( + context, "rate", "--measure-ms", str(RATE_MEASURE_MS), "--count", "2" + ) + rates = [float(fields["tsc_hz"]) for kind, fields in records if kind == "rate"] + if len(rates) != 2: + raise ScriptError(f"host time probe printed {len(rates)} rate records, not 2") + measured = sum(rates) / len(rates) + spread_ppm = (max(rates) - min(rates)) / min(rates) * 1e6 + context.facts["measured_tsc_hz"] = f"{measured:.0f}" + context.facts["rate_spread_ppm"] = f"{spread_ppm:.3f}" + detail = ( + f"tsc_hz={measured:.0f} two {RATE_MEASURE_MS} ms measurements differ by " + f"{spread_ppm:.3f} ppm" + ) + problems: list[str] = [] + if not MIN_TSC_HZ <= measured <= MAX_TSC_HZ: + problems.append(f"[E_TSC_RATE_IMPLAUSIBLE] measured {measured:.0f} Hz") + if spread_ppm > RATE_AGREEMENT_PPM: + problems.append( + f"the TSC rate is unstable: {spread_ppm:.3f} ppm > {RATE_AGREEMENT_PPM} ppm" + ) + declared = context.facts.get("native_tsc_hz") or context.facts.get("tsc_hz") + if declared is None: + # validate-runner runs before the OpenVMM binary is available; the + # VM jobs run H3 with H4 to compare against the backend's rate. + detail += "; not compared with the backend's rate (H3 did not run)" + else: + deviation_ppm = abs(measured - int(declared)) / int(declared) * 1e6 + detail += f"; {deviation_ppm:.1f} ppm from the backend's {declared} Hz" + if deviation_ppm > RATE_TOLERANCE_PPM: + problems.append( + f"the backend rate is {deviation_ppm:.1f} ppm from the measured " + f"rate (limit {RATE_TOLERANCE_PPM:g} ppm)" + ) + if clocksource: + detail += f"; host clocksource {clocksource}" + if clocksource not in HOST_CLOCKSOURCES[context.backend]: + problems.append( + f"the host clocksource is {clocksource}, not " + + " or ".join(HOST_CLOCKSOURCES[context.backend]) + ) + if problems: + return CheckResult("H4", False, "; ".join(problems) + f"; {detail}") + return CheckResult("H4", True, detail) + + +def check_host_skew(context: DoctorContext) -> CheckResult: + arguments = ["skew", "--duration-ms", str(SKEW_PAIR_DURATION_MS)] + rate = context.facts.get("measured_tsc_hz") or context.facts.get("tsc_hz") + if rate is not None: + arguments.extend(("--tsc-hz", rate)) + summary = _probe_record(run_probe(context, *arguments), "skew") + offset = int(summary["max_abs_offset_ns"]) + uncertainty = int(summary["max_uncertainty_ns"]) + stalled = int(summary["stalled_pairs"]) + context.facts["host_skew_ns"] = str(offset) + detail = ( + f"pairs={summary['pairs']} cpus={summary['cpus']} max_abs_offset_ns={offset} " + f"max_uncertainty_ns={uncertainty}" + ) + problems: list[str] = [] + if offset > WARP_BOUND_NS: + problems.append(f"max_abs_offset_ns={offset} exceeds {WARP_BOUND_NS}") + if stalled: + problems.append(f"{stalled} CPU pair(s) stalled") + if summary.get("conclusive") != "1": + problems.append( + f"the measurement is inconclusive (uncertainty {uncertainty} ns)" + ) + if problems: + return CheckResult("H5", False, "; ".join(problems) + f"; {detail}") + return CheckResult("H5", True, detail) + + +def check_guest_warp(context: DoctorContext) -> CheckResult: + for path, description in ( + (context.openvmm, "OpenVMM"), + (context.kernel, "guest kernel"), + (context.initrd, "guest initramfs"), + ): + if not path.is_file(): + return CheckResult("H6", False, f"{description} was not found at {path}") + vcpus = _guest_vcpus() + command = workload_boot_command( + context.openvmm, + context.backend, + context.kernel, + context.initrd, + GUEST_MEMORY_MIB, + "quiet loglevel=0", + processors=vcpus, + ) + command.extend(context.openvmm_args) + script = f"{warp_probe_command()}\necho {GUEST_WARP_MARKER.decode()}\nnvx-exit 0\n" + try: + result = run_guest_script( + command, + script, + GUEST_WARP_MARKER, + timeout=context.timeout, + ) + monitor = TimeAbiMonitor(command) + monitor.feed(result["text"].encode()) + monitor.finish() + boot = monitor.require_boot("the guest boot marker", online_cpus=vcpus) + warp = check_warp_probe(result["text"], cpus=vcpus, context="H6")[-1] + except (RuntimeError, TimeAbiFailure) as error: + first = str(error).splitlines()[0] + return CheckResult("H6", False, f"vcpus={vcpus}: {first}") + context.facts["guest_warp_ns"] = warp["max_abs_offset_ns"] + context.facts.setdefault("tsc_hz", boot["tsc_hz"]) + context.facts.setdefault("lapic_hz", boot["lapic_hz"]) + return CheckResult( + "H6", + True, + f"vcpus={vcpus} max_backward_ns={warp['max_backward_ns']} " + f"max_abs_offset_ns={warp['max_abs_offset_ns']} " + f"max_uncertainty_ns={warp['max_uncertainty_ns']} boot_elapsed_us=" + f"{boot.get('elapsed_us', '?')}", + ) + + +def _linux_clock_state() -> tuple[int, int]: + """Return adjtimex's clock state and status without changing the clock.""" + libc = ctypes.CDLL(None, use_errno=True) + buffer = ctypes.create_string_buffer(256) + state = int(libc.adjtimex(buffer)) + if state < 0: + raise ScriptError(f"adjtimex failed: {os.strerror(ctypes.get_errno())}") + # struct timex: unsigned modes, then the long offset, freq, maxerror, and + # esterror fields, then the int status. + status = int.from_bytes(buffer.raw[40:44], sys.byteorder) + return state, status + + +def _windows_time_source() -> tuple[bool, str]: + completed = subprocess.run( + ["w32tm", "/query", "/status"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if completed.returncode != 0: + raise ScriptError( + "w32tm /query /status failed: " + + (completed.stdout.strip() or completed.stderr.strip()) + ) + fields: dict[str, str] = {} + for line in completed.stdout.splitlines(): + name, separator, value = line.partition(":") + if separator: + fields[name.strip()] = value.strip() + source = fields.get("Source", "unknown") + leap = fields.get("Leap Indicator", "") + unsynchronized = ( + leap.startswith("3") + or "Local CMOS Clock" in source + or "Free-running System Clock" in source + ) + return not unsynchronized, f"source={source} leap_indicator={leap or 'unknown'}" + + +def check_utc(context: DoctorContext) -> CheckResult: + del context + if host_is_windows(): + synchronized, detail = _windows_time_source() + if not synchronized: + return CheckResult("H7", False, f"host UTC is not synchronized: {detail}") + return CheckResult("H7", True, detail) + state, status = _linux_clock_state() + detail = f"adjtimex state={state} status=0x{status:04x}" + if state == ADJTIMEX_TIME_ERROR or status & ADJTIMEX_STA_UNSYNC: + return CheckResult( + "H7", False, f"host UTC is not synchronized (STA_UNSYNC); {detail}" + ) + return CheckResult("H7", True, detail) + + +CHECKS: Mapping[str, Callable[[DoctorContext], CheckResult]] = { + "H1": check_backend, + "H2": check_cpu, + "H3": check_openvmm_preflight, + "H4": check_rate, + "H5": check_host_skew, + "H6": check_guest_warp, + "H7": check_utc, +} + + +def run_checks(context: DoctorContext, checks: Sequence[str]) -> list[CheckResult]: + """Run checks in spec order; an exception fails only its own check.""" + results: list[CheckResult] = [] + for check in CHECK_IDS: + if check not in checks: + continue + try: + result = CHECKS[check](context) + except ( + KeyError, + OSError, + RuntimeError, + subprocess.SubprocessError, + ValueError, + ) as error: + first = str(error).splitlines()[0] if str(error) else type(error).__name__ + result = CheckResult(check, False, first) + results.append(result) + print(result.line(), flush=True) + return results + + +def summary_markdown( + backend: str, results: Sequence[CheckResult], facts: Mapping[str, str] +) -> str: + status = "passed" if all(result.passed for result in results) else "failed" + lines = [ + f"### Time ABI host qualification ({backend}): {status}", + "", + "| Check | Name | Status | Detail |", + "| --- | --- | --- | --- |", + ] + for result in results: + lines.append( + f"| {result.check} | {CHECK_TITLES[result.check]} | " + f"{'pass' if result.passed else '**fail**'} | " + f"{_escape_markdown(result.detail)} |" + ) + reported = ( + ("generation", "Generation"), + ("cpu", "CPU"), + ("microcode", "Microcode"), + ("os", "Host OS"), + ("profile", "Profile"), + ("tsc_hz", "Declared TSC rate (Hz)"), + ("lapic_hz", "LAPIC rate (Hz)"), + ("measured_tsc_hz", "Measured TSC rate (Hz)"), + ("host_skew_ns", "Host skew (ns)"), + ("guest_warp_ns", "Guest warp offset (ns)"), + ) + facts_line = " · ".join( + f"{label}: `{facts[name]}`" for name, label in reported if name in facts + ) + if facts_line: + lines.extend(("", facts_line)) + return "\n".join(lines) + "\n\n" + + +def run(args: argparse.Namespace) -> int: + if args.backend not in OPENVMM_TEST_BACKENDS: + raise ScriptError(f"unsupported backend {args.backend!r}") + checks = tuple(dict.fromkeys(args.checks or CHECK_IDS)) + context = DoctorContext( + backend=args.backend, + openvmm=args.openvmm or openvmm_binary_path(), + kernel=args.kernel or artifact_path(KernelBuildConstants.BINARY_NAME), + initrd=args.initrd or artifact_path(AlpineBuildConstants.INITRAMFS_NAME), + openvmm_args=tuple(args.openvmm_arg), + probe_directory=args.probe_dir or default_probe_directory(), + timeout=args.timeout, + ) + results = run_checks(context, checks) + failed = [result.check for result in results if not result.passed] + generation = context.facts.get("generation") + profile = context.facts.get("profile") + print( + f"Time ABI host qualification on {args.backend}: " + + ("passed" if not failed else f"failed ({', '.join(failed)})") + + (f"; generation {generation}" if generation else "") + + (f", profile {profile}" if profile else ""), + flush=True, + ) + if args.summary is not None: + with args.summary.open("a", encoding="utf-8") as summary: + summary.write(summary_markdown(args.backend, results, context.facts)) + return 1 if failed else 0 + + +def default_probe_directory() -> Path: + tool_cache = os.environ.get("RUNNER_TOOL_CACHE") + if tool_cache: + return Path(tool_cache) / "nvx-host-time-probe" + return BuildConstants.BUILD_DIR / "host-time-probe" + + +def configure_parser(parser: argparse.ArgumentParser) -> None: + parser.description = ( + "Qualify this host for the NVX time ABI (doc/design/time-abi.md, " + "'Host qualification')." + ) + parser.add_argument("--backend", choices=OPENVMM_TEST_BACKENDS, required=True) + parser.add_argument( + "--checks", + nargs="+", + choices=CHECK_IDS, + metavar="ID", + help="checks to run, in spec order (default: H1 to H7)", + ) + parser.add_argument("--openvmm", type=Path, help="OpenVMM binary for H3 and H6") + parser.add_argument("--kernel", type=Path, help="guest kernel for H6") + parser.add_argument("--initrd", type=Path, help="guest initramfs for H6") + parser.add_argument( + "--probe-dir", + type=Path, + help="cache directory for the host probe binary " + "(default: $RUNNER_TOOL_CACHE or build/)", + ) + parser.add_argument( + "--summary", + type=Path, + help="append a Markdown summary, for example $GITHUB_STEP_SUMMARY", + ) + parser.add_argument( + "--timeout", + type=float, + default=120.0, + help="seconds allowed for each probe or guest (default: 120)", + ) + parser.add_argument( + "--openvmm-arg", + action="append", + default=[], + help=argparse.SUPPRESS, + ) + parser.set_defaults(handler=run) diff --git a/scripts/nvx_tools/host_time_probe.rs b/scripts/nvx_tools/host_time_probe.rs new file mode 100644 index 000000000..80d013cf4 --- /dev/null +++ b/scripts/nvx_tools/host_time_probe.rs @@ -0,0 +1,516 @@ +//! Host TSC probe for NVX time ABI host qualification. +//! +//! `nvx.py doctor` builds this file with `rustc` and runs it for the host +//! checks of doc/design/time-abi.md: H2 (the CPU identity and the host's +//! invariant-TSC bit), H4 (the TSC rate against host monotonic time), and H5 +//! (cross-CPU TSC skew between pinned threads). It has no dependencies and +//! prints `NVX-HOST-TIME-PROBE` lines of `key=value` fields; the doctor +//! applies the bounds. + +use std::process::ExitCode; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::thread; +use std::time::{Duration, Instant}; + +const PREFIX: &str = "NVX-HOST-TIME-PROBE"; +const STALL: Duration = Duration::from_secs(2); +const EXIT_USAGE: u8 = 2; +const EXIT_ERROR: u8 = 3; + +#[inline(always)] +fn tsc() -> u64 { + // SAFETY: LFENCE and RDTSC are baseline x86-64 instructions. + unsafe { + core::arch::x86_64::_mm_lfence(); + let value = core::arch::x86_64::_rdtsc(); + core::arch::x86_64::_mm_lfence(); + value + } +} + +fn cpuid(leaf: u32) -> [u32; 4] { + // SAFETY: CPUID is a baseline x86-64 instruction. Newer Rust releases + // declare the intrinsic safe, older ones unsafe. + #[allow(unused_unsafe)] + let result = unsafe { core::arch::x86_64::__cpuid_count(leaf, 0) }; + [result.eax, result.ebx, result.ecx, result.edx] +} + +#[cfg(target_os = "linux")] +mod affinity { + const SET_BYTES: usize = 128; + + unsafe extern "C" { + fn sched_setaffinity(pid: i32, size: usize, mask: *const u8) -> i32; + fn sched_getaffinity(pid: i32, size: usize, mask: *mut u8) -> i32; + } + + pub fn allowed() -> Result, String> { + let mut mask = [0u8; SET_BYTES]; + // SAFETY: the mask buffer holds SET_BYTES bytes. + if unsafe { sched_getaffinity(0, SET_BYTES, mask.as_mut_ptr()) } != 0 { + return Err(format!( + "sched_getaffinity failed: {}", + std::io::Error::last_os_error() + )); + } + Ok((0..SET_BYTES * 8) + .filter(|cpu| mask[cpu / 8] & (1 << (cpu % 8)) != 0) + .collect()) + } + + /// Pins the calling thread to one CPU. + pub fn pin(cpu: usize) -> Result<(), String> { + if cpu >= SET_BYTES * 8 { + return Err(format!("CPU {cpu} is out of range")); + } + let mut mask = [0u8; SET_BYTES]; + mask[cpu / 8] |= 1 << (cpu % 8); + // SAFETY: pid 0 selects the calling thread; the mask holds SET_BYTES. + if unsafe { sched_setaffinity(0, SET_BYTES, mask.as_ptr()) } != 0 { + return Err(format!( + "cannot pin to CPU {cpu}: {}", + std::io::Error::last_os_error() + )); + } + std::thread::yield_now(); + Ok(()) + } +} + +#[cfg(windows)] +mod affinity { + type Handle = *mut core::ffi::c_void; + + #[link(name = "kernel32")] + unsafe extern "system" { + fn GetCurrentThread() -> Handle; + fn GetCurrentProcess() -> Handle; + fn SetThreadAffinityMask(thread: Handle, mask: usize) -> usize; + fn GetProcessAffinityMask(process: Handle, process_mask: *mut usize, system_mask: *mut usize) -> i32; + } + + pub fn allowed() -> Result, String> { + let mut process = 0usize; + let mut system = 0usize; + // SAFETY: both outputs point to valid usize values. + if unsafe { GetProcessAffinityMask(GetCurrentProcess(), &mut process, &mut system) } == 0 { + return Err(format!( + "GetProcessAffinityMask failed: {}", + std::io::Error::last_os_error() + )); + } + Ok((0..usize::BITS as usize) + .filter(|cpu| process & (1usize << cpu) != 0) + .collect()) + } + + /// Pins the calling thread to one CPU of the process's processor group. + pub fn pin(cpu: usize) -> Result<(), String> { + if cpu >= usize::BITS as usize { + return Err(format!("CPU {cpu} is out of range")); + } + // SAFETY: GetCurrentThread returns a pseudo-handle for this thread. + if unsafe { SetThreadAffinityMask(GetCurrentThread(), 1usize << cpu) } == 0 { + return Err(format!( + "cannot pin to CPU {cpu}: {}", + std::io::Error::last_os_error() + )); + } + std::thread::yield_now(); + Ok(()) + } +} + +fn option<'a>(arguments: &'a [String], name: &str) -> Result, String> { + let mut value = None; + let mut index = 0; + while index < arguments.len() { + if arguments[index] == name { + let next = arguments + .get(index + 1) + .ok_or_else(|| format!("{name} requires a value"))?; + value = Some(next.as_str()); + index += 2; + } else if arguments[index].starts_with("--") { + index += 2; + } else { + return Err(format!("unexpected argument {}", arguments[index])); + } + } + Ok(value) +} + +fn number(arguments: &[String], name: &str, default: T) -> Result { + match option(arguments, name)? { + Some(text) => text + .parse() + .map_err(|_| format!("{name} has an invalid value {text:?}")), + None => Ok(default), + } +} + +fn check_options(arguments: &[String], known: &[&str]) -> Result<(), String> { + for argument in arguments.iter().step_by(2) { + if !known.contains(&argument.as_str()) { + return Err(format!("unknown option {argument}")); + } + } + Ok(()) +} + +fn parse_cpus(text: &str) -> Result, String> { + let mut cpus = Vec::new(); + for part in text.split(',') { + let (start, end) = match part.split_once('-') { + Some((start, end)) => (start, end), + None => (part, part), + }; + let start: usize = start.parse().map_err(|_| format!("invalid CPU list {text:?}"))?; + let end: usize = end.parse().map_err(|_| format!("invalid CPU list {text:?}"))?; + if end < start { + return Err(format!("invalid CPU range {part:?}")); + } + cpus.extend(start..=end); + } + cpus.sort_unstable(); + cpus.dedup(); + Ok(cpus) +} + +fn format_cpus(cpus: &[usize]) -> String { + let mut parts = Vec::new(); + let mut index = 0; + while index < cpus.len() { + let start = cpus[index]; + let mut end = start; + while index + 1 < cpus.len() && cpus[index + 1] == end + 1 { + index += 1; + end = cpus[index]; + } + parts.push(if start == end { + start.to_string() + } else { + format!("{start}-{end}") + }); + index += 1; + } + parts.join(",") +} + +fn cpu() -> Result<(), String> { + let [max_basic, ebx, ecx, edx] = cpuid(0); + let mut vendor = Vec::new(); + for register in [ebx, edx, ecx] { + vendor.extend_from_slice(®ister.to_le_bytes()); + } + let [signature, _, features_ecx, _] = if max_basic >= 1 { cpuid(1) } else { [0; 4] }; + let base_family = (signature >> 8) & 0xf; + let family = if base_family == 0xf { + base_family + ((signature >> 20) & 0xff) + } else { + base_family + }; + let model = if base_family == 0x6 || base_family == 0xf { + ((signature >> 4) & 0xf) | (((signature >> 16) & 0xf) << 4) + } else { + (signature >> 4) & 0xf + }; + let max_extended = cpuid(0x8000_0000)[0]; + let invariant_tsc = max_extended >= 0x8000_0007 && (cpuid(0x8000_0007)[3] >> 8) & 1 == 1; + let mut brand = Vec::new(); + if max_extended >= 0x8000_0004 { + for leaf in 0x8000_0002..=0x8000_0004 { + for register in cpuid(leaf) { + brand.extend_from_slice(®ister.to_le_bytes()); + } + } + } + let brand: String = String::from_utf8_lossy(&brand) + .trim_matches(char::from(0)) + .trim() + .replace(' ', "_"); + println!( + "{PREFIX} cpu vendor={} family={family} model={model} stepping={} signature=0x{signature:08x} \ + invariant_tsc={} hypervisor={} brand={brand}", + String::from_utf8_lossy(&vendor), + signature & 0xf, + u8::from(invariant_tsc), + (features_ecx >> 31) & 1, + ); + Ok(()) +} + +/// Pairs a TSC reading with host monotonic time; returns the narrowest of +/// several brackets as (TSC at the midpoint, instant, bracket width). +fn paired_sample() -> (u64, Instant, u64) { + let mut best: Option<(u64, Instant, u64)> = None; + for _ in 0..64 { + let before = tsc(); + let now = Instant::now(); + let after = tsc(); + let width = after.wrapping_sub(before); + if best.is_none_or(|(_, _, narrowest)| width < narrowest) { + best = Some((before.wrapping_add(width / 2), now, width)); + } + } + best.expect("at least one sample") +} + +fn rate(arguments: &[String]) -> Result<(), String> { + check_options(arguments, &["--measure-ms", "--count"])?; + let measure_ms: u64 = number(arguments, "--measure-ms", 1000)?; + let count: u32 = number(arguments, "--count", 2)?; + if measure_ms == 0 || count == 0 { + return Err("--measure-ms and --count must be positive".into()); + } + // One CPU's TSC against host monotonic time; cross-CPU skew is H5's. + let first = *affinity::allowed()?.first().ok_or("no CPU is allowed")?; + affinity::pin(first)?; + for index in 1..=count { + let (start_tsc, start, start_width) = paired_sample(); + thread::sleep(Duration::from_millis(measure_ms)); + let (end_tsc, end, end_width) = paired_sample(); + let elapsed_ns = end.duration_since(start).as_nanos(); + let cycles = end_tsc.wrapping_sub(start_tsc); + let hz = cycles as f64 * 1e9 / elapsed_ns as f64; + println!( + "{PREFIX} rate index={index} cpu={first} tsc_hz={hz:.3} elapsed_ns={elapsed_ns} \ + cycles={cycles} bracket_cycles={}", + start_width.max(end_width), + ); + } + Ok(()) +} + +struct Shared { + sequence: AtomicU64, + responder_tsc: AtomicU64, + ready: AtomicBool, + stop: AtomicBool, +} + +struct Pair { + rounds: u64, + offset_cycles: f64, + uncertainty_cycles: f64, + min_rtt_cycles: u64, + consistent: bool, + stalled: bool, +} + +impl Pair { + fn stalled(rounds: u64) -> Self { + Pair { + rounds, + offset_cycles: 0.0, + uncertainty_cycles: 0.0, + min_rtt_cycles: 0, + consistent: false, + stalled: true, + } + } +} + +/// Answers ping-pong rounds on CPU `cpu` until the initiator stops. +fn respond(shared: &Shared, cpu: usize) -> Result<(), String> { + affinity::pin(cpu)?; + shared.ready.store(true, Ordering::Release); + let mut expected = 1u64; + loop { + while shared.sequence.load(Ordering::Acquire) != expected { + if shared.stop.load(Ordering::Relaxed) { + return Ok(()); + } + std::hint::spin_loop(); + } + shared.responder_tsc.store(tsc(), Ordering::Relaxed); + shared.sequence.store(expected + 1, Ordering::Release); + expected += 2; + } +} + +/// Spins until `done` returns true; false if `STALL` passed first. +fn spin_until(mut done: impl FnMut() -> bool) -> bool { + let mut spins = 0u32; + let mut started: Option = None; + while !done() { + spins = spins.wrapping_add(1); + if spins & 0xffff == 0 && started.get_or_insert_with(Instant::now).elapsed() > STALL { + return false; + } + std::hint::spin_loop(); + } + true +} + +/// Estimates the responder's TSC offset from CPU `cpu`. Each round bounds it +/// to [t2 - t3, t2 - t1]; consistent rounds intersect to a tight interval, +/// otherwise the minimum round trip gives the estimate. +fn initiate(shared: &Shared, cpu: usize, duration: Duration) -> Result { + affinity::pin(cpu)?; + if !spin_until(|| shared.ready.load(Ordering::Acquire)) { + return Ok(Pair::stalled(0)); + } + let deadline = Instant::now() + duration; + let mut sequence = 1u64; + let mut rounds = 0u64; + let mut lower = i128::MIN; + let mut upper = i128::MAX; + let mut min_rtt = u64::MAX; + let mut min_rtt_offset = 0.0; + loop { + let t1 = tsc(); + shared.sequence.store(sequence, Ordering::Release); + if !spin_until(|| shared.sequence.load(Ordering::Acquire) == sequence + 1) { + return Ok(Pair::stalled(rounds)); + } + let t3 = tsc(); + let t2 = shared.responder_tsc.load(Ordering::Relaxed); + sequence += 2; + rounds += 1; + let (t1, t2, t3) = (i128::from(t1), i128::from(t2), i128::from(t3)); + lower = lower.max(t2 - t3); + upper = upper.min(t2 - t1); + let rtt = (t3 - t1) as u64; + if rtt < min_rtt { + min_rtt = rtt; + min_rtt_offset = (2 * t2 - t1 - t3) as f64 / 2.0; + } + if rounds % 256 == 0 && Instant::now() >= deadline { + break; + } + } + let consistent = lower <= upper; + let (offset_cycles, uncertainty_cycles) = if consistent { + ((lower + upper) as f64 / 2.0, (upper - lower) as f64 / 2.0) + } else { + (min_rtt_offset, min_rtt as f64 / 2.0) + }; + Ok(Pair { + rounds, + offset_cycles, + uncertainty_cycles, + min_rtt_cycles: min_rtt, + consistent, + stalled: false, + }) +} + +fn measure_pair(initiator: usize, responder: usize, duration: Duration) -> Result { + let shared = Arc::new(Shared { + sequence: AtomicU64::new(0), + responder_tsc: AtomicU64::new(0), + ready: AtomicBool::new(false), + stop: AtomicBool::new(false), + }); + let peer = Arc::clone(&shared); + let handle = thread::spawn(move || respond(&peer, responder)); + let result = initiate(&shared, initiator, duration); + shared.stop.store(true, Ordering::Relaxed); + handle + .join() + .map_err(|_| String::from("the responder thread panicked"))??; + result +} + +fn skew(arguments: &[String]) -> Result<(), String> { + check_options(arguments, &["--duration-ms", "--cpus", "--tsc-hz", "--bound-ns"])?; + let duration_ms: u64 = number(arguments, "--duration-ms", 5)?; + let bound_ns: f64 = number(arguments, "--bound-ns", 1000.0)?; + let allowed = affinity::allowed()?; + let cpus = match option(arguments, "--cpus")? { + Some(text) => parse_cpus(text)?, + None => allowed.clone(), + }; + if let Some(cpu) = cpus.iter().find(|cpu| !allowed.contains(cpu)) { + return Err(format!("CPU {cpu} is not in the process affinity mask")); + } + let tsc_hz = match option(arguments, "--tsc-hz")? { + Some(text) => text + .parse::() + .map_err(|_| format!("--tsc-hz has an invalid value {text:?}"))?, + None => { + affinity::pin(cpus[0])?; + let (start_tsc, start, _) = paired_sample(); + thread::sleep(Duration::from_millis(100)); + let (end_tsc, end, _) = paired_sample(); + end_tsc.wrapping_sub(start_tsc) as f64 * 1e9 / end.duration_since(start).as_nanos() as f64 + } + }; + let to_ns = |cycles: f64| cycles * 1e9 / tsc_hz; + let duration = Duration::from_millis(duration_ms.max(1)); + let mut pairs = 0; + let mut stalled = 0; + let mut inconsistent = 0; + let mut max_offset = 0.0f64; + let mut max_uncertainty = 0.0f64; + for (index, &first) in cpus.iter().enumerate() { + for &second in &cpus[index + 1..] { + let pair = measure_pair(first, second, duration)?; + pairs += 1; + if pair.stalled { + stalled += 1; + println!("{PREFIX} pair cpus={first}-{second} rounds={} stalled=1", pair.rounds); + continue; + } + inconsistent += usize::from(!pair.consistent); + max_offset = max_offset.max(to_ns(pair.offset_cycles).abs()); + max_uncertainty = max_uncertainty.max(to_ns(pair.uncertainty_cycles)); + println!( + "{PREFIX} pair cpus={first}-{second} rounds={} offset_ns={:.1} uncertainty_ns={:.1} \ + min_rtt_ns={:.1} consistent={} stalled=0", + pair.rounds, + to_ns(pair.offset_cycles), + to_ns(pair.uncertainty_cycles), + to_ns(pair.min_rtt_cycles as f64), + u8::from(pair.consistent), + ); + } + } + let conclusive = stalled == 0 && max_uncertainty < bound_ns; + println!( + "{PREFIX} skew pairs={pairs} cpus={} max_abs_offset_ns={:.0} max_uncertainty_ns={:.0} \ + stalled_pairs={stalled} inconsistent_pairs={inconsistent} duration_ms={duration_ms} \ + tsc_hz={tsc_hz:.0} conclusive={}", + format_cpus(&cpus), + max_offset.round(), + max_uncertainty.round(), + u8::from(conclusive), + ); + Ok(()) +} + +fn usage() -> String { + format!( + "usage: {} cpu\n {} rate [--measure-ms N] [--count K]\n \ + {} skew [--duration-ms N] [--cpus LIST] [--tsc-hz HZ] [--bound-ns N]", + "nvx-host-time-probe", "nvx-host-time-probe", "nvx-host-time-probe" + ) +} + +fn main() -> ExitCode { + let arguments: Vec = std::env::args().skip(1).collect(); + let Some(command) = arguments.first() else { + eprintln!("{}", usage()); + return ExitCode::from(EXIT_USAGE); + }; + let options = &arguments[1..]; + let result = match command.as_str() { + "cpu" if options.is_empty() => cpu(), + "rate" => rate(options), + "skew" => skew(options), + _ => { + eprintln!("{}", usage()); + return ExitCode::from(EXIT_USAGE); + } + }; + match result { + Ok(()) => ExitCode::SUCCESS, + Err(error) => { + eprintln!("nvx-host-time-probe: {error}"); + ExitCode::from(EXIT_ERROR) + } + } +} diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 9e2742f6f..37b1c36e0 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -10,6 +10,7 @@ from __future__ import annotations from collections.abc import Mapping, Sequence +from dataclasses import dataclass MARKER_PREFIX = "NVX-TIME-ABI: " VIOLATION_PREFIX = "NVX-TIME-ABI-VIOLATION: " @@ -31,11 +32,32 @@ WARP_PROBE_PATH = "/sbin/nvx-time-probe" WARP_SUMMARY_PREFIX = "NVX-TIME-PROBE warp " WARP_DETAIL_PREFIX = "NVX-TIME-PROBE warp-detail " -CPU_GENERATIONS: Mapping[tuple[str, int, int], str] = { - ("GenuineIntel", 6, 85): "skylake-sp", - ("GenuineIntel", 6, 106): "icelake-sp", - ("GenuineIntel", 6, 207): "emeraldrapids", -} +PROFILE_VENDORS: Mapping[str, str] = {"GenuineIntel": "intel"} + + +@dataclass(frozen=True) +class CpuGeneration: + """One CPU generation of the spec's CPU profile catalog.""" + + name: str + vendor: str + family: int + model: int + steppings: range + + @property + def profile_id(self) -> str: + """The catalog's v1 profile, which every backend shares.""" + return f"{PROFILE_VENDORS[self.vendor]}.{self.name}.v1" + + +# Model 85 also covers Cascade Lake (steppings 5-7) and Cooper Lake (10-11), +# which have no profile. +CPU_GENERATIONS: tuple[CpuGeneration, ...] = ( + CpuGeneration("skylake-sp", "GenuineIntel", 6, 85, range(5)), + CpuGeneration("icelake-sp", "GenuineIntel", 6, 106, range(16)), + CpuGeneration("emeraldrapids", "GenuineIntel", 6, 207, range(16)), +) # Guest boot markers that init prints after the time ABI boot check. GUEST_BOOT_MARKERS = ("ALPINE-MICROVM-BOOT-OK", "NVX-GUEST-BOOT-OK:") _MAX_PENDING_LINE = 64 * 1024 @@ -45,9 +67,19 @@ class TimeAbiFailure(RuntimeError): """Raised when a guest or its console output violates the time ABI.""" -def cpu_generation(vendor: str, family: int, model: int) -> str | None: - """Return the time ABI generation name of a CPU, or None if unknown.""" - return CPU_GENERATIONS.get((vendor, family, model)) +def cpu_generation( + vendor: str, family: int, model: int, stepping: int +) -> CpuGeneration | None: + """Return the time ABI generation of a CPU, or None if it has none.""" + for generation in CPU_GENERATIONS: + if ( + generation.vendor == vendor + and generation.family == family + and generation.model == model + and stepping in generation.steppings + ): + return generation + return None def parse_fields(text: str) -> dict[str, str]: diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index ffde21510..7719ec65a 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -2,8 +2,11 @@ from __future__ import annotations +import contextlib +import io import queue import socket +import subprocess import sys import tempfile import unittest @@ -12,7 +15,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) -from nvx_tools import benchmark, openvmm_process, time_abi # noqa: E402 +from nvx_tools import benchmark, doctor, openvmm_process, time_abi # noqa: E402 from nvx_tools.time_abi import TimeAbiFailure, TimeAbiMonitor # noqa: E402 BOOT_LINE = ( @@ -96,13 +99,25 @@ def test_describes_only_time_abi_exit_statuses(self): self.assertIsNone(time_abi.describe_exit_status(status)) def test_maps_cpu_generations(self): - self.assertEqual(time_abi.cpu_generation("GenuineIntel", 6, 85), "skylake-sp") - self.assertEqual(time_abi.cpu_generation("GenuineIntel", 6, 106), "icelake-sp") + def name(model: int, stepping: int, vendor: str = "GenuineIntel"): + generation = time_abi.cpu_generation(vendor, 6, model, stepping) + return generation.name if generation else None + + self.assertEqual(name(85, 4), "skylake-sp") + self.assertEqual(name(106, 6), "icelake-sp") + self.assertEqual(name(207, 2), "emeraldrapids") + # Cascade Lake and Cooper Lake share model 85 but have no profile. + self.assertIsNone(name(85, 7)) + self.assertIsNone(name(85, 11)) + self.assertIsNone(name(143, 8)) + self.assertIsNone(name(1, 1, "AuthenticAMD")) + + def test_names_the_catalog_profiles(self): + # One profile per generation serves every backend. self.assertEqual( - time_abi.cpu_generation("GenuineIntel", 6, 207), "emeraldrapids" + [generation.profile_id for generation in time_abi.CPU_GENERATIONS], + ["intel.skylake-sp.v1", "intel.icelake-sp.v1", "intel.emeraldrapids.v1"], ) - self.assertIsNone(time_abi.cpu_generation("GenuineIntel", 6, 143)) - self.assertIsNone(time_abi.cpu_generation("AuthenticAMD", 25, 1)) class MonitorTests(unittest.TestCase): @@ -462,5 +477,464 @@ def test_tcp_console_monitor_reports_violations(self): peer.close() +def doctor_context(root: Path, backend: str = "kvm") -> doctor.DoctorContext: + return doctor.DoctorContext( + backend=backend, + openvmm=root / "openvmm", + kernel=root / "vmlinux", + initrd=root / "initramfs.cpio.gz", + openvmm_args=(), + probe_directory=root / "probe", + timeout=30, + ) + + +def completed(stdout: str, returncode: int = 0, stderr: str = ""): + return subprocess.CompletedProcess(["x"], returncode, stdout, stderr) + + +CPU = { + "vendor": "GenuineIntel", + "family": "6", + "model": "106", + "stepping": "6", + "microcode": "0xffffffff", + "brand": "Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz", + "os": "Linux 6.6.150.1-1.azl3", + "invariant_tsc": "yes", +} + + +class DoctorTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.root = Path(self.temporary.name) + + def tearDown(self): + self.temporary.cleanup() + + def test_check_lines_escape_details_for_field_parsers(self): + line = doctor.CheckResult("H5", False, 'say "x" \\ y').line() + self.assertTrue(line.startswith("NVX-DOCTOR: check=H5 status=fail ")) + fields = time_abi.parse_fields(line.removeprefix(doctor.DOCTOR_PREFIX)) + self.assertEqual(fields["detail"], 'say "x" \\ y') + + def test_backend_check_requires_a_usable_device(self): + context = doctor_context(self.root) + existing: set[str] = set() + + def exists(path: Path) -> bool: + return path.as_posix() in existing + + with ( + patch.object(doctor, "host_is_windows", return_value=False), + patch.object(doctor.Path, "exists", exists), + patch.object(doctor.os, "access", return_value=True) as access, + ): + self.assertIn("does not exist", doctor.check_backend(context).detail) + existing.add("/dev/kvm") + self.assertTrue(doctor.check_backend(context).passed) + existing.add("/dev/mshv") + self.assertIn("select MSHV", doctor.check_backend(context).detail) + existing.discard("/dev/mshv") + access.return_value = False + self.assertIn("not readable", doctor.check_backend(context).detail) + with patch.object(doctor, "host_is_windows", return_value=False): + whp = doctor.check_backend(doctor_context(self.root, "whp")) + self.assertFalse(whp.passed) + self.assertIn("requires Windows", whp.detail) + + def test_cpu_check_names_the_generation_and_applies_the_invariant_tsc_rule(self): + context = doctor_context(self.root) + with patch.object(doctor, "host_cpu", return_value=dict(CPU)): + result = doctor.check_cpu(context) + self.assertTrue(result.passed) + self.assertIn("generation=icelake-sp", result.detail) + self.assertIn("profile=intel.icelake-sp.v1", result.detail) + self.assertEqual(context.facts["generation"], "icelake-sp") + for unknown in (dict(CPU, model="143"), dict(CPU, model="85", stepping="7")): + with patch.object(doctor, "host_cpu", return_value=unknown): + result = doctor.check_cpu(doctor_context(self.root)) + self.assertFalse(result.passed) + self.assertTrue(result.detail.startswith("[E_PROFILE_HOST_UNKNOWN] ")) + emerald = dict(CPU, model="207", stepping="2") + with patch.object(doctor, "host_cpu", return_value=emerald): + for backend in ("kvm", "mshv", "whp"): + result = doctor.check_cpu(doctor_context(self.root, backend)) + self.assertTrue(result.passed, result.detail) + self.assertIn("profile=intel.emeraldrapids.v1", result.detail) + variant = dict(CPU, invariant_tsc="no (missing nonstop_tsc)") + with patch.object(doctor, "host_cpu", return_value=variant): + result = doctor.check_cpu(doctor_context(self.root, "mshv")) + self.assertTrue(result.detail.startswith("[E_PROFILE_UNSUPPORTED] ")) + # A WHP host qualifies on measured properties instead. + self.assertTrue(doctor.check_cpu(doctor_context(self.root, "whp")).passed) + + def test_reads_the_first_processor_from_cpuinfo(self): + path = self.root / "cpuinfo" + path.write_text( + "processor\t: 0\nvendor_id\t: GenuineIntel\ncpu family\t: 6\n" + "model\t\t: 85\nstepping\t: 4\nmicrocode\t: 0x2007108\n" + "flags\t\t: fpu tsc constant_tsc nonstop_tsc\n\n" + "processor\t: 1\nvendor_id\t: Other\n", + encoding="utf-8", + ) + fields = doctor._linux_cpuinfo(path) # pyright: ignore[reportPrivateUsage] + self.assertEqual(fields["vendor_id"], "GenuineIntel") + self.assertEqual(fields["model"], "85") + self.assertIn("nonstop_tsc", fields["flags"]) + + def test_preflight_parses_openvmm_verification(self): + # The line format of OpenVMM's openvmm_entry verify.rs. + line = ( + "NVX-TIME-ABI-VERIFY: v=1 status=ok backend=kvm " + "cpu_profile=intel.icelake-sp.v1 tsc_hz=2793437000 " + "native_tsc_hz=2793437000 lapic_hz=1000000000 msr_route=ExitToVmm " + "sync=CommonOffset\n" + ) + + def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: + context = doctor_context(self.root, backend) + for path in (context.openvmm, context.kernel, context.initrd): + path.write_bytes(b"") + return context + + context = context_with_files() + with patch.object( + doctor.subprocess, "run", return_value=completed(line) + ) as run: + result = doctor.check_openvmm_preflight(context) + self.assertTrue(result.passed, result.detail) + self.assertEqual(context.facts["native_tsc_hz"], "2793437000") + self.assertEqual(context.facts["profile"], "intel.icelake-sp.v1") + command = run.call_args.args[0] + self.assertEqual(command[-1], "--x-time-abi-verify") + self.assertNotIn("--x-time-abi-v1", command) + self.assertEqual(command[command.index("--kernel") + 1], str(context.kernel)) + switched = context_with_files() + switched.openvmm_args = ("--x-time-abi-v1",) + with patch.object( + doctor.subprocess, "run", return_value=completed(line) + ) as run: + self.assertTrue(doctor.check_openvmm_preflight(switched).passed) + self.assertEqual( + run.call_args.args[0][-2:], ["--x-time-abi-v1", "--x-time-abi-verify"] + ) + failure = ( + "NVX-TIME-ABI-VERIFY: v=1 status=fail backend=kvm " + 'code=E_TSC_SYNC_UNSUPPORTED detail="failed: \\"sync\\" unsupported"\n' + ) + cases = ( + ( + completed(failure, 1), + '[E_TSC_SYNC_UNSUPPORTED] OpenVMM verification failed (exit 1): failed: "sync" unsupported', + ), + ( + completed("", 2, "error: unexpected argument '--x-time-abi-v1' found"), + "predates the time ABI", + ), + (completed(line.replace("1000000000", "200000000")), "[E_LAPIC_RATE_"), + ( + completed(line.replace("tsc_hz=2793437000", "tsc_hz=4000", 1)), + "[E_TSC_RATE_", + ), + ( + completed(line.replace("backend=kvm", "backend=mshv")), + "verified backend mshv", + ), + ) + for result_value, message in cases: + with self.subTest(message=message): + with patch.object(doctor.subprocess, "run", return_value=result_value): + result = doctor.check_openvmm_preflight(context_with_files()) + self.assertFalse(result.passed) + self.assertIn(message, result.detail) + for selected, passed in ( + ("intel.icelake-sp.v2", True), + ("intel.skylake-sp.v1", False), + # OpenVMM's interim host profile until it selects catalog profiles. + ("interim.host.kvm.v1", True), + ("interim.host.mshv.v1", False), + ): + with self.subTest(selected=selected): + context = context_with_files() + context.facts["profile"] = "intel.icelake-sp.v1" + output = line.replace("intel.icelake-sp.v1", selected) + with patch.object( + doctor.subprocess, "run", return_value=completed(output) + ): + result = doctor.check_openvmm_preflight(context) + self.assertEqual(result.passed, passed, result.detail) + self.assertEqual(context.facts["profile"], selected) + missing = doctor.check_openvmm_preflight(doctor_context(self.root / "none")) + self.assertFalse(missing.passed) + self.assertIn("was not found", missing.detail) + + def test_rate_check_requires_stability_the_backend_rate_and_a_tsc_clocksource(self): + def records(first: float, second: float): + return [ + ("rate", {"tsc_hz": f"{first:.3f}"}), + ("rate", {"tsc_hz": f"{second:.3f}"}), + ] + + context = doctor_context(self.root) + context.facts["tsc_hz"] = "2793437000" + with ( + patch.object( + doctor, "run_probe", return_value=records(2793437500, 2793437800) + ), + patch.object(doctor, "host_clocksource", return_value="tsc"), + ): + result = doctor.check_rate(context) + self.assertTrue(result.passed, result.detail) + cases = ( + (records(2793437000, 2793440000), "tsc", "unstable"), + (records(2794000000, 2794000001), "tsc", "ppm from the measured"), + (records(2793437000, 2793437000), "hpet", "clocksource is hpet"), + ) + for probe_records, clocksource, message in cases: + with self.subTest(message=message): + context = doctor_context(self.root) + context.facts["tsc_hz"] = "2793437000" + with ( + patch.object(doctor, "run_probe", return_value=probe_records), + patch.object(doctor, "host_clocksource", return_value=clocksource), + ): + result = doctor.check_rate(context) + self.assertFalse(result.passed) + self.assertIn(message, result.detail) + mshv = doctor_context(self.root, "mshv") + mshv.facts["tsc_hz"] = "2300000000" + with ( + patch.object( + doctor, "run_probe", return_value=records(2300000100, 2300000200) + ), + patch.object( + doctor, "host_clocksource", return_value="hyperv_clocksource_tsc_page" + ), + ): + self.assertTrue(doctor.check_rate(mshv).passed) + with ( + patch.object( + doctor, "run_probe", return_value=records(2300000100, 2300000200) + ), + patch.object(doctor, "host_clocksource", return_value="tsc"), + ): + standalone = doctor.check_rate(doctor_context(self.root)) + # Without H3 (validate-runner), H4 still checks stability. + self.assertTrue(standalone.passed, standalone.detail) + self.assertIn("not compared with the backend's rate", standalone.detail) + with ( + patch.object( + doctor, "run_probe", return_value=records(2300000100, 2300009100) + ), + patch.object(doctor, "host_clocksource", return_value="tsc"), + ): + unstable = doctor.check_rate(doctor_context(self.root)) + self.assertFalse(unstable.passed) + self.assertIn("unstable", unstable.detail) + + def test_host_skew_check_applies_the_bound(self): + summary = { + "pairs": "28", + "cpus": "0-7", + "max_abs_offset_ns": "40", + "max_uncertainty_ns": "120", + "stalled_pairs": "0", + "conclusive": "1", + } + context = doctor_context(self.root) + context.facts["measured_tsc_hz"] = "2793437000" + with patch.object(doctor, "run_probe", return_value=[("skew", summary)]) as run: + self.assertTrue(doctor.check_host_skew(context).passed) + self.assertIn("--tsc-hz", run.call_args.args) + for changes, message in ( + ({"max_abs_offset_ns": "1500"}, "exceeds 1000"), + ({"stalled_pairs": "2", "conclusive": "0"}, "2 CPU pair(s) stalled"), + ({"conclusive": "0"}, "inconclusive"), + ): + with ( + self.subTest(message=message), + patch.object( + doctor, "run_probe", return_value=[("skew", summary | changes)] + ), + ): + result = doctor.check_host_skew(doctor_context(self.root)) + self.assertFalse(result.passed) + self.assertIn(message, result.detail) + + def test_guest_warp_check_requires_the_boot_marker_and_the_skew_bound(self): + context = doctor_context(self.root) + for path in (context.openvmm, context.kernel, context.initrd): + path.write_bytes(b"") + text = ( + BOOT_LINE.replace("cpus=4", "cpus=8") + + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + + warp_output(pairs=28) + ) + with ( + patch.object(doctor.os, "cpu_count", return_value=8), + patch.object( + doctor, "run_guest_script", return_value={"text": text} + ) as run, + ): + result = doctor.check_guest_warp(context) + self.assertTrue(result.passed, result.detail) + self.assertEqual(context.facts["guest_warp_ns"], "15") + command = run.call_args.args[0] + self.assertEqual(command[command.index("--processors") + 1], "8") + with ( + patch.object(doctor.os, "cpu_count", return_value=6), + patch.object( + doctor, + "run_guest_script", + return_value={"text": warp_output(pairs=6)}, + ), + ): + result = doctor.check_guest_warp(doctor_context(self.root)) + self.assertFalse(result.passed) + self.assertIn("vcpus=4", result.detail) + self.assertIn("NVX-TIME-ABI boot marker", result.detail) + missing = doctor.check_guest_warp(doctor_context(self.root / "missing")) + self.assertIn("OpenVMM was not found", missing.detail) + + def test_utc_check_reads_the_host_clock_discipline(self): + context = doctor_context(self.root) + with patch.object(doctor, "host_is_windows", return_value=False): + with patch.object(doctor, "_linux_clock_state", return_value=(0, 0x2001)): + self.assertTrue(doctor.check_utc(context).passed) + with patch.object(doctor, "_linux_clock_state", return_value=(5, 0x0041)): + self.assertIn("STA_UNSYNC", doctor.check_utc(context).detail) + status = ( + "Leap Indicator: 0(no warning)\nStratum: 2\n" + "Source: VM IC Time Synchronization Provider\n" + ) + with ( + patch.object(doctor, "host_is_windows", return_value=True), + patch.object(doctor.subprocess, "run", return_value=completed(status)), + ): + result = doctor.check_utc(context) + self.assertTrue(result.passed, result.detail) + self.assertIn("VM IC Time Synchronization Provider", result.detail) + for unsynchronized in ( + status.replace("0(no warning)", "3(not synchronized)"), + "Leap Indicator: 0\nSource: Local CMOS Clock\n", + ): + with ( + patch.object(doctor, "host_is_windows", return_value=True), + patch.object( + doctor.subprocess, "run", return_value=completed(unsynchronized) + ), + ): + self.assertFalse(doctor.check_utc(context).passed) + + def test_run_prints_lines_writes_the_summary_and_fails_closed(self): + def passing(check: str): + def run_check(context: doctor.DoctorContext) -> doctor.CheckResult: + context.facts["generation"] = "emeraldrapids" + return doctor.CheckResult(check, True, f"{check} ok") + + return run_check + + def broken(context: doctor.DoctorContext) -> doctor.CheckResult: + del context + raise doctor.ScriptError("rustc is required to build the host time probe") + + summary = self.root / "summary.md" + arguments = doctor_parser().parse_args( + [ + "--backend", + "mshv", + "--checks", + "H5", + "H2", + "--summary", + str(summary), + "--probe-dir", + str(self.root), + ] + ) + stdout = io.StringIO() + with ( + patch.dict(doctor.CHECKS, {"H2": passing("H2"), "H5": passing("H5")}), + contextlib.redirect_stdout(stdout), + ): + self.assertEqual(doctor.run(arguments), 0) + lines = stdout.getvalue().splitlines() + self.assertEqual( + [line.split()[1] for line in lines if line.startswith("NVX-DOCTOR")], + ["check=H2", "check=H5"], + ) + self.assertIn("passed; generation emeraldrapids", lines[-1]) + self.assertIn("(mshv): passed", summary.read_text(encoding="utf-8")) + with ( + patch.dict(doctor.CHECKS, {"H2": passing("H2"), "H5": broken}), + contextlib.redirect_stdout(io.StringIO()), + ): + self.assertEqual(doctor.run(arguments), 1) + text = summary.read_text(encoding="utf-8") + self.assertIn("**fail** | rustc is required", text) + self.assertIn("Generation: `emeraldrapids`", text) + + def test_builds_the_host_probe_once_per_source_version(self): + target = doctor.probe_path(self.root) + self.assertRegex(target.name, r"^nvx-host-time-probe-[0-9a-f]{16}(\.exe)?$") + + def rustc(command: list[str], **_kwargs: object): + Path(command[command.index("-o") + 1]).write_bytes(b"probe") + return completed("") + + with patch.object(doctor.shutil, "which", return_value=None): + with self.assertRaisesRegex(doctor.ScriptError, "rustc is required"): + doctor.build_probe(self.root) + with ( + patch.object(doctor.shutil, "which", return_value="rustc"), + patch.object(doctor.subprocess, "run", side_effect=rustc) as run, + ): + self.assertEqual(doctor.build_probe(self.root), target) + self.assertEqual(doctor.build_probe(self.root), target) + run.assert_called_once() + self.assertEqual(target.read_bytes(), b"probe") + target.unlink() + with ( + patch.object(doctor.shutil, "which", return_value="rustc"), + patch.object( + doctor.subprocess, "run", return_value=completed("", 1, "error[E0425]") + ), + ): + with self.assertRaisesRegex(doctor.ScriptError, "E0425"): + doctor.build_probe(self.root) + + def test_parses_probe_records(self): + context = doctor_context(self.root) + context.probe = self.root / "probe.exe" + output = ( + "noise\nNVX-HOST-TIME-PROBE rate index=1 tsc_hz=1.5\n" + "NVX-HOST-TIME-PROBE skew pairs=1 cpus=0-1\n" + ) + with patch.object(doctor.subprocess, "run", return_value=completed(output)): + records = doctor.run_probe(context, "skew") + self.assertEqual( + records, + [ + ("rate", {"index": "1", "tsc_hz": "1.5"}), + ("skew", {"pairs": "1", "cpus": "0-1"}), + ], + ) + with patch.object( + doctor.subprocess, "run", return_value=completed("", 3, "no CPU") + ): + with self.assertRaisesRegex(doctor.ScriptError, "skew failed: no CPU"): + doctor.run_probe(context, "skew") + + +def doctor_parser(): + import argparse + + parser = argparse.ArgumentParser() + doctor.configure_parser(parser) + return parser + + if __name__ == "__main__": unittest.main() From 8203988cc2a735f4c0d5e6e4a02b2d5b0ed8efa6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 07:10:27 -0700 Subject: [PATCH 020/188] ci: qualify runner host time with nvx.py doctor The time ABI's host qualification replaces validate-runner's nonstop_tsc check: every job now runs nvx.py doctor with H1 (backend device or API), H2 (CPU fingerprint, generation, the profile that auto selects, and the invariant-TSC rule), H4 (TSC rate stability and the host clocksource), H5 (host cross-CPU skew), and H7 (host UTC synchronized), on Linux and Windows runners alike. The doctor prints one NVX-DOCTOR line per check, appends the generation, profile, rates, and skew to the job summary, and fails with the spec's stable codes, for example E_PROFILE_HOST_UNKNOWN for an unknown generation. It takes a few seconds once the host probe is cached in the runner tool cache. H3 and H6 need the OpenVMM binary and guest artifacts, which jobs download after validate-runner; the smp scenario's warp probe covers H6 in every microVM job, and H3 waits for OpenVMM's --x-time-abi-verify. Runner labels stay as they are: per-PR jobs capture and restore on one runner. Refs #211 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 41 +++++++--------- doc/ci.md | 27 ++++++++--- scripts/test_nvx_tools.py | 55 +++++++++++++--------- 3 files changed, 71 insertions(+), 52 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index 568bbf6e6..b02f2ae39 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -117,31 +117,26 @@ runs: throw "Hyper-V SVGA firmware was not found: $svgaFirmware" } - - name: Validate host TSC + # Time ABI host qualification (doc/design/time-abi.md, "Host + # qualification"); it replaces the nonstop_tsc check. H3 and H6 need the + # OpenVMM binary and guest artifacts, which jobs download later. + - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash - run: | - set -euo pipefail - # Guests on a host without an invariant TSC intermittently see - # cross-vCPU TSC warps, which make Linux mark the guest TSC unstable - # (#211). - echo "Kernel: $(uname -r)" - echo "CPU: $(awk -F': ' '/^model name/ { print $2; exit }' /proc/cpuinfo)" - echo "Clocksource: $(cat /sys/devices/system/clocksource/clocksource0/current_clocksource 2>/dev/null || echo unknown)" - flags=" $(awk -F': ' '/^flags/ { print $2; exit }' /proc/cpuinfo) " - for flag in constant_tsc nonstop_tsc tsc_known_freq tsc_reliable; do - case "${flags}" in - *" ${flag} "*) echo "TSC flag ${flag}: present" ;; - *) echo "TSC flag ${flag}: absent" ;; - esac - done - case "${flags}" in - *" nonstop_tsc "*) ;; - *) - echo "::error::Runner host does not expose an invariant TSC (nonstop_tsc); redeploy the VM on a host that does" >&2 - exit 1 - ;; - esac + run: >- + python3 scripts/nvx.py doctor + --backend "${{ inputs.backend }}" + --checks H1 H2 H4 H5 H7 + --summary "${GITHUB_STEP_SUMMARY}" + + - name: Qualify host time on Windows + if: runner.os == 'Windows' + shell: powershell + run: >- + python scripts\nvx.py doctor + --backend "${{ inputs.backend }}" + --checks H1 H2 H4 H5 H7 + --summary "$env:GITHUB_STEP_SUMMARY" - name: Validate KVM if: inputs.backend == 'kvm' diff --git a/doc/ci.md b/doc/ci.md index 1ef1fb076..44b1da1cb 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -55,13 +55,20 @@ exposes an invariant TSC. The control log is kept as `restore-processors-tsc-control.log`. The control only classifies the failure; the restore still fails. -Linux runners must expose an invariant TSC, reported as `nonstop_tsc` in -`/proc/cpuinfo`. The `validate-runner` action prints each runner's kernel, CPU -model, clocksource, and TSC flags, and fails the job when `nonstop_tsc` is -missing. On an MSHV runner VM whose Azure host hid the invariant TSC, -never-restored guests also hit cross-vCPU TSC warps during CPU activation, and -keeping every host CPU out of idle removed them (#211). Redeploy such a VM on a -host that exposes an invariant TSC instead of retrying its jobs. +Every job that uses the `validate-runner` action first qualifies its runner +for the time ABI with `nvx.py doctor --checks H1 H2 H4 H5 H7` (see +[Host qualification](#host-qualification)), which replaces the earlier +`nonstop_tsc` check and takes a few seconds. It reports the CPU generation, +the CPU profile that `auto` selects, the measured TSC rate, and the host +skew in the log and the job summary, and fails the job with a stable code +when the runner is not qualified, for example `E_PROFILE_HOST_UNKNOWN` on an +unknown CPU generation or `E_PROFILE_UNSUPPORTED` on a Linux host without +`constant_tsc` and `nonstop_tsc`. Runner labels do not encode the generation; +per-PR CI captures and restores on one runner, so generations never mix. On an +MSHV runner VM whose Azure host hid the invariant TSC, never-restored guests +also hit cross-vCPU TSC warps during CPU activation, and keeping every host +CPU out of idle removed them (#211). Redeploy such a VM on a host that exposes +an invariant TSC instead of retrying its jobs. The `restore-tsc-sync` scenario repeats the restore-processor sequence with the test-only kernel option `clearcpuid=tsc_adjust`. Linux normally skips its @@ -184,6 +191,12 @@ doctor builds with `rustc` once per source version into CI). H3 and H6 need the OpenVMM binary and the guest artifacts; `--openvmm`, `--kernel`, and `--initrd` override their default build paths. +In CI, `validate-runner` runs H1, H2, H4, H5, and H7 before every job, +because jobs download OpenVMM and the guest artifacts only later. H6's +property is covered by the `smp` scenario, which runs the same warp probe at +1, 2, 4, and 8 vCPUs in every microVM job. Without H3, H4 checks the rate +stability without comparing it against the backend's rate. + ## Rust crate The required `aci-edge-sandboxes` job checks the [`aci_edge_sandboxes` crate](../aci_edge_sandboxes/README.md) on diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index a087ab8fc..95c1ba75a 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3177,7 +3177,7 @@ def test_windows_runner_requires_inbox_pcat_firmware(self): ): self.assertIn(firmware, configuration) - def test_linux_runners_require_an_invariant_tsc(self): + def test_runners_qualify_their_host_time_before_every_job(self): validate_runner = ( BuildConstants.REPO_ROOT / ".github" @@ -3189,13 +3189,39 @@ def test_linux_runners_require_an_invariant_tsc(self): BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" ).read_text(encoding="utf-8") - step = validate_runner.split(" - name: Validate host TSC\n", 1)[1] - step = step.split("\n\n - name: ", 1)[0] - self.assertIn(" if: runner.os != 'Windows'\n", step) - check = "\n".join( - line.removeprefix(" ") - for line in step.split(" run: |\n", 1)[1].splitlines() + # The doctor replaces the nonstop_tsc check (doc/design/time-abi.md). + self.assertNotIn("Validate host TSC", validate_runner) + for name, shell, command, summary in ( + ( + "Qualify host time on Linux", + "bash", + "python3 scripts/nvx.py doctor", + '--summary "${GITHUB_STEP_SUMMARY}"', + ), + ( + "Qualify host time on Windows", + "powershell", + "python scripts\\nvx.py doctor", + '--summary "$env:GITHUB_STEP_SUMMARY"', + ), + ): + with self.subTest(step=name): + step = validate_runner.split(f" - name: {name}\n", 1)[1] + step = step.split("\n\n", 1)[0] + self.assertIn(f" shell: {shell}\n", step) + self.assertIn(command, step) + self.assertIn('--backend "${{ inputs.backend }}"', step) + self.assertIn("--checks H1 H2 H4 H5 H7", step) + self.assertIn(summary, step) + args = nvx.parse_args( + ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "H5", "H7"] ) + self.assertEqual(args.checks, ["H1", "H2", "H4", "H5", "H7"]) + self.assertLess( + validate_runner.index("Validate Linux toolchain"), + validate_runner.index("Qualify host time on Linux"), + ) + function = linux_setup.split("require_invariant_tsc() {\n", 1)[1] function = "require_invariant_tsc() {\n" + function.split("\n}\n", 1)[0] function += "\n}\n" @@ -3217,21 +3243,6 @@ def test_linux_runners_require_an_invariant_tsc(self): f"model name\t: Test CPU\nflags\t\t: {flags}\n", encoding="utf-8", ) - with self.subTest(flags=flags, check="validate-runner"): - result = subprocess.run( - ["bash", "-c", check.replace("/proc/cpuinfo", str(cpuinfo))], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - self.assertEqual(result.returncode == 0, invariant, result.stderr) - self.assertIn("CPU: Test CPU", result.stdout) - self.assertEqual( - "::error::Runner host does not expose an invariant TSC" - in result.stderr, - not invariant, - ) with self.subTest(flags=flags, check="setup-linux-runner"): result = subprocess.run( [ From c6c196702dd4b9c2efd2aa8d0bb11d6fb8378186 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:10:08 -0700 Subject: [PATCH 021/188] doc: fold the perf baseline and restore attribution into the time ABI spec Record the measured costs that the time ABI removes: the WHP emulated clock's per-read cost after an SMP restore, the capture-time clocksource waits on MSHV and WHP, and the cold-start effect of no_timer_check from the Hyper-V identity. Note that MSHV creates application processors serially at about 14 ms each, which the synchronized TSC set does not add to. Name the lifecycle profile records that attribute a time ABI restore: restore.time_abi_clock, restore.guest_resume, and, for an acknowledged restore, restore.guest_repair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 88be22983..1bfe403a8 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -245,8 +245,9 @@ settled cell on the registered hosts. **WHP decision gate: met.** Suspending partition time, writing the target, verifying the frozen values, and resuming kept every pair of vCPUs within 70 ns over 60 s on bare metal and on both Azure runner generations, so WHP -never traps RDTSC. The emulated clock it replaces cost 45 to 75 µs per guest -timestamp read. +never traps RDTSC. The emulated clock it replaces cost about 45 µs per guest +timestamp read on bare metal and 66 to 70 µs on Azure (p50), on every vCPU of +an SMP-restored VM for its lifetime. ## CPU profiles @@ -1099,17 +1100,29 @@ Expected effects: | Change | Expected effect | | --- | --- | -| WHP restored SMP without RDTSC emulation (if the WHP spike meets 1 µs) | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50) | +| WHP restored SMP without RDTSC emulation | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50) and the 45 to 70 µs cost of every guest timestamp read after an SMP restore | | Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits, most visible on WHP | | Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | -| No capture-time clocksource waits | Removes up to 5 s of capture latency on MSHV and WHP; outside the restore metrics | -| No LAPIC calibration and no `tsc-early` window at cold boot | Unchanged or faster cold boot | +| No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | +| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP; KVM unchanged | +| MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Added cold-boot cost, reported as the first run's `elapsed_us`; budget 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP; the gate is authoritative | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore repair and checks before the acknowledgement | Run in one helper process; the RCU release and deferred checks run after the acknowledgement | Expected wins are tracked separately and do not relax the gate. +**Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle +profile records three exclusive time ABI restore phases: +`restore.time_abi_clock` (restore steps 11 to 16), `restore.guest_resume` +(from the VP release to the guest's first selection of the restore packet), +and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that +selection to the arrival of the `0x605` acknowledgement). The +`restore.guest_repair_gate` milestone still spans from the VP release to the +release of the acknowledgement boundary. The guest reports its own repair +checks as the restore marker's `elapsed_us`, which covers ungated restores +too. + ## Test matrix **Unit tests (OpenVMM).** Rate boundary (250 ppm accepted, one hertz beyond From 544f92eebf9eebbcdd11cc85f7e438d983ca50e7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:14:54 -0700 Subject: [PATCH 022/188] guest: start nvx-time's daemon lines on a fresh console line The daemon writes the restore marker after the RCU release and the deferred C7, and its watcher and discipline events at any time, while a shell or workload may be mid-line on the shared console. In restore-matrix runs on KVM with core's packet v4 (openvmm 7500b7472), the marker landed behind the shell prompt ("/ # NVX-TIME-ABI: v=1 phase=restore status=ok ..."), so a consumer matching markers at line start missed it in three of five restores. Start every console line and portb event the daemon writes with a newline. The kernel log record is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-time.c | 16 ++++++++++++++++ scripts/test_guest_time.py | 7 +++++++ 2 files changed, 23 insertions(+) diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 60e27284a..4f9ccd8f2 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -147,6 +147,9 @@ static bool g_report_only; static bool g_test; static uint32_t g_generation; static bool g_ports; +// Set in the daemon. Its lines can arrive while a shell or workload is in the +// middle of a console line, so each one starts with a newline. +static bool g_async_output; // --------------------------------------------------------------------------- // Small utilities @@ -477,8 +480,13 @@ static void escape_text(const char *text, char *out, size_t limit) static void console_write(const char *line) { + char buffer[EVENT_MAX + 128]; int fd = g_test ? -1 : open("/dev/console", O_WRONLY | O_NOCTTY | O_CLOEXEC); + if (g_async_output) { + snprintf(buffer, sizeof(buffer), "\n%s", line); + line = buffer; + } if (fd < 0) { (void)write_all(g_test ? STDOUT_FILENO : STDERR_FILENO, line, strlen(line)); @@ -593,6 +601,8 @@ static void emit_event(const char *code, const char *source, enum phase phase, close(fd); } if (enable_ports() == 0) { + if (g_async_output) + outb('\n', PORTB_DATA); for (const char *c = line; *c != '\0'; c++) outb((unsigned char)*c, PORTB_DATA); } @@ -2710,6 +2720,7 @@ static int start_daemon(int kmsg, uint64_t tsc_hz, uint64_t lapic_hz, daemon.tsc_hz = tsc_hz; daemon.lapic_hz = lapic_hz; daemon.last_accepted_ns = last_accepted_ns; + g_async_output = true; (void)setsid(); if (null >= 0) { (void)dup2(null, STDIN_FILENO); @@ -3499,6 +3510,11 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "kmsg") == 0) return test_lines(argc, argv, LINES_KMSG); + if (strcmp(name, "console") == 0 && argc == 2) { + g_async_output = strcmp(argv[1], "async") == 0; + console_write(argv[0]); + return 0; + } if (strcmp(name, "boot-log") == 0) return test_lines(argc, argv, LINES_BOOT_LOG); if (strcmp(name, "syslog") == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 130c2d43a..37f7efbc5 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -514,6 +514,13 @@ def syslog(text_lines: list[str]) -> list[str]: ) self.assertEqual(syslog([*lines, own_event]), ["ok"]) + def test_daemon_lines_start_on_a_fresh_console_line(self): + # The daemon prints the restore marker and runtime events while a + # shell may be in the middle of a line (its prompt, for example). + line = "NVX-TIME-ABI: v=1 phase=restore status=ok\n" + self.assertEqual(self.run_test("console", line, "sync"), line) + self.assertEqual(self.run_test("console", line, "async"), "\n" + line) + def timer_list( self, sections: list[tuple[int, str, int, str, int]], From efe8ef45695bb3fb5987e358c5d38108d0bfc1c6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:34:32 -0700 Subject: [PATCH 023/188] doc: settle the time ABI anchors, K1, and the exhaustive check Apply the orchestrator's anchor decision: both anchors re-sample a bounded number of times and fail with E_TSC_ANCHOR beyond 100 us; a frozen write needs no restore pairing. State the effect on the restore advance, and that saved per-VP TSC values are omitted from the VP restore. Add the boot-only kernel integrity check K1 (no W+X mapping) with code G_KERNEL_WX (status 193), and define the CI-only exhaustive check that the guest provides as nvx-time exhaustive: checks X1 to X6, its output lines, and its exit status. Scope the KVM_GET_CLOCK/KVM_SET_CLOCK removal to the microVM downtime path, and name E_PROFILE_UNKNOWN for a restore whose --cpu-profile names another profile than the snapshot's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 73 +++++++++++++++++++++++++++--------------- 1 file changed, 48 insertions(+), 25 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 1bfe403a8..fc2e84a5b 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -233,14 +233,14 @@ settled cell on the registered hosts. | TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; the hypervisor raises #GP for `IA32_TSC_ADJUST` | | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC fail qualification (`azure-azlinux-2`) | Set by the CPUID override: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`), so those hosts qualify on the warp probe and rate stability (the guest TSC matched the declared rate against host QPC within 0.001 ppm over 118 s) | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 10 µs | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads (0.1 to 0.6 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | TBD(whp) | +| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, re-sampled a bounded number of times | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | TBD(whp) | | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and thaw at the first VP run (56 to 312 µs for 1 to 8 VPs) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | | Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | -| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | +| Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | **WHP decision gate: met.** Suspending partition time, writing the target, verifying the frozen values, and resuming kept every pair of vCPUs within @@ -465,9 +465,10 @@ the guest and tests can tell the two paths apart. The restore anchor is the instant of the synchronized TSC set. On KVM and WHP the guest TSC runs from that instant (WHP resumes partition time right after the read-back); on MSHV it is frozen until the first VP runs. Either -way, the guest monotonic advance across a restore lies in `[D, D + R]`, -where `R` is the time from the restore anchor to the first restored -instruction. Wall-clock repair absorbs `R` for `CLOCK_REALTIME`. +way, the guest monotonic advance across a restore lies in `[D, D + R]`, up +to the two anchors' pairing errors (each at most 100 µs), where `R` is the +time from the restore anchor to the first restored instruction. Wall-clock +repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. ## Restore algorithm @@ -529,9 +530,11 @@ In the worker, with every VP stopped: (`E_TSC_TARGET_OVERFLOW` if it exceeds 64 bits). The backend writes `TSC_target` to every instantiated VP at that instant. Per-VP TSC values - from saved VP state are superseded and never applied afterwards. The - restore anchor's pairing is logged above 10 µs but not bounded: a pairing - error shifts every VP's TSC alike, by at most that amount. + from saved VP state are omitted from the VP restore and never applied. A + restore anchor that pairs a TSC read with host time (KVM's common offset) + is re-sampled a bounded number of times and fails with `E_TSC_ANCHOR` if + no pair is within 100 µs; a pairing error shifts every VP's TSC alike, by + at most that amount. A frozen write needs no such pairing. 13. Read back: every instantiated VP holds the synchronized value as defined by its backend (`E_TSC_SYNC_READBACK`). 14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every @@ -564,8 +567,8 @@ state units are quiesced, capture: PIT channel 0 is not counting periodically (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`, `E_PIT_ACTIVE`); 2. takes the capture anchor: VP 0's TSC paired with a host time sample taken - within 10 µs of it (`E_TSC_ANCHOR`), plus the host identities - (`E_HOST_IDENTITY`); and + within 100 µs of it, keeping the tightest of a bounded number of samples + (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); and 3. records the declared rates, the CPU profile, the effective CPUID, and the process's generation counter. @@ -786,17 +789,33 @@ partition-wide on every backend, so `C3` reads them on CPU 0 only. | `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0; at boot, the state file is published and the daemon started | boot, capture, restore | | `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | | `C12` | A time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | +| `K1` | Kernel integrity, not a clock property: the kernel's boot-time W+X audit logged no `Found insecure W+X mapping` record | boot | Capture and restore checks never wait. The boot `C7` check and the deferred restore `C7` check may poll for at most 200 ms, because a CPU switches to -one-shot mode at its first tick. The CI conformance suite adds an exhaustive -check on every CPU: every leaf `0x40000006..=0x400000ff`; no base -`0x40000100..=0x4000ff00` (step `0x100`) carrying `KVMKVMKVM` or another -hypervisor signature (the NVX kernel has no KVM guest support, so the boot -check leaves this static property of the backend to CI); every `C3` MSR; -writes of 0 and 1 to `0x40000118` and of 2 (which must fail); writes to every -read-only identity MSR (which must fail); and reads of `IA32_TSC_ADJUST` and -`IA32_TSC_DEADLINE` (which must fail). +one-shot mode at its first tick. + +**Exhaustive check (CI only).** The CI conformance suite also runs +`nvx-time exhaustive`, provided by the guest. It is a test tool: it reports +and exits instead of powering off, and production boots never run it. Every +check runs on every online CPU: + +| ID | Check | +| --- | --- | +| `X1` | Every leaf `0x40000006..=0x400000ff` returns all zeros or the Intel out-of-range result (the highest basic leaf's result for the same subleaf), as the [identity rules](#hypervisor-identity) require | +| `X2` | No base `0x40000100..=0x4000ff00` (step `0x100`) carries `KVMKVMKVM` or another hypervisor signature; the NVX kernel has no KVM guest support, so the boot check leaves this static backend property to CI | +| `X3` | Every `C3` MSR | +| `X4` | `0x40000118` accepts writes of 0 and 1, each read back, and rejects 2; the check leaves it at 1 | +| `X5` | Writes to every read-only identity MSR fail | +| `X6` | Reads of `IA32_TSC_ADJUST` and `IA32_TSC_DEADLINE` fail | + +It prints one line per check and CPU, then a summary, and exits with status +0 if every check passed and 1 otherwise: + +```text +NVX-TIME-ABI-EXHAUSTIVE: v=1 check= cpu= status= detail="" +NVX-TIME-ABI-EXHAUSTIVE: v=1 status= cpus= failures= +``` On success the check prints one line to the console: @@ -811,7 +830,8 @@ cannot lose it; its console cost (one port exit per byte) is part of the cold-boot cost the performance gate measures. On failure it prints `status=fail check= detail=""` in the same -format, emits a violation event with code `G_CONFORMANCE_`, and powers +format, emits a violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` +for `K1`), and powers off with status 193. The boot step of `C12` steps the clock before any workload starts and is not counted as a discontinuity. The only other line with this prefix is the non-fatal `phase=runtime status=uncertain` line of @@ -1010,7 +1030,7 @@ code. | `E_SNAPSHOT_VERSION` | Manifest version is not 6; the snapshot must be recaptured | Restore | | `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | | `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | -| `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM | Cold boot, restore | +| `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM, or a restore's explicit `--cpu-profile` names another profile than the snapshot's | Cold boot, restore | | `E_PROFILE_DIGEST` | Recorded, embedded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | | `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile, or to profiles of more than one generation | Cold boot | | `E_PROFILE_TIME_BITS` | The profile is invalid or violates the CPU time bits | Cold boot, restore | @@ -1028,7 +1048,7 @@ code. | `E_HOST_IDENTITY` | Host identity, boot identity, or clocks unavailable | Capture, restore | | `E_DOWNTIME_NEGATIVE` | Downtime below zero | Restore | | `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days | Restore | -| `E_TSC_ANCHOR` | Capture anchor unavailable or not paired within 10 µs | Capture | +| `E_TSC_ANCHOR` | An anchor is unavailable, or no sample pairs within 100 µs after bounded re-sampling | Capture, restore | | `E_TSC_TARGET_OVERFLOW` | `TSC_target` exceeds 64 bits | Restore | | `E_TSC_SYNC_READBACK` | A VP does not hold the synchronized value | Restore | | `E_VP_LATE_CREATION` | A VP was instantiated after the synchronized set | Restore | @@ -1045,7 +1065,7 @@ signals, and 255): | Status | Class | Guest codes | | ---: | --- | --- | -| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C12` | +| 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C12`, `G_KERNEL_WX` | | 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_KMSG_OVERRUN` | | 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | @@ -1201,9 +1221,12 @@ Removed from OpenVMM: - Leaf `0x15` synthesis (`tsc_frequency_cpuid_leaves`) on every backend. - Per-backend downtime paths (`advance_snapshot_time`), per-VP TSC advancement (`advance_tsc`), TSC-deadline advancement, periodic LAPIC - advancement, and wall-clock-only downtime (`calculate_snapshot_downtime`). -- KVM: `KVM_GET_CLOCK`/`KVM_SET_CLOCK`, kvmclock MSR state, KVM CPUID leaves, - `KVM_SET_TSC_KHZ`, and restore-time `IA32_TSC` writes. + advancement, wall-clock-only downtime (`calculate_snapshot_downtime`), and + the per-VP TSC writes of saved VP state on restore. +- KVM: the microVM downtime use of `KVM_GET_CLOCK`/`KVM_SET_CLOCK` (the + Hyper-V reference time source keeps them; it requires `hv1`, which the time + ABI never enables), kvmclock MSR state, KVM CPUID leaves, `KVM_SET_TSC_KHZ`, + and restore-time `IA32_TSC` writes. - MSHV: BSP-copy TSC alignment and exact rate equality. - WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` with its RDTSC, RDTSCP, and `IA32_TSC` exits. From 47d16412026b137d577020dbc949e23e7d70c503 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:21:50 -0700 Subject: [PATCH 024/188] scripts: qualify hosts on measured time properties only Host qualification now gates only on measured properties, alike on every backend: the guest warp probe at 1 us, the TSC rate stability, and the CPU profile (fingerprint and verification). The host OS's own invariant-TSC flag is evidence: Azure WHP hosts lack it while their guests measure tens of nanoseconds of skew and no rate error. - H2 records constant_tsc and nonstop_tsc (or the Windows CPUID bit) and no longer fails a host without them. - H4 records the Linux host clocksource and no longer requires tsc on KVM or the TSC page on MSHV. - H6 runs an idle-inducing warp schedule: two rounds of nvx-time-probe warp over every CPU pair with all vCPUs halted for 1 s between them. A host without an invariant TSC corrects guest TSC timing when an idle host CPU wakes, which busy probe rounds never trigger (#265). The shared warp_probe_script() fragment and the round-aware check_warp_probe() serve the microVM scenarios too. - validate-runner runs the cheap host-level checks H1, H2, and H4; the guest warp probe needs a time ABI boot and belongs in the microVM scenarios. H5 and H7 stay available for interactive qualification. - setup-linux-runner.sh reports a missing invariant TSC as a warning instead of refusing to provision the runner. The job summary lists the invariant-TSC flags and the clocksource as evidence. Refs #211, #265 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 10 +- doc/ci.md | 47 +++-- scripts/nvx_tools/doctor.py | 63 +++--- scripts/nvx_tools/time_abi.py | 64 +++++- scripts/setup/setup-linux-runner.sh | 16 +- scripts/test_nvx_tools.py | 22 +- scripts/test_time_abi.py | 230 +++++++++++++++------ 7 files changed, 309 insertions(+), 143 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index b02f2ae39..b307e0856 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -118,15 +118,17 @@ runs: } # Time ABI host qualification (doc/design/time-abi.md, "Host - # qualification"); it replaces the nonstop_tsc check. H3 and H6 need the - # OpenVMM binary and guest artifacts, which jobs download later. + # qualification"): the backend, the CPU fingerprint and generation, and + # the TSC rate stability. It replaces the nonstop_tsc check, which is now + # recorded as evidence only. The guest warp probe runs in the microVM + # scenarios, which boot the time ABI. - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash run: >- python3 scripts/nvx.py doctor --backend "${{ inputs.backend }}" - --checks H1 H2 H4 H5 H7 + --checks H1 H2 H4 --summary "${GITHUB_STEP_SUMMARY}" - name: Qualify host time on Windows @@ -135,7 +137,7 @@ runs: run: >- python scripts\nvx.py doctor --backend "${{ inputs.backend }}" - --checks H1 H2 H4 H5 H7 + --checks H1 H2 H4 --summary "$env:GITHUB_STEP_SUMMARY" - name: Validate KVM diff --git a/doc/ci.md b/doc/ci.md index 44b1da1cb..5a07ce801 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -56,19 +56,21 @@ exposes an invariant TSC. The control log is kept as failure; the restore still fails. Every job that uses the `validate-runner` action first qualifies its runner -for the time ABI with `nvx.py doctor --checks H1 H2 H4 H5 H7` (see -[Host qualification](#host-qualification)), which replaces the earlier +for the time ABI with `nvx.py doctor --checks H1 H2 H4` (see +[Host qualification](#host-qualification)): the backend, the CPU fingerprint +and generation, and the TSC rate stability. This replaces the earlier `nonstop_tsc` check and takes a few seconds. It reports the CPU generation, -the CPU profile that `auto` selects, the measured TSC rate, and the host -skew in the log and the job summary, and fails the job with a stable code -when the runner is not qualified, for example `E_PROFILE_HOST_UNKNOWN` on an -unknown CPU generation or `E_PROFILE_UNSUPPORTED` on a Linux host without -`constant_tsc` and `nonstop_tsc`. Runner labels do not encode the generation; -per-PR CI captures and restores on one runner, so generations never mix. On an -MSHV runner VM whose Azure host hid the invariant TSC, never-restored guests -also hit cross-vCPU TSC warps during CPU activation, and keeping every host -CPU out of idle removed them (#211). Redeploy such a VM on a host that exposes -an invariant TSC instead of retrying its jobs. +the CPU profile that `auto` selects, and the measured TSC rate in the log and +the job summary, and fails the job with a stable code when the runner is not +qualified, for example `E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation. +Qualification gates only on measured properties, alike on every backend: the +host OS's invariant-TSC flags and clocksource are recorded as evidence, and +the guest warp probe in the microVM scenarios measures the skew that a host +without an invariant TSC causes. On such an MSHV runner VM, never-restored +guests hit cross-vCPU TSC warps when an idle host CPU woke (#211, #265), which +is why the probe schedule includes idle gaps. Runner labels do not encode the +generation; per-PR CI captures and restores on one runner, so generations +never mix. The `restore-tsc-sync` scenario repeats the restore-processor sequence with the test-only kernel option `clearcpuid=tsc_adjust`. Linux normally skips its @@ -177,11 +179,11 @@ brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. | Check | Implementation | | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | -| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A Linux host must report `constant_tsc` and `nonstop_tsc` (`E_PROFILE_UNSUPPORTED`), so `azure-azlinux-2` stays unqualified. A Windows host's invariant-TSC CPUID bit is only reported: Azure WHP hosts cannot see one, and the spec qualifies them on the measured warp and rate stability | +| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check: Azure WHP hosts lack them while their guests measure tens of nanoseconds of skew | | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names; OpenVMM's interim `interim.host..v1` profile is accepted until it selects catalog profiles. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | -| H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A KVM host's clocksource must be `tsc`; an MSHV root's may also be `hyperv_clocksource_tsc_page` | +| H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | -| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker, and `nvx-time-probe warp` stays within 1,000 ns | +| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker, and the idle-inducing warp schedule stays within 1,000 ns: two rounds of `nvx-time-probe warp` over every CPU pair with all vCPUs halted for 1 s between them, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265) | | H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows | H2, H4, and H5 use a dependency-free host probe, @@ -191,11 +193,16 @@ doctor builds with `rustc` once per source version into CI). H3 and H6 need the OpenVMM binary and the guest artifacts; `--openvmm`, `--kernel`, and `--initrd` override their default build paths. -In CI, `validate-runner` runs H1, H2, H4, H5, and H7 before every job, -because jobs download OpenVMM and the guest artifacts only later. H6's -property is covered by the `smp` scenario, which runs the same warp probe at -1, 2, 4, and 8 vCPUs in every microVM job. Without H3, H4 checks the rate -stability without comparing it against the backend's rate. +Qualification gates on measured properties, alike on every backend: the guest +warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and +the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource +are evidence only. In CI, `validate-runner` runs the cheap host-level checks +H1, H2, and H4 before every job; jobs download OpenVMM and the guest artifacts +only later. The guest warp probe needs a time ABI boot, so it belongs in the +microVM boot and restore scenarios, which assert its verdict once OpenVMM +boots the time ABI by default. Without H3, H4 checks the rate stability +without comparing it against the backend's rate. H5 and H7 remain available +for interactive qualification. ## Rust crate diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 96c89772c..426338219 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -31,12 +31,15 @@ MAX_TSC_HZ, MIN_TSC_HZ, WARP_BOUND_NS, + WARP_PROBE_COMPLETION_MARKER, + WARP_PROBE_IDLE_SECONDS, TimeAbiFailure, TimeAbiMonitor, check_warp_probe, cpu_generation, parse_fields, - warp_probe_command, + warp_probe_script, + warp_rounds, ) CHECK_IDS = ("H1", "H2", "H3", "H4", "H5", "H6", "H7") @@ -59,19 +62,12 @@ SKEW_PAIR_DURATION_MS = 5 GUEST_VCPU_COUNTS = (8, 4, 2, 1) GUEST_MEMORY_MIB = 128 -GUEST_WARP_MARKER = b"NVX-DOCTOR-WARP-DONE" -# Azure WHP hosts cannot see an invariant TSC; the spec qualifies them on the -# measured warp (H6) and rate stability (H4), so the bit is only reported. -# A Linux host whose root sees no invariant TSC (azure-azlinux-2) stays -# unqualified, which the spec's restore matrix expects. -WHP_REQUIRES_INVARIANT_TSC = False +# Qualification gates only on measured properties, alike on every backend: +# the guest warp probe, the TSC rate stability, and the CPU profile. The host +# OS's invariant-TSC flags and clocksource are recorded as evidence only; +# Azure WHP hosts lack the flag while their guests measure tens of +# nanoseconds of skew. LINUX_INVARIANT_TSC_FLAGS = ("constant_tsc", "nonstop_tsc") -# KVM rewrites per-vCPU TSC offsets on a host whose TSC Linux distrusts. An -# MSHV root's Linux reads time through the hypervisor's TSC page instead. -HOST_CLOCKSOURCES: Mapping[str, tuple[str, ...]] = { - "kvm": ("tsc",), - "mshv": ("tsc", "hyperv_clocksource_tsc_page"), -} CLOCKSOURCE_PATH = Path( "/sys/devices/system/clocksource/clocksource0/current_clocksource" ) @@ -358,14 +354,6 @@ def check_cpu(context: DoctorContext) -> CheckResult: "(skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids " f"6/207); {detail}", ) - required = context.backend != "whp" or WHP_REQUIRES_INVARIANT_TSC - if required and info["invariant_tsc"] != "yes": - return CheckResult( - "H2", - False, - "[E_PROFILE_UNSUPPORTED] the host does not expose an invariant TSC, " - f"which every CPU profile requires; {detail}", - ) return CheckResult("H2", True, detail) @@ -489,7 +477,7 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: def check_rate(context: DoctorContext) -> CheckResult: clocksource = "" - if context.backend in HOST_CLOCKSOURCES: + if not host_is_windows(): clocksource = host_clocksource() context.facts["host_clocksource"] = clocksource records = run_probe( @@ -527,12 +515,7 @@ def check_rate(context: DoctorContext) -> CheckResult: f"rate (limit {RATE_TOLERANCE_PPM:g} ppm)" ) if clocksource: - detail += f"; host clocksource {clocksource}" - if clocksource not in HOST_CLOCKSOURCES[context.backend]: - problems.append( - f"the host clocksource is {clocksource}, not " - + " or ".join(HOST_CLOCKSOURCES[context.backend]) - ) + detail += f"; host clocksource {clocksource} (evidence)" if problems: return CheckResult("H4", False, "; ".join(problems) + f"; {detail}") return CheckResult("H4", True, detail) @@ -585,31 +568,37 @@ def check_guest_warp(context: DoctorContext) -> CheckResult: processors=vcpus, ) command.extend(context.openvmm_args) - script = f"{warp_probe_command()}\necho {GUEST_WARP_MARKER.decode()}\nnvx-exit 0\n" try: result = run_guest_script( command, - script, - GUEST_WARP_MARKER, + warp_probe_script() + "nvx-exit 0\n", + WARP_PROBE_COMPLETION_MARKER, timeout=context.timeout, ) monitor = TimeAbiMonitor(command) monitor.feed(result["text"].encode()) monitor.finish() boot = monitor.require_boot("the guest boot marker", online_cpus=vcpus) - warp = check_warp_probe(result["text"], cpus=vcpus, context="H6")[-1] + rounds = check_warp_probe( + result["text"], cpus=vcpus, context="H6", rounds=warp_rounds(vcpus) + ) except (RuntimeError, TimeAbiFailure) as error: first = str(error).splitlines()[0] return CheckResult("H6", False, f"vcpus={vcpus}: {first}") - context.facts["guest_warp_ns"] = warp["max_abs_offset_ns"] + + def worst(name: str) -> int: + return max(int(warp[name]) for warp in rounds) + + context.facts["guest_warp_ns"] = str(worst("max_abs_offset_ns")) context.facts.setdefault("tsc_hz", boot["tsc_hz"]) context.facts.setdefault("lapic_hz", boot["lapic_hz"]) return CheckResult( "H6", True, - f"vcpus={vcpus} max_backward_ns={warp['max_backward_ns']} " - f"max_abs_offset_ns={warp['max_abs_offset_ns']} " - f"max_uncertainty_ns={warp['max_uncertainty_ns']} boot_elapsed_us=" + f"vcpus={vcpus} rounds={len(rounds)} idle_s={WARP_PROBE_IDLE_SECONDS} " + f"max_backward_ns={worst('max_backward_ns')} " + f"max_abs_offset_ns={worst('max_abs_offset_ns')} " + f"max_uncertainty_ns={worst('max_uncertainty_ns')} boot_elapsed_us=" f"{boot.get('elapsed_us', '?')}", ) @@ -731,6 +720,8 @@ def summary_markdown( ("measured_tsc_hz", "Measured TSC rate (Hz)"), ("host_skew_ns", "Host skew (ns)"), ("guest_warp_ns", "Guest warp offset (ns)"), + ("invariant_tsc", "Host invariant TSC (evidence)"), + ("host_clocksource", "Host clocksource (evidence)"), ) facts_line = " · ".join( f"{label}: `{facts[name]}`" for name, label in reported if name in facts diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 37b1c36e0..589b410f7 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -32,6 +32,12 @@ WARP_PROBE_PATH = "/sbin/nvx-time-probe" WARP_SUMMARY_PREFIX = "NVX-TIME-PROBE warp " WARP_DETAIL_PREFIX = "NVX-TIME-PROBE warp-detail " +WARP_PROBE_COMPLETION_MARKER = b"NVX-WARP-PROBE-OK" +WARP_PROBE_FAILURE_MARKER = b"NVX-WARP-PROBE-FAIL" +WARP_PROBE_FAILURE_STATUS = 97 +# The idle-inducing warp schedule: probe rounds separated by halted vCPUs. +WARP_PROBE_ROUNDS = 2 +WARP_PROBE_IDLE_SECONDS = 1 PROFILE_VENDORS: Mapping[str, str] = {"GenuineIntel": "intel"} @@ -343,16 +349,61 @@ def warp_probe_command(*, cpus: str | None = None) -> str: return command if cpus is None else f"{command} --cpus {cpus}" +def warp_rounds(cpus: int) -> int: + """Return how many probe rounds the warp schedule runs on ``cpus`` CPUs.""" + return 1 if cpus < 2 else WARP_PROBE_ROUNDS + + +def warp_probe_script() -> str: + """Return a guest shell fragment that runs the idle-inducing warp schedule. + + The probe runs over every online CPU in WARP_PROBE_ROUNDS rounds with all + vCPUs halted for WARP_PROBE_IDLE_SECONDS between rounds: a host without an + invariant TSC corrects guest TSC timing when an idle host CPU wakes, which + busy probe rounds alone never trigger (#265). The CPU list is read when + the fragment runs, so one post-restore script serves every restore target. + A failed round prints WARP_PROBE_FAILURE_MARKER and powers the guest off + with WARP_PROBE_FAILURE_STATUS; otherwise the fragment prints + WARP_PROBE_COMPLETION_MARKER. + """ + probe = warp_probe_command(cpus='"$warp_cpus"') + return ( + 'warp_cpus="$(cat /sys/devices/system/cpu/online)"\n' + f"warp_rounds={WARP_PROBE_ROUNDS}\n" + 'case "$warp_cpus" in\n' + " *[-,]*) ;;\n" + " *) warp_rounds=1 ;;\n" + "esac\n" + "warp_round=1\n" + "while :; do\n" + " warp_status=0\n" + f" {probe} || warp_status=$?\n" + ' if [ "$warp_status" -ne 0 ]; then\n' + f' echo "{WARP_PROBE_FAILURE_MARKER.decode()} ' + 'status=$warp_status round=$warp_round"\n' + f" nvx-exit {WARP_PROBE_FAILURE_STATUS}\n" + f" exit {WARP_PROBE_FAILURE_STATUS}\n" + " fi\n" + ' [ "$warp_round" -lt "$warp_rounds" ] || break\n' + f" sleep {WARP_PROBE_IDLE_SECONDS}\n" + " warp_round=$((warp_round + 1))\n" + "done\n" + f"echo {WARP_PROBE_COMPLETION_MARKER.decode()}\n" + ) + + def check_warp_probe( text: str, *, cpus: int, context: str, + rounds: int | None = None, ) -> list[dict[str, str]]: """Validate every warp probe summary in console output. The spec bounds both the backward step and the ping-pong offset by 1 us and forbids stalled pairs; an inconclusive measurement proves nothing. + ``rounds``, when given, is the exact number of probe rounds expected. """ summaries: list[dict[str, str]] = [] details: list[dict[str, str]] = [] @@ -374,9 +425,16 @@ def check_warp_probe( f"{context}: the guest warp probe printed {len(summaries)} summaries " f"and {len(details)} detail lines" ) + if rounds is not None and len(summaries) != rounds: + raise TimeAbiFailure( + f"{context}: the guest warp probe ran {len(summaries)} rounds instead " + f"of {rounds}" + ) expected_pairs = cpus * (cpus - 1) // 2 results: list[dict[str, str]] = [] - for summary, detail in zip(summaries, details, strict=True): + for index, (summary, detail) in enumerate( + zip(summaries, details, strict=True), start=1 + ): merged = {**summary, **detail} problems: list[str] = [] try: @@ -404,8 +462,10 @@ def check_warp_probe( if merged.get("verdict") != "PASS": problems.append(f"verdict={merged.get('verdict')}") if problems: + where = f" in round {index}" if len(summaries) > 1 else "" raise TimeAbiFailure( - f"{context}: cross-vCPU TSC skew check failed: " + "; ".join(problems) + f"{context}: cross-vCPU TSC skew check failed{where}: " + + "; ".join(problems) ) results.append(merged) return results diff --git a/scripts/setup/setup-linux-runner.sh b/scripts/setup/setup-linux-runner.sh index 6b36c51bc..d195318d4 100644 --- a/scripts/setup/setup-linux-runner.sh +++ b/scripts/setup/setup-linux-runner.sh @@ -60,11 +60,15 @@ version_at_least() { [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -n 1)" = "$2" ] } -require_invariant_tsc() { - # Guests on a host without an invariant TSC intermittently see cross-vCPU - # TSC warps, which make Linux mark the guest TSC unstable (#211). - grep -Eq '^flags[[:space:]]*:.*[[:space:]]nonstop_tsc([[:space:]]|$)' "$1" || - die "host does not expose an invariant TSC (nonstop_tsc); redeploy the VM on a host that does" +report_invariant_tsc() { + # The host OS's invariant-TSC flag is evidence only. CI qualifies a runner + # on measured properties instead: the guest warp probe and the TSC rate + # stability (doc/ci.md, "Host qualification"; #211, #265). + if grep -Eq '^flags[[:space:]]*:.*[[:space:]]nonstop_tsc([[:space:]]|$)' "$1"; then + echo "host exposes an invariant TSC (nonstop_tsc)" + else + echo "warning: host does not expose an invariant TSC (nonstop_tsc); CI qualifies it on the measured warp and rate stability" >&2 + fi } validate_runner_name() { @@ -818,7 +822,7 @@ case "$backend" in kvm | mshv) ;; *) die "--backend must be kvm or mshv" ;; esac -require_invariant_tsc /proc/cpuinfo +report_invariant_tsc /proc/cpuinfo [ "$(id -u)" -ne 0 ] || die "run this script as the SSH administrator, not root" require_command sudo sudo -n true || die "passwordless sudo is required" diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 95c1ba75a..e1a0377bb 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3189,7 +3189,8 @@ def test_runners_qualify_their_host_time_before_every_job(self): BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" ).read_text(encoding="utf-8") - # The doctor replaces the nonstop_tsc check (doc/design/time-abi.md). + # The doctor replaces the nonstop_tsc check: the backend, the CPU + # fingerprint and generation, and the TSC rate stability. self.assertNotIn("Validate host TSC", validate_runner) for name, shell, command, summary in ( ( @@ -3211,22 +3212,24 @@ def test_runners_qualify_their_host_time_before_every_job(self): self.assertIn(f" shell: {shell}\n", step) self.assertIn(command, step) self.assertIn('--backend "${{ inputs.backend }}"', step) - self.assertIn("--checks H1 H2 H4 H5 H7", step) + self.assertIn("--checks H1 H2 H4\n", step) self.assertIn(summary, step) args = nvx.parse_args( - ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "H5", "H7"] + ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4"] ) - self.assertEqual(args.checks, ["H1", "H2", "H4", "H5", "H7"]) + self.assertEqual(args.checks, ["H1", "H2", "H4"]) self.assertLess( validate_runner.index("Validate Linux toolchain"), validate_runner.index("Qualify host time on Linux"), ) - function = linux_setup.split("require_invariant_tsc() {\n", 1)[1] - function = "require_invariant_tsc() {\n" + function.split("\n}\n", 1)[0] + # The runner setup records the host's invariant-TSC flag as evidence. + self.assertNotIn("require_invariant_tsc", linux_setup) + function = linux_setup.split("report_invariant_tsc() {\n", 1)[1] + function = "report_invariant_tsc() {\n" + function.split("\n}\n", 1)[0] function += "\n}\n" self.assertLess( - linux_setup.index("require_invariant_tsc /proc/cpuinfo\n"), + linux_setup.index("report_invariant_tsc /proc/cpuinfo\n"), linux_setup.index('sudo -n true || die "passwordless sudo is required"'), ) if os.name != "posix": @@ -3250,14 +3253,15 @@ def test_runners_qualify_their_host_time_before_every_job(self): "-c", 'die() { echo "$*" >&2; exit 1; }\n' f"{function}" - f"require_invariant_tsc '{cpuinfo}'\n", + f"report_invariant_tsc '{cpuinfo}'\n", ], capture_output=True, text=True, timeout=10, check=False, ) - self.assertEqual(result.returncode == 0, invariant, result.stderr) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual("warning:" in result.stderr, not invariant) def test_runner_setups_install_backend_native_openvmm_targets(self): linux_setup = ( diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 7719ec65a..e2e3d259d 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -5,6 +5,7 @@ import contextlib import io import queue +import shutil import socket import subprocess import sys @@ -278,6 +279,88 @@ def test_rejects_skew_stalls_and_inconclusive_measurements(self): with self.assertRaisesRegex(TimeAbiFailure, message): time_abi.check_warp_probe(text, cpus=4, context="restore 4") + def test_counts_rounds_and_names_the_failing_round(self): + self.assertEqual( + [time_abi.warp_rounds(cpus) for cpus in (1, 2, 8)], + [1, time_abi.WARP_PROBE_ROUNDS, time_abi.WARP_PROBE_ROUNDS], + ) + two = warp_output() + warp_output(offset=30) + results = time_abi.check_warp_probe(two, cpus=4, context="boot", rounds=2) + self.assertEqual( + [result["max_abs_offset_ns"] for result in results], ["15", "30"] + ) + with self.assertRaisesRegex(TimeAbiFailure, "ran 1 rounds instead of 2"): + time_abi.check_warp_probe(warp_output(), cpus=4, context="boot", rounds=2) + with self.assertRaisesRegex( + TimeAbiFailure, "failed in round 2: max_backward_ns=2000" + ): + time_abi.check_warp_probe( + warp_output() + warp_output(backward=2000, verdict="FAIL"), + cpus=4, + context="boot", + ) + + def test_warp_fragment_idles_between_rounds_and_powers_off_on_failure(self): + shell = shutil.which("sh") + if shell is None and (git := shutil.which("git")) is not None: + candidate = Path(git).parent.parent / "bin" / "sh.exe" + shell = str(candidate) if candidate.is_file() else None + if shell is None: + self.skipTest("POSIX shell is unavailable") + fragment = ( + time_abi.warp_probe_script() + .replace(time_abi.WARP_PROBE_PATH, "probe") + .replace("nvx-exit", "nvx_exit") + ) + for online, failing_round, calls, sleeps in ( + ("0-3", 0, 2, 1), + ("0-3", 2, 2, 1), + ("0-3", 1, 1, 0), + ("0", 0, 1, 0), + ): + with self.subTest(online=online, failing_round=failing_round): + result = subprocess.run( + [shell], + input=( + f"cat() {{ printf '%s\\n' '{online}'; }}\n" + "calls=0\n" + "probe() {\n" + " calls=$((calls + 1))\n" + ' printf "probe %s\\n" "$*"\n' + f' [ "$calls" -ne {failing_round} ]\n' + "}\n" + 'sleep() { printf "SLEEP %s\\n" "$1"; }\n' + 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' + + fragment + + "echo after\n" + ), + text=True, + capture_output=True, + timeout=10, + check=False, + ) + lines = result.stdout.splitlines() + self.assertEqual( + lines.count(f"probe warp --bound-ns 1000 --cpus {online}"), + calls, + result.stdout + result.stderr, + ) + self.assertEqual( + lines.count(f"SLEEP {time_abi.WARP_PROBE_IDLE_SECONDS}"), sleeps + ) + if failing_round: + self.assertEqual(result.returncode, 97) + self.assertIn( + f"NVX-WARP-PROBE-FAIL status=1 round={failing_round}", lines + ) + self.assertIn("NVX-EXIT 97", lines) + self.assertNotIn("NVX-WARP-PROBE-OK", lines) + self.assertNotIn("after", lines) + else: + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("NVX-WARP-PROBE-OK", lines) + self.assertIn("after", lines) + VIOLATION = ( b"NVX-TIME-ABI-VIOLATION: v=1 code=G_RCU_STALL source=watcher phase=runtime " @@ -544,7 +627,9 @@ def exists(path: Path) -> bool: self.assertFalse(whp.passed) self.assertIn("requires Windows", whp.detail) - def test_cpu_check_names_the_generation_and_applies_the_invariant_tsc_rule(self): + def test_cpu_check_names_the_generation_and_records_invariant_tsc_as_evidence( + self, + ): context = doctor_context(self.root) with patch.object(doctor, "host_cpu", return_value=dict(CPU)): result = doctor.check_cpu(context) @@ -563,12 +648,17 @@ def test_cpu_check_names_the_generation_and_applies_the_invariant_tsc_rule(self) result = doctor.check_cpu(doctor_context(self.root, backend)) self.assertTrue(result.passed, result.detail) self.assertIn("profile=intel.emeraldrapids.v1", result.detail) + # The host OS's invariant-TSC flags are evidence, never a gate. variant = dict(CPU, invariant_tsc="no (missing nonstop_tsc)") with patch.object(doctor, "host_cpu", return_value=variant): - result = doctor.check_cpu(doctor_context(self.root, "mshv")) - self.assertTrue(result.detail.startswith("[E_PROFILE_UNSUPPORTED] ")) - # A WHP host qualifies on measured properties instead. - self.assertTrue(doctor.check_cpu(doctor_context(self.root, "whp")).passed) + for backend in ("kvm", "mshv", "whp"): + context = doctor_context(self.root, backend) + result = doctor.check_cpu(context) + self.assertTrue(result.passed, result.detail) + self.assertIn("invariant_tsc=no (missing nonstop_tsc)", result.detail) + self.assertEqual( + context.facts["invariant_tsc"], "no (missing nonstop_tsc)" + ) def test_reads_the_first_processor_from_cpuinfo(self): path = self.root / "cpuinfo" @@ -670,67 +760,67 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: self.assertFalse(missing.passed) self.assertIn("was not found", missing.detail) - def test_rate_check_requires_stability_the_backend_rate_and_a_tsc_clocksource(self): + def test_rate_check_gates_on_stability_and_the_backend_rate_only(self): def records(first: float, second: float): return [ ("rate", {"tsc_hz": f"{first:.3f}"}), ("rate", {"tsc_hz": f"{second:.3f}"}), ] + def check( + context: doctor.DoctorContext, + probe_records: list[tuple[str, dict[str, str]]], + clocksource: str = "tsc", + windows: bool = False, + ) -> doctor.CheckResult: + with ( + patch.object(doctor, "run_probe", return_value=probe_records), + patch.object(doctor, "host_clocksource", return_value=clocksource), + patch.object(doctor, "host_is_windows", return_value=windows), + ): + return doctor.check_rate(context) + context = doctor_context(self.root) context.facts["tsc_hz"] = "2793437000" - with ( - patch.object( - doctor, "run_probe", return_value=records(2793437500, 2793437800) - ), - patch.object(doctor, "host_clocksource", return_value="tsc"), - ): - result = doctor.check_rate(context) + result = check(context, records(2793437500, 2793437800)) self.assertTrue(result.passed, result.detail) - cases = ( - (records(2793437000, 2793440000), "tsc", "unstable"), - (records(2794000000, 2794000001), "tsc", "ppm from the measured"), - (records(2793437000, 2793437000), "hpet", "clocksource is hpet"), - ) - for probe_records, clocksource, message in cases: + self.assertIn("host clocksource tsc (evidence)", result.detail) + for probe_records, message in ( + (records(2793437000, 2793440000), "unstable"), + (records(2794000000, 2794000001), "ppm from the measured"), + ): with self.subTest(message=message): context = doctor_context(self.root) context.facts["tsc_hz"] = "2793437000" - with ( - patch.object(doctor, "run_probe", return_value=probe_records), - patch.object(doctor, "host_clocksource", return_value=clocksource), - ): - result = doctor.check_rate(context) + result = check(context, probe_records) self.assertFalse(result.passed) self.assertIn(message, result.detail) - mshv = doctor_context(self.root, "mshv") - mshv.facts["tsc_hz"] = "2300000000" - with ( - patch.object( - doctor, "run_probe", return_value=records(2300000100, 2300000200) - ), - patch.object( - doctor, "host_clocksource", return_value="hyperv_clocksource_tsc_page" - ), + # The clocksource is evidence on every backend. + for backend, clocksource in ( + ("kvm", "hpet"), + ("mshv", "hyperv_clocksource_tsc_page"), ): - self.assertTrue(doctor.check_rate(mshv).passed) - with ( - patch.object( - doctor, "run_probe", return_value=records(2300000100, 2300000200) - ), - patch.object(doctor, "host_clocksource", return_value="tsc"), - ): - standalone = doctor.check_rate(doctor_context(self.root)) + with self.subTest(backend=backend, clocksource=clocksource): + context = doctor_context(self.root, backend) + context.facts["native_tsc_hz"] = "2300000000" + result = check(context, records(2300000100, 2300000200), clocksource) + self.assertTrue(result.passed, result.detail) + self.assertIn( + f"host clocksource {clocksource} (evidence)", result.detail + ) + self.assertEqual(context.facts["host_clocksource"], clocksource) + windows = check( + doctor_context(self.root, "whp"), + records(2300000100, 2300000200), + windows=True, + ) + self.assertTrue(windows.passed, windows.detail) + self.assertNotIn("clocksource", windows.detail) # Without H3 (validate-runner), H4 still checks stability. + standalone = check(doctor_context(self.root), records(2300000100, 2300000200)) self.assertTrue(standalone.passed, standalone.detail) self.assertIn("not compared with the backend's rate", standalone.detail) - with ( - patch.object( - doctor, "run_probe", return_value=records(2300000100, 2300009100) - ), - patch.object(doctor, "host_clocksource", return_value="tsc"), - ): - unstable = doctor.check_rate(doctor_context(self.root)) + unstable = check(doctor_context(self.root), records(2300000100, 2300009100)) self.assertFalse(unstable.passed) self.assertIn("unstable", unstable.detail) @@ -763,15 +853,14 @@ def test_host_skew_check_applies_the_bound(self): self.assertFalse(result.passed) self.assertIn(message, result.detail) - def test_guest_warp_check_requires_the_boot_marker_and_the_skew_bound(self): + def test_guest_warp_check_runs_the_idle_schedule_and_applies_the_bound(self): context = doctor_context(self.root) for path in (context.openvmm, context.kernel, context.initrd): path.write_bytes(b"") - text = ( - BOOT_LINE.replace("cpus=4", "cpus=8") - + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" - + warp_output(pairs=28) + boot = ( + BOOT_LINE.replace("cpus=4", "cpus=8") + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" ) + text = boot + warp_output(pairs=28) + warp_output(pairs=28, offset=60) with ( patch.object(doctor.os, "cpu_count", return_value=8), patch.object( @@ -780,21 +869,30 @@ def test_guest_warp_check_requires_the_boot_marker_and_the_skew_bound(self): ): result = doctor.check_guest_warp(context) self.assertTrue(result.passed, result.detail) - self.assertEqual(context.facts["guest_warp_ns"], "15") - command = run.call_args.args[0] + self.assertIn("rounds=2 idle_s=1", result.detail) + self.assertEqual(context.facts["guest_warp_ns"], "60") + command, script, marker = run.call_args.args self.assertEqual(command[command.index("--processors") + 1], "8") - with ( - patch.object(doctor.os, "cpu_count", return_value=6), - patch.object( - doctor, - "run_guest_script", - return_value={"text": warp_output(pairs=6)}, + self.assertEqual(script, time_abi.warp_probe_script() + "nvx-exit 0\n") + self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + cases = ( + (boot + warp_output(pairs=28), "ran 1 rounds instead of 2"), + ( + boot + warp_output(pairs=28) + warp_output(pairs=28, offset=1500), + "in round 2: max_abs_offset_ns=1500", ), - ): - result = doctor.check_guest_warp(doctor_context(self.root)) - self.assertFalse(result.passed) - self.assertIn("vcpus=4", result.detail) - self.assertIn("NVX-TIME-ABI boot marker", result.detail) + (warp_output(pairs=28) * 2, "NVX-TIME-ABI boot marker"), + ) + for output, message in cases: + with ( + self.subTest(message=message), + patch.object(doctor.os, "cpu_count", return_value=8), + patch.object(doctor, "run_guest_script", return_value={"text": output}), + ): + result = doctor.check_guest_warp(context) + self.assertFalse(result.passed) + self.assertIn("vcpus=8", result.detail) + self.assertIn(message, result.detail) missing = doctor.check_guest_warp(doctor_context(self.root / "missing")) self.assertIn("OpenVMM was not found", missing.detail) From b60ba3b3b9897d83aa28d13bb0248c1dddb1000c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:54:08 -0700 Subject: [PATCH 025/188] doc: fold the KVM spike results into the time ABI spec - Hypervisor leaves: zeros are promised only for the explicit list 0x40000006..=0x4000000f and 0x40000080..=0x40000082; every other leaf returns zeros or the out-of-range result (KVM's 256-entry table cannot list the range). Enumerate every hypervisor-range CPUID read of Linux 6.18 as NVX builds it, from mshyperv.c, vmware.c, and hypervisor.c. - Invariant TSC on KVM: KVM 6.3+ hides the bit until its own TSC_INVARIANT_CONTROL is set, so the backend mirrors the guest-visible value that core saves; the TLFS deviation now says MSHV and WHP only. - Rationale for the common offset: Linux 6.6 KVM discards a restored IA32_TSC near boot (458 ms lost on azure-kvm-5). - available_clocksource is tsc alone once the tick is one-shot (C6). - KVM measurements: monotonic tracking, warp, the RCU stall without suppression, and the four MSR exits per boot. - Host access: our account opens /dev/kvm only on azure-kvm-5 and /dev/mshv only on azure-azlinux-5, so cross-VM restore of one generation runs on WHP only; KVM and MSHV gain reachable cross-generation cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 126 ++++++++++++++++++++++++++++------------- 1 file changed, 88 insertions(+), 38 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index fc2e84a5b..54ce44deb 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -84,16 +84,35 @@ configures exactly these six leaves on every backend: sets only `FrequencyRegsAvailable` (bit 8). `0x40000004` recommends nothing and sets the spinlock retry count to "never notify". -Every other leaf in `0x40000006..=0x400000ff` returns either all zeros or the -vendor's architectural out-of-range result (on Intel, the result of the -highest basic leaf for the same subleaf). OpenVMM programs explicit zero -results for `0x40000006..=0x4000000f` and `0x40000080..=0x40000082`, which -Linux and Hyper-V-aware software probe, so they read zero on every backend; -KVM cannot list the whole range in its CPUID table. No leaf returns Hyper-V -feature data, a `VS#1` interface signature at `0x40000081`, or any hypervisor -signature. No base `0x40000100..=0x4000ff00` (step `0x100`) carries a KVM, -Xen, VMware, or other hypervisor signature, so Linux selects only the Hyper-V -platform. +OpenVMM also programs explicit zero results for `0x40000006..=0x4000000f` +and `0x40000080..=0x40000082`, so they read zero on every backend. The +contract promises zeros only for these leaves: every other leaf in +`0x40000006..=0x400000ff` returns either all zeros or the vendor's +architectural out-of-range result (on Intel, the result of the highest basic +leaf for the same subleaf). KVM returns that result for every leaf missing +from its CPUID table, whose 256 entries cannot list the whole range. + +Every leaf beyond the identity leaves that Linux 6.18 reads with this +identity is an explicit zero. Built as NVX builds it (`CONFIG_HYPERVISOR_GUEST` +without `CONFIG_HYPERV`, `CONFIG_KVM_GUEST`, or Xen, ACRN, Jailhouse, and +bhyve guest support), Linux probes only VMware and Hyper-V, and these are all +its reads in the range: + +| Leaf | Reader | Condition | +| --- | --- | --- | +| `0x40000000` | `vmware_platform`, `ms_hyperv_platform`, `ms_hyperv_init_platform` | Always | +| `0x40000003` to `0x40000005` | `ms_hyperv_platform`, `ms_hyperv_init_platform` | Always | +| `0x40000081` | `ms_hyperv_msi_ext_dest_id` | x2APIC available without interrupt remapping | +| `0x40000082` | `ms_hyperv_msi_ext_dest_id` | `0x40000081` EAX is `VS#1`: never | +| `0x4000000a` | `ms_hyperv_init_platform` | Maximum leaf at least `0x4000000a`: never | +| `0x4000000c` | `ms_hyperv_init_platform` | `HV_ISOLATION` in `0x40000003` EBX: never | +| `0x40000010` | `vmware_select_hypercall` | VMware signature at `0x40000000`: never | + +The other explicit zeros serve Hyper-V-aware software that probes further. +No leaf returns Hyper-V feature data, a `VS#1` interface signature at +`0x40000081`, or any hypervisor signature. No base `0x40000100..=0x4000ff00` +(step `0x100`) carries a KVM, Xen, VMware, or other hypervisor signature, so +Linux selects only the Hyper-V platform. ### Synthetic MSRs @@ -180,8 +199,9 @@ The guest must then be in this state: - `/proc/cpuinfo` flags on every CPU include `tsc`, `constant_tsc`, `nonstop_tsc`, `tsc_known_freq`, `tsc_reliable`, `rdtscp`, `hypervisor`, and `arat`, and exclude `tsc_deadline_timer` and `tsc_adjust`. -- `current_clocksource` is `tsc`. `available_clocksource` contains `tsc` and - nothing other than `tsc`, `refined-jiffies`, and `jiffies`. +- `current_clocksource` is `tsc`, and `available_clocksource` is `tsc` + alone: Linux lists `refined-jiffies` and `jiffies` only while the reading + CPU's tick is periodic, which ends at that CPU's first tick. - Ticks are tickless-idle with high-resolution timers. Every online CPU's tick device is `lapic` in one-shot mode from its first tick onward. No `pit` or `hpet` clock event device and no broadcast device exist. @@ -202,8 +222,13 @@ Because the kernel no longer checks cross-vCPU TSC consistency, the from the physical rate `F_d` by up to the rate tolerance after a restore. - `HV_X64_MSR_TSC_INVARIANT_CONTROL` can be cleared after it is set; Hyper-V and KVM raise #GP instead. -- CPUID `0x80000007` EDX[8] is set by the CPU profile from boot. It does not - depend on writes to `HV_X64_MSR_TSC_INVARIANT_CONTROL`. +- On MSHV and WHP, CPUID `0x80000007` EDX[8] is set by the CPU profile from + boot; it does not wait for a write to `HV_X64_MSR_TSC_INVARIANT_CONTROL`. + KVM 6.3 and newer follow the TLFS and hide the bit while KVM's own copy of + the control is 0, so the KVM backend mirrors the guest-visible value into + KVM (see [Backend obligations](#backend-obligations)). Linux writes 1 in + `ms_hyperv_init_platform`, before `identify_boot_cpu` and the APs read the + bit, so the booted guest is identical on every backend. - `0x40000002` carries an NVX signature and the time ABI version, not a Hyper-V build number. - `0x40000005` reports the VP capacity in both EAX and EBX. @@ -225,8 +250,8 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), plus CPUID exits for leaves 1, 6, 7, `0xA`, `0x15`, `0x16`, and `0x80000007` | -| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), plus CPUID exits for leaves 1, 6, 7, `0xA`, `0x15`, `0x16`, and `0x80000007` | +| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | @@ -242,6 +267,18 @@ settled cell on the registered hosts. | Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | | Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | +**KVM common offset.** Restoring each VP's `IA32_TSC` is unreliable on KVM. +Linux 6.6 KVM, which the Azure KVM runners run, treats a host `IA32_TSC` +write within about 1 s of the TSC timeline started at vCPU creation as a +synchronization attempt and discards the written value, the first write of a +restore included; every CI shell snapshot is taken within a second of boot. +With the legacy restore path on `azure-kvm-5`, a restored guest's monotonic +clock advanced 54.7 ms across a 512.5 ms host interval, losing 458 ms. Linux +6.7 applies the heuristic only after a first user-space write +(`user_set_tsc`). `KVM_VCPU_TSC_OFFSET` sets the offset exactly on every +kernel from 5.16, so the backend never writes `IA32_TSC` and core omits the +saved per-VP TSC values from the VP restore. + **WHP decision gate: met.** Suspending partition time, writing the target, verifying the frozen values, and resuming kept every pair of vCPUs within 70 ns over 60 s on bare metal and on both Azure runner generations, so WHP @@ -410,9 +447,10 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this 3. **CI warp probe.** Conformance, restore-matrix, and soak runs execute the guest warp probe after boot and after every restore and fail above 1 µs. 4. **Backend live skew.** Each backend shows that live skew stays within the - bound after release. The spikes measured at most 63 ns on KVM, at most - 516 ns on MSHV (dual-socket bare metal, bounded by the probe's round - trip), and at most 70 ns on WHP. + bound after release. The spikes measured at most 63 ns of ping-pong + offset and 7.2 ns of backward step on KVM, at most 516 ns on MSHV + (dual-socket bare metal, bounded by the probe's round trip), and at most + 70 ns on WHP. The guest warp probe (`nvx-time-probe warp`, built from `guest/common/nvx-time-probe.c` and installed as `/sbin/nvx-time-probe`) @@ -468,7 +506,11 @@ after the read-back); on MSHV it is frozen until the first VP runs. Either way, the guest monotonic advance across a restore lies in `[D, D + R]`, up to the two anchors' pairing errors (each at most 100 µs), where `R` is the time from the restore anchor to the first restored instruction. Wall-clock -repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. +repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. The KVM spike +kept guest `CLOCK_MONOTONIC` within −0.09 to +2.8 ms of the host interval +across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive +part came from a host sample taken before the per-VP TSC save, which the +paired capture anchor removes. ## Restore algorithm @@ -782,7 +824,7 @@ partition-wide on every backend, so `C3` reads them on CPU 0 only. | `C3` | MSR `0x40000002` equals the CPU's VP index on every CPU; on CPU 0, `0x40000022` equals `F` with `floor(F / 1000)` equal to the kernel's `cpu MHz` in kHz, `0x40000023` equals 1,000,000,000 or 200,000,000, `0x40000118` equals 1, and reads of `0x40000000`, `0x40000001`, and `0x40000020` fail with `EIO` | boot; restore (CPU 0 `0x40000022` only) | | `C4` | The kernel log contains `Hypervisor detected: Microsoft Hyper-V`, `Hyper-V: privilege flags low 0x8860,`, `Hyper-V: LAPIC Timer Frequency: 0x989680` or `0x1e8480`, and `clocksource: Switched to clocksource tsc` as its last clocksource switch; it contains no record matching a watcher pattern other than `G_CLOCKSOURCE_SWITCH`, and none containing `Fast TSC calibration`, `Refined TSC clocksource calibration`, `kvm-clock`, or `APIC timer: using supplied frequency` | boot | | `C5` | `/proc/cpuinfo` flags as listed in [Clocksource, tick, and PIT](#clocksource-tick-and-pit) on every CPU | boot, restore (new CPUs) | -| `C6` | `current_clocksource` is `tsc`; `available_clocksource` contains only `tsc`, `refined-jiffies`, and `jiffies` | boot, capture, restore | +| `C6` | `current_clocksource` is `tsc`; `available_clocksource` lists `tsc` and no other clocksource except `refined-jiffies` and `jiffies`, which Linux lists only before the reading CPU's first tick (after boot it is `tsc` alone) | boot, capture, restore | | `C7` | `/proc/timer_list`: every online CPU's tick device is `lapic` with `hrtimer_interrupt` in one-shot mode; no `pit` or `hpet` device; no broadcast device | boot, capture, restore (deferred) | | `C8` | `/sys/bus/vmbus` and `/sys/devices/system/cpu/cpufreq/policy0` are absent; `rcu_cpu_stall_suppress` is 0 and `rcu_cpu_stall_timeout` is 21 | boot | | `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | @@ -802,7 +844,7 @@ check runs on every online CPU: | ID | Check | | --- | --- | -| `X1` | Every leaf `0x40000006..=0x400000ff` returns all zeros or the Intel out-of-range result (the highest basic leaf's result for the same subleaf), as the [identity rules](#hypervisor-identity) require | +| `X1` | The explicit zero leaves are zero, and every other leaf in `0x40000006..=0x400000ff` returns all zeros or the Intel out-of-range result (the highest basic leaf's result for the same subleaf), as the [identity rules](#hypervisor-identity) require | | `X2` | No base `0x40000100..=0x4000ff00` (step `0x100`) carries `KVMKVMKVM` or another hypervisor signature; the NVX kernel has no KVM guest support, so the boot check leaves this static backend property to CI | | `X3` | Every `C3` MSR | | `X4` | `0x40000118` accepts writes of 0 and 1, each read back, and rejects 2; the check leaves it at 1 | @@ -939,6 +981,10 @@ event and power off with status 195. ### RCU grace period release +Stall suppression is required: on KVM, restores after 30 s of downtime +without it report `rcu_preempt self-detected stall` at 1 and 8 vCPUs, because +jiffies jump by `D`; with it, no stall appears at 1, 2, 4, or 8 vCPUs. + Stall suppression may be released only after the grace period that was in flight at capture has ended, or the restored guest reports a false stall. `membarrier(MEMBARRIER_CMD_GLOBAL)` does not guarantee this: Linux 6.18 skips @@ -1107,8 +1153,8 @@ metrics, in their log and the job summary; an unknown generation fails qualification explicitly. Runner labels are not used and runners are not re-registered: per-PR CI captures and restores on the same runner, and same-generation cross-VM restore is validated by the fleet restore matrix -(`p6-restore-matrix`) on the registered hosts. Routing CI jobs by generation -labels is optional future work. +(`p6-restore-matrix`) on the hosts our account can use. Routing CI jobs by +generation labels is optional future work. ## Performance expectations and acceptance gate @@ -1155,14 +1201,17 @@ and `0x40000118` save and restore; capabilities derivation (`hv1` and 2 through 5; packet v4 and time-sample golden vectors and their guest-side parser; four-byte portb reads; and the generation counter. -**Conformance.** The boot and restore checks pass on all 18 registered hosts -at 1, 2, 4, and 8 vCPUs, plus the exhaustive CI check and the warp probe on -every backend. +**Conformance.** The boot and restore checks pass at 1, 2, 4, and 8 vCPUs on +all 18 registered hosts, plus the exhaustive CI check and the warp probe on +every backend. The fleet runs them on the hosts our SSH account can use, +which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 +and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on +the other KVM and MSHV runners, which only CI jobs exercise. **Restore matrix.** Every case runs with zero violations; rejected cases must fail with the listed code. Per-PR CI runs the same-host cases on one runner; -the full matrix, including the cross-VM cases, runs on the registered hosts -as the fleet restore matrix. +the full matrix, including the cross-VM cases, runs on the hosts our account +can use as the fleet restore matrix. | Case | Hosts | Expected | | --- | --- | --- | @@ -1173,8 +1222,8 @@ as the fleet restore matrix. | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | -| Across VMs of one generation | `azure-kvm-1` to `-2`; `azure-azlinux-3` to `-4`; `azure-windows-1` to `-2`; `azure-windows-3` to `-4` | Restored | -| Across generations | `azure-windows-1` (8370C) to `-3` (8573C) | `E_CPU_GENERATION` | +| Across VMs of one generation | `azure-windows-1` to `-2`; `azure-windows-3` to `-4`. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | +| Across generations | `azure-windows-1` (8370C) to `-3` (8573C); prometheus32 to `azure-kvm-5` (KVM); prometheus30 to `azure-azlinux-5` (MSHV) | `E_CPU_GENERATION` | | Host without invariant TSC at the host OS | `azure-azlinux-2` (out of CI rotation) | Qualified only if `H4` and `H6` pass; otherwise `nvx.py doctor` fails | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | @@ -1184,11 +1233,11 @@ as the fleet restore matrix. No test reboots a host. The orchestrator asks the user before any real reboot. -**Reliability.** Repeated snapshot scenarios on every registered host at 1, -2, 4, and 8 vCPUs, with the warp probe and the watcher active, report no RCU, -soft-lockup, hung-task, clocksource, or warp message. The CI debug-kernel -variant runs the same-host cases with the soft-lockup and hung-task detectors -enabled. +**Reliability.** Repeated snapshot scenarios at 1, 2, 4, and 8 vCPUs on the +hosts the fleet can use (see Conformance), with the warp probe and the +watcher active, report no RCU, soft-lockup, hung-task, clocksource, or warp +message. The CI debug-kernel variant runs the same-host cases with the +soft-lockup and hung-task detectors enabled. **Performance.** The gate above, computed by the performance agent from the CI benchmark matrix. @@ -1267,8 +1316,9 @@ Migration impact: - Hosts that fail qualification, such as `azure-azlinux-2`, cannot run microVMs until replaced. - Per-PR CI captures and restores on the same runner. Same-generation - cross-VM restore is validated by the fleet restore matrix on the - registered hosts, not in per-PR CI. + cross-VM restore is validated by the fleet restore matrix on WHP only; no + usable KVM or MSHV pair of one generation exists, so the simulated host + reboot covers those backends. - The `cold_start_clocksource` metric on KVM changes meaning (from `kvm-clock` to `tsc`). The change is accepted without an exemption; a gate failure on it is investigated as a regression. From 7085cdf7931451c9e94781c0dd8ac59932cb128c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:16:06 -0700 Subject: [PATCH 026/188] doc: record the measured time-sample epsilon on KVM and MSHV Replace TBD(kvm) and TBD(mshv) in the uncertainty bounds with measured values. A guest probe bracketed each 0xa7 selector write with CLOCK_REALTIME reads, against a release build of OpenVMM ea570db7e: two runs of 2,000 samples per boot at 1, 2, and 4 or 8 vCPUs. - KVM: 4.1 us p50 on prometheus32 (bare metal, Linux 7.0) and 3.8 to 4.1 us on azure-kvm-5 (nested, Linux 6.6). - MSHV: 6.3 us p50 on prometheus30 (bare metal) and 12.7 to 13.0 us on azure-azlinux-5 (nested). Every value is far below the 1 ms restore and boot bound. A sample above the 50 us discipline bound is rare (worst 102 us), and the poll retries it. WHP stays TBD(whp). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 54ce44deb..aaa3b493e 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -775,14 +775,23 @@ otherwise, and then fails with a stable code: | Discipline poll (time samples) | 50 µs | 3 | `G_SAMPLE_UNCERTAIN`: the poll is skipped and the code is recorded in the state file and on the console; never fatal | `epsilon` is half the round trip of one PMIO write exit plus two -`CLOCK_REALTIME` reads, so it is expected to stay far below both bounds on -every backend. The backend agents measure it: +`CLOCK_REALTIME` reads, so it stays far below both bounds on every backend. +Measured from a guest with a release build of OpenVMM (`ea570db7e`) and a +probe that brackets each selector write with `CLOCK_REALTIME` reads, two runs +of 2,000 samples per boot at 1, 2, and 4 or 8 vCPUs, which made no +difference: -| Backend | Expected `epsilon` (p50 / p99) | -| --- | --- | -| KVM | TBD(kvm) | -| MSHV | TBD(mshv) | -| WHP | TBD(whp) | +| Backend | Host | `epsilon` p50 / p99 | Worst sample | +| --- | --- | --- | --- | +| KVM | prometheus32 (bare metal, Linux 7.0) | 4.1 µs / 5 to 15 µs | 25 µs | +| KVM | `azure-kvm-5` (nested, Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | +| MSHV | prometheus30 (bare metal) | 6.3 µs / 7.1 µs | 37 µs | +| MSHV | `azure-azlinux-5` (nested) | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | +| WHP | | TBD(whp) | | + +The second run of each boot overlaps the console's drain of the first run's +output, and it has the higher KVM p99s. A sample above the 50 µs discipline +bound is rare, and the poll retries it. ### Generation counter and generation ID From a25dffa4e0e3e115b2754597995ff925229c1058 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:16:16 -0700 Subject: [PATCH 027/188] guest: add nvx-time exhaustive, the CI-only exhaustive check The spec moved the CI conformance suite's exhaustive check into the guest as `nvx-time exhaustive`. Run checks X1 to X6 on every online CPU, pinned there for inline CPUID, with MSRs through /dev/cpu/N/msr: - X1: every leaf 0x40000006..=0x400000ff reads zero, or the Intel out-of-range result except at the explicit zero leaves; - X2: no base 0x40000100..=0x4000ff00 reads anything else, so none carries a hypervisor signature; - X3: every C3 property on this CPU; - X4: 0x40000118 accepts 0 and 1, each read back, rejects 2, and is left at 1; - X5: writes to the identity MSRs fail; - X6: reads of IA32_TSC_ADJUST and IA32_TSC_DEADLINE fail. It prints one NVX-TIME-ABI-EXHAUSTIVE line per check and CPU, then a summary, and exits 1 on any failure; it never powers off. On KVM with openvmm 7500b7472 every check passes after plain restores and after 8-to-4 and 8-to-8 prefix restores. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 4 +- guest/common/nvx-time.c | 314 +++++++++++++++++++++++++++++++++++++ scripts/test_guest_time.py | 24 +++ 3 files changed, 341 insertions(+), 1 deletion(-) diff --git a/doc/build.md b/doc/build.md index 41bd6b3b4..7b90db740 100644 --- a/doc/build.md +++ b/doc/build.md @@ -150,7 +150,9 @@ skew bound. `nvx-time` implements the time ABI's checks, the violation watcher, the restore packet v4 and time-sample parsers, the snapshot agent's time steps, and the wall-clock discipline. Its boot check also fails when the kernel's boot-time W+X audit reports a writable and -executable mapping. The +executable mapping. `nvx-time exhaustive` runs the CI-only exhaustive check +(checks X1 to X6 on every online CPU); it reports and exits, and production +boots never run it. The `--report-only` flag (or the `nvx_time_abi=report-only` kernel command-line token) reports violations without powering off, for runs against VMMs that predate the ABI. Both time tools are linked statically against musl diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 4f9ccd8f2..53db1d49f 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -1865,6 +1865,301 @@ static bool check_cpuinfo(struct checks *checks, const bool *selected, return cpuinfo_result(&parse); } +// --------------------------------------------------------------------------- +// Exhaustive check (CI only) +// --------------------------------------------------------------------------- + +static int msr_write(int fd, uint32_t msr, uint64_t value) +{ + ssize_t count = pwrite(fd, &value, sizeof(value), (off_t)msr); + + if (count == (ssize_t)sizeof(value)) + return 0; + return count < 0 ? -errno : -EIO; +} + +// The leaves that OpenVMM programs as explicit zeros. +static bool explicit_zero_leaf(uint32_t leaf) +{ + return (leaf >= 0x40000006 && leaf <= 0x4000000f) || + (leaf >= 0x40000080 && leaf <= 0x40000082); +} + +// X1 and X2 for one CPUID result: all zeros or, except for an explicit zero +// leaf, the Intel out-of-range result (the highest basic leaf's result for +// the same subleaf). +static bool exhaustive_leaf_ok(uint32_t leaf, const uint32_t value[4], + const uint32_t out_of_range[4], char *detail, + size_t size) +{ + char signature[13]; + + if (all_zero(value) || + (!explicit_zero_leaf(leaf) && + memcmp(value, out_of_range, 4 * sizeof(uint32_t)) == 0)) + return true; + memcpy(signature, &value[1], 4); + memcpy(signature + 4, &value[2], 4); + memcpy(signature + 8, &value[3], 4); + signature[12] = '\0'; + for (int index = 0; index < 12; index++) { + if (signature[index] < 0x20 || signature[index] > 0x7e) + signature[index] = '.'; + } + snprintf(detail, size, + "leaf 0x%08" PRIx32 " is %08" PRIx32 " %08" PRIx32 " %08" PRIx32 + " %08" PRIx32 " (signature \"%s\"), expected zero%s", + leaf, value[0], value[1], value[2], value[3], signature, + explicit_zero_leaf(leaf) ? "" : " or the highest basic leaf"); + return false; +} + +struct exhaustive { + int failures; + uint64_t tsc_hz; + uint64_t lapic_hz; + struct checks cpuinfo; +}; + +static void exhaustive_report(struct exhaustive *state, const char *id, int cpu, + bool passed, const char *detail) +{ + char escaped[DETAIL_MAX]; + + escape_text(detail, escaped, sizeof(escaped) - 1); + printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 check=%s cpu=%d status=%s " + "detail=\"%s\"\n", + id, cpu, passed ? "pass" : "fail", escaped); + if (!passed) + state->failures++; +} + +// X1 (every leaf 0x40000006..=0x400000ff) when STEP is 1, or X2 (every base +// 0x40000100..=0x4000ff00) when STEP is 0x100, on the current CPU. +static void exhaustive_leaves(struct exhaustive *state, const char *id, int cpu, + uint32_t first, uint32_t last, uint32_t step, + const uint32_t out_of_range[4]) +{ + char detail[DETAIL_MAX] = ""; + int failing = 0; + int count = 0; + + for (uint32_t leaf = first; leaf <= last; leaf += step, count++) { + char reason[DETAIL_MAX]; + uint32_t value[4]; + + cpuid(leaf, 0, value); + if (!exhaustive_leaf_ok(leaf, value, out_of_range, reason, + sizeof(reason)) && + failing++ == 0) + snprintf(detail, sizeof(detail), "%s", reason); + } + if (failing == 0) { + snprintf(detail, sizeof(detail), "%d %s read zero or the highest basic leaf", + count, step == 1 ? "leaves" : "bases"); + } else if (failing > 1) { + size_t length = strlen(detail); + + snprintf(detail + length, sizeof(detail) - length, "; %d of %d fail", + failing, count); + } + exhaustive_report(state, id, cpu, failing == 0, detail); +} + +// X3: every C3 property on this CPU. +static void exhaustive_x3(struct exhaustive *state, int fd, int cpu) +{ + static const uint32_t faulting[] = {0x40000000, 0x40000001, 0x40000020}; + char detail[DETAIL_MAX]; + uint64_t index = 0; + uint64_t tsc = 0; + uint64_t lapic = 0; + uint64_t control = 0; + uint64_t value; + bool ok = false; + int error; + + if ((error = msr_read(fd, 0x40000002, &index)) != 0 || + index != (uint64_t)cpu) + snprintf(detail, sizeof(detail), + "MSR 0x40000002 (VP index) is %" PRIu64 " (error %d)", index, + error); + else if ((error = msr_read(fd, 0x40000022, &tsc)) != 0 || tsc == 0 || + (state->tsc_hz != 0 && tsc != state->tsc_hz)) + snprintf(detail, sizeof(detail), + "MSR 0x40000022 (TSC rate) is %" PRIu64 + " (error %d; first CPU %" PRIu64 ")", + tsc, error, state->tsc_hz); + else if (state->cpuinfo.cpu_khz_known[cpu] && + tsc / 1000 != state->cpuinfo.cpu_khz[cpu]) + snprintf(detail, sizeof(detail), + "floor(F / 1000) is %" PRIu64 " kHz but cpu MHz is %" PRIu64 + " kHz", + tsc / 1000, state->cpuinfo.cpu_khz[cpu]); + else if ((error = msr_read(fd, 0x40000023, &lapic)) != 0 || + (lapic != 1000000000 && lapic != 200000000) || + (state->lapic_hz != 0 && lapic != state->lapic_hz)) + snprintf(detail, sizeof(detail), + "MSR 0x40000023 (LAPIC rate) is %" PRIu64 + " (error %d; first CPU %" PRIu64 ")", + lapic, error, state->lapic_hz); + else if ((error = msr_read(fd, 0x40000118, &control)) != 0 || control != 1) + snprintf(detail, sizeof(detail), + "MSR 0x40000118 is %" PRIu64 " (error %d)", control, error); + else + ok = true; + for (size_t item = 0; ok && item < ARRAY_SIZE(faulting); item++) { + error = msr_read(fd, faulting[item], &value); + if (error != -EIO) { + ok = false; + snprintf(detail, sizeof(detail), + "MSR 0x%08" PRIx32 " read did not fail with EIO (error %d)", + faulting[item], error); + } + } + if (ok) { + snprintf(detail, sizeof(detail), + "vp_index=%d tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 + " invariant_control=1", + cpu, tsc, lapic); + if (state->tsc_hz == 0) { + state->tsc_hz = tsc; + state->lapic_hz = lapic; + } + } + exhaustive_report(state, "X3", cpu, ok, detail); +} + +// X4: MSR 0x40000118 accepts 0 and 1, each read back, and rejects 2. It is +// left at 1, the value Linux wrote at boot, whatever the outcome. +static void exhaustive_x4(struct exhaustive *state, int fd, int cpu) +{ + char detail[DETAIL_MAX] = "0 and 1 accepted, 2 rejected, left at 1"; + uint64_t value = 0; + bool ok = true; + int error = 0; + + for (uint64_t written = 0; ok && written <= 1; written++) { + if ((error = msr_write(fd, 0x40000118, written)) != 0 || + (error = msr_read(fd, 0x40000118, &value)) != 0 || + value != written) { + ok = false; + snprintf(detail, sizeof(detail), + "after writing %" PRIu64 ", MSR 0x40000118 is %" PRIu64 + " (error %d)", + written, value, error); + } + } + if (ok && (error = msr_write(fd, 0x40000118, 2)) != -EIO) { + ok = false; + snprintf(detail, sizeof(detail), + "write of 2 to MSR 0x40000118 did not fail with EIO (error %d)", + error); + } + (void)msr_write(fd, 0x40000118, 1); + if (ok && (msr_read(fd, 0x40000118, &value) != 0 || value != 1)) { + ok = false; + snprintf(detail, sizeof(detail), + "MSR 0x40000118 is %" PRIu64 " after the check", value); + } + exhaustive_report(state, "X4", cpu, ok, detail); +} + +// X5: writes to the identity MSRs that are read-only or absent fail. X6: +// reads of IA32_TSC_ADJUST and IA32_TSC_DEADLINE fail. +static void exhaustive_faults(struct exhaustive *state, int fd, int cpu, + bool writes) +{ + static const uint32_t read_only[] = {0x40000000, 0x40000001, 0x40000002, + 0x40000020, 0x40000022, 0x40000023}; + static const uint32_t absent[] = {0x3b, 0x6e0}; + const uint32_t *list = writes ? read_only : absent; + size_t length = writes ? ARRAY_SIZE(read_only) : ARRAY_SIZE(absent); + char detail[DETAIL_MAX]; + bool ok = true; + + snprintf(detail, sizeof(detail), "%zu %s fail with EIO", length, + writes ? "writes" : "reads"); + for (size_t item = 0; ok && item < length; item++) { + uint64_t value = 0; + int error = writes ? msr_write(fd, list[item], 0) + : msr_read(fd, list[item], &value); + + if (error != -EIO) { + ok = false; + snprintf(detail, sizeof(detail), + "%s MSR 0x%" PRIx32 " did not fail with EIO (error %d)", + writes ? "write of 0 to" : "read of", list[item], error); + } + } + exhaustive_report(state, writes ? "X5" : "X6", cpu, ok, detail); +} + +// The CI-only exhaustive check: X1 to X6 on every online CPU. It reports and +// exits, never powers off, and production boots never run it. +static int exhaustive_run(void) +{ + static const char *const ids[] = {"X1", "X2", "X3", "X4", "X5", "X6"}; + struct cpuinfo_parse parse; + struct exhaustive state; + bool cpus[MAX_CPUS]; + cpu_set_t original; + char text[256]; + int online; + + memset(&state, 0, sizeof(state)); + if (read_text("/sys/devices/system/cpu/online", text, sizeof(text)) < 0 || + (online = parse_cpu_list(text, cpus)) <= 0) { + printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 status=fail cpus=0 failures=1\n"); + return 1; + } + cpuinfo_init(&parse, &state.cpuinfo, cpus, false); + (void)for_each_file_line("/proc/cpuinfo", cpuinfo_line, &parse); + if (sched_getaffinity(0, sizeof(original), &original) != 0) + CPU_ZERO(&original); + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + char detail[DETAIL_MAX]; + uint32_t basic[4]; + uint32_t out_of_range[4]; + char path[64]; + int fd; + + if (!cpus[cpu]) + continue; + if (pin_to_cpu(cpu) != 0) { + for (size_t item = 0; item < ARRAY_SIZE(ids); item++) + exhaustive_report(&state, ids[item], cpu, false, + "cannot run on this CPU"); + continue; + } + cpuid(0, 0, basic); + cpuid(basic[0], 0, out_of_range); + exhaustive_leaves(&state, "X1", cpu, 0x40000006, 0x400000ff, 1, + out_of_range); + exhaustive_leaves(&state, "X2", cpu, 0x40000100, 0x4000ff00, 0x100, + out_of_range); + snprintf(path, sizeof(path), "/dev/cpu/%d/msr", cpu); + fd = open(path, O_RDWR | O_CLOEXEC); + if (fd < 0) { + snprintf(detail, sizeof(detail), "open %s: %s", path, + strerror(errno)); + for (size_t item = 2; item < ARRAY_SIZE(ids); item++) + exhaustive_report(&state, ids[item], cpu, false, detail); + continue; + } + exhaustive_x3(&state, fd, cpu); + exhaustive_x4(&state, fd, cpu); + exhaustive_faults(&state, fd, cpu, true); + exhaustive_faults(&state, fd, cpu, false); + close(fd); + } + if (CPU_COUNT(&original) > 0) + (void)sched_setaffinity(0, sizeof(original), &original); + printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 status=%s cpus=%d failures=%d\n", + state.failures == 0 ? "ok" : "fail", online, state.failures); + return state.failures == 0 ? 0 : 1; +} + // C6: `tsc` is current, and only tsc, refined-jiffies, or jiffies is // available. static bool check_clocksource_values(struct checks *checks, const char *current, @@ -3515,6 +3810,22 @@ static int cmd_test(int argc, char **argv) console_write(argv[0]); return 0; } + if (strcmp(name, "exhaustive-leaf") == 0 && argc == 9) { + uint32_t value[4]; + uint32_t out_of_range[4]; + char detail[DETAIL_MAX]; + + for (int index = 0; index < 4; index++) { + value[index] = (uint32_t)strtoul(argv[1 + index], NULL, 0); + out_of_range[index] = (uint32_t)strtoul(argv[5 + index], NULL, 0); + } + if (exhaustive_leaf_ok((uint32_t)strtoul(argv[0], NULL, 0), value, + out_of_range, detail, sizeof(detail))) + printf("pass\n"); + else + printf("fail %s\n", detail); + return 0; + } if (strcmp(name, "boot-log") == 0) return test_lines(argc, argv, LINES_BOOT_LOG); if (strcmp(name, "syslog") == 0) @@ -3580,6 +3891,7 @@ static int usage(void) " restore-finish [--ack] [--new-cpus LIST]\n" " sample\n" " generation-id\n" + " exhaustive\n" " test NAME ARGUMENTS...\n"); return 2; } @@ -3624,5 +3936,7 @@ int main(int argc, char **argv) return cmd_sample(); if (strcmp(command, "generation-id") == 0 && count == 1) return cmd_generation_id(); + if (strcmp(command, "exhaustive") == 0 && count == 1) + return exhaustive_run(); return usage(); } diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 37f7efbc5..6d103af75 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -521,6 +521,30 @@ def test_daemon_lines_start_on_a_fresh_console_line(self): self.assertEqual(self.run_test("console", line, "sync"), line) self.assertEqual(self.run_test("console", line, "async"), "\n" + line) + def test_exhaustive_leaf_rules(self): + # X1 and X2: a leaf reads zero or the highest basic leaf's result, + # except the explicit zero leaves, which read zero. + out_of_range = ("0x00000d0b", "0x00000001", "0x00000002", "0x00000003") + zero = ("0", "0", "0", "0") + + def leaf(number: str, value: tuple[str, ...]) -> str: + return self.run_test( + "exhaustive-leaf", number, *value, *out_of_range + ).strip() + + self.assertEqual(leaf("0x40000006", zero), "pass") + self.assertEqual(leaf("0x40000010", out_of_range), "pass") + self.assertEqual(leaf("0x40000100", zero), "pass") + self.assertEqual(leaf("0x40000100", out_of_range), "pass") + explicit = leaf("0x40000081", out_of_range) + self.assertTrue(explicit.startswith("fail leaf 0x40000081 is "), explicit) + self.assertTrue(explicit.endswith("expected zero"), explicit) + # KVM's signature leaf moved to a higher base. + kvm = ("0x40000101", "0x4b4d564b", "0x564b4d56", "0x0000004d") + signature = leaf("0x40000100", kvm) + self.assertIn('(signature "KVMKVMKVM...")', signature) + self.assertTrue(signature.endswith("or the highest basic leaf"), signature) + def timer_list( self, sections: list[tuple[int, str, int, str, int]], From 67ae4ab744aa08d6ea282ea4a51aae239788a7c4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:17:57 -0700 Subject: [PATCH 028/188] doc: correct the KVM and MSHV epsilon ranges State the full ranges across every run: prometheus32's worst sample was 33 us (first round) and its p99 starts at 4.4 us; prometheus30's p50 and p99 are ranges too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index aaa3b493e..8c36a1a7c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -783,9 +783,9 @@ difference: | Backend | Host | `epsilon` p50 / p99 | Worst sample | | --- | --- | --- | --- | -| KVM | prometheus32 (bare metal, Linux 7.0) | 4.1 µs / 5 to 15 µs | 25 µs | +| KVM | prometheus32 (bare metal, Linux 7.0) | 4.0 to 4.1 µs / 4 to 15 µs | 33 µs | | KVM | `azure-kvm-5` (nested, Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | -| MSHV | prometheus30 (bare metal) | 6.3 µs / 7.1 µs | 37 µs | +| MSHV | prometheus30 (bare metal) | 6.3 to 6.4 µs / 6.8 to 7.1 µs | 37 µs | | MSHV | `azure-azlinux-5` (nested) | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | | WHP | | TBD(whp) | | From 2af3168864f42374e2a3661deaafd1ce69d55a52 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:40:07 -0700 Subject: [PATCH 029/188] guest: give the Ubuntu guest enough memory to unpack its initramfs The Ubuntu initramfs unpacks to about 100 MiB in about 6,700 files, and the kernel unpacks it into a tmpfs root capped at half of RAM. At the 256 MiB default the unpack fails ("Initramfs unpacking failed: write error", with the release v0.1.0-dev.89c192e79f0d image and today's alike). The guest then boots a truncated root that lacks everything after the failure point, and its writes fail. A --restore-processors restore of the Ubuntu guest under the time ABI failed for this reason: nvx-snapshot could not write /etc/machine-id (EIO). 320 MiB is the smallest size measured to unpack the image completely (azure-kvm-5: 256 MiB fails, 320 and 384 MiB pass). Make 512 MiB the default, which leaves room for the image to grow. The Ubuntu sandbox layer runs under the Alpine control initramfs and keeps 256 MiB. Also list the module lockdown in the documented Ubuntu boot sequence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/guests/ubuntu-guest.md | 16 ++++++++++------ doc/run.md | 11 ++++++----- scripts/nvx_tools/guests.py | 2 +- scripts/test_microvm_tests.py | 4 ++-- scripts/test_nvx_tools.py | 4 ++-- 5 files changed, 21 insertions(+), 16 deletions(-) diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 6d8821f21..00bd180d3 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -243,12 +243,15 @@ Initial descriptors: | Guest | Initramfs | Default memory | Sandbox control | | --- | --- | ---: | --- | | `alpine` | `initramfs.cpio.gz` | 128 MiB | yes | -| `ubuntu` | `initramfs-ubuntu.cpio.gz` | 256 MiB initially | no | +| `ubuntu` | `initramfs-ubuntu.cpio.gz` | 512 MiB | no | -The Ubuntu memory default is an engineering starting point, not a permanent -ABI. Ubuntu Base contains approximately 86 MiB of uncompressed regular-file -payload before supplemental packages and NVX helpers. CI must measure the -actual boot high-water mark before documenting a supported minimum. +The Ubuntu memory default is an engineering choice, not a permanent ABI. The +initramfs unpacks to about 100 MiB in about 6,700 files, and the kernel +unpacks it into a tmpfs root capped at half of RAM. At 256 MiB the unpack +fails (`Initramfs unpacking failed: write error`) and the guest boots with a +truncated root that cannot take writes such as `/etc/machine-id`. The +smallest size measured to unpack it completely is 320 MiB, and 512 MiB leaves +room for the image to grow and for the workload. ### Common build operations @@ -433,7 +436,8 @@ The Ubuntu initramfs boot remains identical to Alpine at the machine level: 2. OpenVMM loads the selected Ubuntu initramfs as the Linux direct initrd. 3. OpenVMM prepends its xe9/hvc console parameters and fixed virtio-mmio discovery. -4. `/init` mounts procfs, sysfs, devtmpfs, and tmpfs. +4. `/init` mounts procfs, sysfs, and devtmpfs, switches kernel module loading + off, and mounts tmpfs. 5. `/init` configures loopback, optional static networking, and optional HostFs. 6. `/init` handles an explicit `nvx_exec` workload or opens a root shell. diff --git a/doc/run.md b/doc/run.md index 7859e2db3..f3277e1a9 100644 --- a/doc/run.md +++ b/doc/run.md @@ -25,7 +25,7 @@ A successful Alpine boot prints both `ALPINE-MICROVM-BOOT-OK` and python3 scripts/nvx.py run --guest ubuntu ``` -Ubuntu defaults to 256 MiB and prints `NVX-GUEST-BOOT-OK: ubuntu`. It is +Ubuntu defaults to 512 MiB and prints `NVX-GUEST-BOOT-OK: ubuntu`. It is Ubuntu userland with the NVX kernel, not a stock Ubuntu kernel or systemd VM. Boot Azure Linux 3.0 userland the same way with `--guest azurelinux`. The kernel unpacks its rootfs into a RAM filesystem capped at half of guest memory, @@ -95,10 +95,11 @@ If the artifacts are already installed in the repository layout, use `openvmm/target/release/openvmm[.exe]`, `build/vmlinux`, and `build/initramfs.cpio.gz` instead of the paths above. -For Ubuntu userland, use 256 MiB initially and select +For Ubuntu userland, use 512 MiB and select `guest/initramfs-ubuntu.cpio.gz` or -`build/initramfs-ubuntu.cpio.gz` as the initrd. The kernel path remains -unchanged. +`build/initramfs-ubuntu.cpio.gz` as the initrd. Below 320 MiB, the kernel +cannot unpack the whole Ubuntu initramfs and boots a truncated root. The +kernel path remains unchanged. Direct OpenVMM launches accept generic directional network defaults: @@ -238,7 +239,7 @@ translations and additions: | `nvx.py run` | Direct OpenVMM option | | --- | --- | | `--guest alpine` | `--initrd .../initramfs.cpio.gz` on a fresh boot | -| `--guest ubuntu` | `--initrd .../initramfs-ubuntu.cpio.gz` and a 256 MiB default on a fresh boot | +| `--guest ubuntu` | `--initrd .../initramfs-ubuntu.cpio.gz` and a 512 MiB default on a fresh boot | | `--guest azurelinux` | `--initrd .../initramfs-azurelinux.cpio.gz` and a 512 MiB default on a fresh boot | | `--hypervisor auto` | `--hypervisor kvm` on Linux or `--hypervisor whp` on Windows | | `--memory-mib N` | `--memory NM` | diff --git a/scripts/nvx_tools/guests.py b/scripts/nvx_tools/guests.py index 8368b73dd..ca3ed9667 100644 --- a/scripts/nvx_tools/guests.py +++ b/scripts/nvx_tools/guests.py @@ -57,7 +57,7 @@ def package_manifest_name(self) -> str: architecture=UbuntuBuildConstants.ARCHITECTURE, initramfs_name=UbuntuBuildConstants.INITRAMFS_NAME, package_manifest_format=UbuntuBuildConstants.PACKAGE_MANIFEST_FORMAT, - default_memory_mib=256, + default_memory_mib=512, boot_marker=b"NVX-GUEST-BOOT-OK: ubuntu", sandbox_control=False, native_build_supported=True, diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 3c0c5cf8e..3c55b917e 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2546,7 +2546,7 @@ def test_runner_dispatches_public_l3_l4_egress_acceptance(self): def require(path: Path, _description: str) -> Path: return path - for guest, memory_mib in (("alpine", 128), ("ubuntu", 256)): + for guest, memory_mib in (("alpine", 128), ("ubuntu", 512)): with ( self.subTest(guest=guest), tempfile.TemporaryDirectory() as temporary, @@ -3538,7 +3538,7 @@ def require(path: Path, _description: str) -> Path: BuildConstants.BUILD_DIR / "initramfs-ubuntu.cpio.gz", requested, ) - self.assertEqual(run_guest_boot.call_args.kwargs["memory_mib"], 256) + self.assertEqual(run_guest_boot.call_args.kwargs["memory_mib"], 512) def test_runner_omits_console_snapshot_from_ubuntu_defaults(self): def require(path: Path, _description: str) -> Path: diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index e1a0377bb..f081c1598 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -735,7 +735,7 @@ def require(path: Path, _description: str) -> Path: nvx.command_run(args) command = format_command.call_args.args[0] - self.assertEqual(command[command.index("--memory") + 1], "256M") + self.assertEqual(command[command.index("--memory") + 1], "512M") self.assertEqual( Path(command[command.index("--initrd") + 1]).name, "initramfs-ubuntu.cpio.gz", @@ -4869,7 +4869,7 @@ def test_guest_descriptors_preserve_alpine_and_select_guest_outputs(self): ubuntu_guest.initramfs_name, "initramfs-ubuntu.cpio.gz", ) - self.assertEqual(ubuntu_guest.default_memory_mib, 256) + self.assertEqual(ubuntu_guest.default_memory_mib, 512) self.assertFalse(ubuntu_guest.sandbox_control) self.assertEqual( azurelinux_guest.initramfs_name, From e03be986ed0b3e5f166004e498d3331c8ef5208e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:44:30 -0700 Subject: [PATCH 030/188] scripts: say what Azure WHP hosts lack in invariant TSC Qualifying the eleven in-rotation runners showed that the Windows host OS on every Azure WHP runner (8370C and 8573C) reads the CPUID invariant-TSC bit as set. What Azure WHP lacks is the TscInvariantSupport capability in the WHP feature banks, so it cannot offer invariant TSC to a partition, and OpenVMM sets the bit through the CPUID override. The doctor comment and the H2 description said the hosts lack the flag; state what they lack instead. Behavior is unchanged: the host bit stays evidence only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 2 +- scripts/nvx_tools/doctor.py | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 5a07ce801..184fcee20 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -179,7 +179,7 @@ brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. | Check | Implementation | | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | -| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check: Azure WHP hosts lack them while their guests measure tens of nanoseconds of skew | +| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names; OpenVMM's interim `interim.host..v1` profile is accepted until it selects catalog profiles. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | | H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 426338219..0123fb819 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -64,9 +64,10 @@ GUEST_MEMORY_MIB = 128 # Qualification gates only on measured properties, alike on every backend: # the guest warp probe, the TSC rate stability, and the CPU profile. The host -# OS's invariant-TSC flags and clocksource are recorded as evidence only; -# Azure WHP hosts lack the flag while their guests measure tens of -# nanoseconds of skew. +# OS's invariant-TSC flags and clocksource are recorded as evidence only, +# because they don't decide what a guest observes: Azure WHP hosts show the +# CPUID bit but cannot offer invariant TSC to partitions, and their guests +# measure tens of nanoseconds of skew. LINUX_INVARIANT_TSC_FLAGS = ("constant_tsc", "nonstop_tsc") CLOCKSOURCE_PATH = Path( "/sys/devices/system/clocksource/clocksource0/current_clocksource" From c16627c1dbbfedcfc5574029abdcd27a63f88b32 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:43:23 -0700 Subject: [PATCH 031/188] guest: turn off bracketed paste in the Ubuntu guest shell With a complete Ubuntu root, bash's readline finds the linux terminfo entry and enables bracketed paste. It then prefixes the output of every accepted line with ESC[?2004l, so console readers that match a line's start, such as nvxmatrix's command sections, miss the first output line. Before 25bac37 the truncated root lacked terminfo, which hid this. Turn bracketed paste off in nvx-bashrc before the boot marker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/ubuntu/nvx-bashrc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/guest/ubuntu/nvx-bashrc b/guest/ubuntu/nvx-bashrc index a1ceaff9d..208b1baeb 100644 --- a/guest/ubuntu/nvx-bashrc +++ b/guest/ubuntu/nvx-bashrc @@ -1,2 +1,5 @@ PS1='nvx-ubuntu# ' +# Bracketed paste prefixes the output of every accepted line with an escape +# sequence, so a console reader that matches lines would miss the first one. +bind 'set enable-bracketed-paste off' printf '%s\n' 'NVX-GUEST-BOOT-OK: ubuntu' From a8bd21541a86b28d45ca2589ab0bffc227a88778 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 09:57:22 -0700 Subject: [PATCH 032/188] doc: settle the last time ABI TBDs from the MSHV spike and the WHP backend MSHV (spike and edae5f77d): - MSR-index intercepts (type 12) with no synthetic features; native synthetic MSRs are forbidden because the native TSC_FREQUENCY reports the destination's rate. CPUID overrides take 43 to 62 us. - Linux writes TSC_INVARIANT_CONTROL once, on the BSP. - Forced invariant TSC: an 8-vCPU cold boot takes 1,385 ms without the bit and 299 ms with it; the time bits override the fingerprint. - Restores read back equal on 20 of 20; partition time now resumes right after the read-back, so guest time runs from the restore anchor on every backend. - The warp probe reports its offset uncertainty, and CI fails an inconclusive verdict. - Profiles set ITS_NO or BHI_CTRL only where the generation is immune and every backend can present the value. WHP backend (cb7c769a3): the capture anchor and the measured epsilon close both TBD(whp) entries. Profiles: the legacy L2-cache MSR stubs raise #GP on every backend under the time ABI, closing TBD(profiles). Qualification follows the orchestrator's measured-property decision (ci 7740cff): host invariant TSC and clocksource are evidence only, and H6 runs the idle-inducing warp schedule. No TBD marker remains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 128 +++++++++++++++++++++++++---------------- 1 file changed, 77 insertions(+), 51 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 8c36a1a7c..8b8676ce9 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -8,8 +8,6 @@ MSHV, and WHP. It is authoritative for the `time-abi-v1` integration branches of `microsoft/nvx` and `nanvix/openvmm`. When implemented, it replaces the time rules in [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) and the clock tokens in [Cold boot](cold-boot.md#effective-command-line). -Entries marked `TBD()` wait for that agent's backend implementation or -measurements. Notation: @@ -128,9 +126,9 @@ on every backend: | Any other MSR in the range | | #GP | #GP | `HV_X64_MSR_TSC_INVARIANT_CONTROL` is one partition-wide value. It is guest -state: it is saved with the VM and restored with it. Linux writes 1 at boot -with an unchecked `wrmsrq`, so this write must never fault on any backend. -Clearing it after setting it is accepted. +state: it is saved with the VM and restored with it. Linux writes 1 once at +boot, on the BSP, with an unchecked `wrmsrq`, so this write must never fault +on any backend. Clearing it after setting it is accepted. The CPU time bits hide two architectural timer MSRs, and every backend makes them raise #GP: `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` @@ -159,12 +157,13 @@ independent of any host's rate. Linux trusts the TSC because of `AccessTscInvariantControls`, not because of CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag and -spares each AP a delay-loop calibration (about 150 ms per AP on Azure MSHV). -The bit is policy, not a fingerprint feature: profiles set it even where a -nested hypervisor hides it from host fingerprints (Azure's MSHV and WHP L1 -partitions), every backend exposes it (WHP through its CPUID override), and -host qualification backs it with measured invariance (see -[Host qualification](#host-qualification)). +spares each AP a delay-loop calibration of about 150 ms (on Azure MSHV, an +8-vCPU cold boot takes 1,385 ms without the bit and 299 ms with it). The time +bits override the fingerprint: the bit is policy, not a fingerprint feature, +so profiles set it even where a nested hypervisor hides it from host +fingerprints (Azure's MSHV and WHP L1 partitions), every backend exposes it +(WHP through its CPUID override), and host qualification backs it with +measured invariance (see [Host qualification](#host-qualification)). ### Rates @@ -249,22 +248,22 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | | Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), plus CPUID exits for leaves 1, 6, 7, `0xA`, `0x15`, `0x16`, and `0x80000007` | -| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are never enabled: they pre-empt the intercepts | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | +| Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | | TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; the hypervisor raises #GP for `IA32_TSC_ADJUST` | | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC fail qualification (`azure-azlinux-2`) | Set by the CPUID override: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`), so those hosts qualify on the warp probe and rate stability (the guest TSC matched the declared rate against host QPC within 0.001 ppm over 118 s) | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, re-sampled a bounded number of times | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | TBD(whp) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and thaw at the first VP run (56 to 312 µs for 1 to 8 VPs) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | +| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, re-sampled a bounded number of times | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | -| Unknown MSRs | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | #GP from the hypervisor | #GP; the `MYSTERY_MSRS` stubs are TBD(profiles) | +| Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [CPU profiles](#cpu-profiles)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | | Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | **KVM common offset.** Restoring each VP's `IA32_TSC` is unreliable on KVM. @@ -340,6 +339,14 @@ profiles' derivation policy). `IA32_UCODE_REV` is not pinned in v1. The effective guest CPUID is a pure function of the profile, the VM topology, and the time ABI's identity leaves. +Mitigation-relevant bits follow the hardware. A profile sets `ITS_NO` +(`IA32_ARCH_CAPABILITIES` bit 62) or `BHI_CTRL` (`CPUID.7.2:EDX[4]`) only if +every CPU of its generation is immune or has the control and every backend +can present the value; MSHV cannot intercept `IA32_ARCH_CAPABILITIES`. +Without `ITS_NO`, the hypervisor bit makes Linux enable its ITS mitigation. +NVX keeps every such mitigation enabled, and the kernel must not leave its +thunk pages writable and executable (boot check `K1`). + **Selection.** A microVM always has a profile. A cold boot uses `--cpu-profile `, or `auto` (the default), which selects the highest revision of the single profile whose generation covers the host's vendor, @@ -380,7 +387,11 @@ bare-metal hosts (one per backend) and fifteen Azure hosts: | `intel.icelake-sp.v1` | 6/106 | Xeon Platinum 8370C runners | KVM, MSHV, WHP | | `intel.emeraldrapids.v1` | 6/207 | Xeon Platinum 8573C runners | MSHV, WHP (no KVM host exists) | -The fate of the `MYSTERY_MSRS` stubs is TBD(profiles). +No profile pins the legacy P6 L2-cache MSRs (`0x88..=0x8a`, `0x116`, +`0x118..=0x11b`, and `0x11e`) that the KVM and WHP backends stub for Windows +booting from the PCAT BIOS. A microVM boots its kernel directly and never +probes them, so on a time ABI partition they raise #GP on every backend, as +MSHV's hypervisor already does. ## TSC rate policy and LAPIC rate rule @@ -442,15 +453,17 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this 1. **VMM synchronized set.** Restore writes one target to every instantiated VP at one host instant and verifies it by read-back before any VP runs (`E_TSC_SYNC_READBACK`). The VMM introduces no skew of its own. -2. **Host qualification.** `nvx.py doctor` and `validate-runner` run a host - TSC skew probe and a guest warp probe; a host above 1 µs is not qualified. -3. **CI warp probe.** Conformance, restore-matrix, and soak runs execute the - guest warp probe after boot and after every restore and fail above 1 µs. +2. **Host qualification.** `nvx.py doctor` runs a host TSC skew probe (`H5`) + and the guest warp probe (`H6`); a host above 1 µs is not qualified. +3. **CI warp probe.** Every CI microVM boot and restore scenario, and the + conformance, restore-matrix, and soak runs, execute the guest warp probe + after boot and after every restore and fail above 1 µs. 4. **Backend live skew.** Each backend shows that live skew stays within the bound after release. The spikes measured at most 63 ns of ping-pong - offset and 7.2 ns of backward step on KVM, at most 516 ns on MSHV - (dual-socket bare metal, bounded by the probe's round trip), and at most - 70 ns on WHP. + offset and 7.2 ns of backward step on KVM; on MSHV, zero warps and an + equal read-back on 20 of 20 restores, with offsets of at most 516 ns on + dual-socket bare metal (bounded by the probe's round trip) and 195 ns on + Azure; and at most 70 ns on WHP. The guest warp probe (`nvx-time-probe warp`, built from `guest/common/nvx-time-probe.c` and installed as `/sbin/nvx-time-probe`) @@ -464,7 +477,13 @@ pair: rounds, `t2 - (t1 + t3) / 2` at the round with the minimum round trip. Both must be at most 1,000 ns, and no pair may stall for more than 2 s. -Cycle values convert to nanoseconds with `F`. +Cycle values convert to nanoseconds with `F`. Each offset estimate is +uncertain by half its pair's minimum round trip, which the probe reports +(`max_uncertainty_ns`); a verdict counts only if every pair's uncertainty is +below the bound (`conclusive=1`), and CI fails an inconclusive one. Round +trips are about 0.25 µs within a socket and 1 µs across sockets, so 100 ms +per pair gives more than 10,000 rounds and an uncertainty of at most about +0.5 µs on dual-socket hosts. ## Downtime semantics and source selection @@ -500,12 +519,13 @@ path, a UTC delta that differs from `D` by more than 1 s is logged as a host wall-clock step and does not change `D`. The packet reports the source, so the guest and tests can tell the two paths apart. -The restore anchor is the instant of the synchronized TSC set. On KVM and -WHP the guest TSC runs from that instant (WHP resumes partition time right -after the read-back); on MSHV it is frozen until the first VP runs. Either -way, the guest monotonic advance across a restore lies in `[D, D + R]`, up +The restore anchor is the instant of the synchronized TSC set, and the guest +TSC runs from that instant on every backend: KVM never stops it, and MSHV and +WHP resume partition time right after the read-back. The guest monotonic +advance across a restore therefore lies in `[D, D + R]`, up to the two anchors' pairing errors (each at most 100 µs), where `R` is the -time from the restore anchor to the first restored instruction. Wall-clock +time from the restore anchor to the first restored instruction; it falls +short of `D + R` only by the frozen write (at most about 0.5 ms). Wall-clock repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. The KVM spike kept guest `CLOCK_MONOTONIC` within −0.09 to +2.8 ms of the host interval across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive @@ -588,8 +608,7 @@ In the worker, with every VP stopped: 16. Seal the time fields of the restore packet: `D`, its source, the rate deviation, `g`, and the test-hook flag. 17. Start the state units, with host input gated when the restore requires an - acknowledgement; publish the readiness event; release the VPs. On MSHV, - partition time thaws when the first VP runs. + acknowledgement; publish the readiness event; release the VPs. The guest then repairs its clocks (see [Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. @@ -759,9 +778,8 @@ theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 The same bracket applies to `utc_ns` in the restore packet, around the first `0xa5` write. A sample costs one write and four reads. OpenVMM cannot observe -a running VP's TSC on every backend, and frozen time on MSHV thaws at an -unobservable instant, so the guest forms the UTC and TSC pair itself through -this bracket; its `CLOCK_REALTIME` is derived from the TSC. +a running VP's TSC on every backend, so the guest forms the UTC and TSC pair +itself through this bracket; its `CLOCK_REALTIME` is derived from the TSC. ### Uncertainty bounds @@ -787,11 +805,16 @@ difference: | KVM | `azure-kvm-5` (nested, Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | | MSHV | prometheus30 (bare metal) | 6.3 to 6.4 µs / 6.8 to 7.1 µs | 37 µs | | MSHV | `azure-azlinux-5` (nested) | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | -| WHP | | TBD(whp) | | +| WHP | prometheus28 (bare metal) | 20.7 to 21.3 µs / 29.4 to 30.5 µs | 52 µs | +| WHP | `azure-windows-1` (8370C, nested) | 24.8 to 25.2 µs / 36.5 to 41.9 µs | 794 µs | +| WHP | `azure-windows-3` (8573C, nested) | 30.5 to 30.8 µs / 46.0 to 46.4 µs | 251 µs | The second run of each boot overlaps the console's drain of the first run's -output, and it has the higher KVM p99s. A sample above the 50 µs discipline -bound is rare, and the poll retries it. +output, and it has the higher KVM p99s. The WHP rows come from the WHP +agent's probe (two runs of 2,000 samples). A sample above the 50 µs +discipline bound is rare except on WHP, whose p99 approaches it; a poll then +retries, and a poll whose three attempts all fail is skipped without harm +(`G_SAMPLE_UNCERTAIN`). ### Generation counter and generation ID @@ -1132,27 +1155,30 @@ failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. ## Host qualification -`nvx.py doctor --backend ` qualifies a host interactively, and the -`validate-runner` action runs the same checks before every CI job. Both print -one `NVX-DOCTOR: check= status= detail=...` line per check and -fail if any check fails. The time checks replace the `nonstop_tsc` check. +`nvx.py doctor --backend ` qualifies a host interactively by running +every check. The `validate-runner` action runs `H1`, `H2`, and `H4` before +every CI job and `H3` in microVM jobs, and every CI microVM boot and restore +scenario runs the `H6` warp schedule; `H5` and `H7` run in `doctor` only. +Both tools print one `NVX-DOCTOR: check= status= detail=...` +line per check and fail if any check fails. The time checks replace the +`nonstop_tsc` check. | ID | Check | | --- | --- | | `H1` | The backend device or API is present and usable | -| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`) | +| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | -| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d`; on Linux the host clocksource is `tsc` (KVM rewrites per-vCPU TSC offsets on a host with an unstable TSC) | +| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d`. The Linux host clocksource is reported as evidence only: MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, and KVM rewrites per-vCPU TSC offsets only on a host that marked its TSC unstable without that clocksource, which `H6` measures | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | -| `H6` | Guest warp probe: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000 | +| `H6` | Guest warp probe on an idle-inducing schedule: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and runs the probe twice with every vCPU halted for 1 s between the runs (once with one vCPU); every run reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000, no stall, and a conclusive verdict | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | -Qualification gates on these measured properties and on the profile's -features. Hosts whose OS sees no invariant TSC (Azure WHP and nested MSHV -partitions) still expose the CPUID invariant-TSC bit through the profile, so -the warp probe (`H6`) and rate stability (`H4`) are the evidence for TSC -invariance there; a host that fails them, such as `azure-azlinux-2`, is not -qualified. +Qualification gates alike on every backend, on measured properties only: the +CPU profile (`H2` and `H3`), rate stability (`H4`), and the idle-scheduled +warp probe (`H6`). Hosts whose OS sees no invariant TSC (Azure WHP and nested +MSHV partitions) still expose the CPUID invariant-TSC bit through the +profile, so `H4` and `H6` are the evidence for TSC invariance there; a host +that fails them, such as `azure-azlinux-2`, is not qualified. Generation names used in logs, reports, and job summaries are `skylake-sp` (family 6, model 85), `icelake-sp` (6/106), and `emeraldrapids` (6/207). From 26a771066be8ddfb2b989650af5fb3a2dd9e2ee1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 10:15:27 -0700 Subject: [PATCH 033/188] doc: decide the time ABI profile brand and microcode, and fold the profiles spike Decisions the orchestrator left to the spec (profiles report, open questions 4 and 5): - The brand string is generic per generation, "Intel(R) Xeon(R) Processor ()" without a frequency, so every SKU of a generation presents the same informational brand and derivation needs no common brand across the source hosts. The v1 profiles are regenerated before any release. - IA32_UCODE_REV is not pinned: under a hypervisor Linux disables its microcode loader and skips its microcode checks, so the value only reaches /proc/cpuinfo and no restore check depends on it. From the profiles spike: every backend programs the whole profile explicitly and never relies on hypervisor defaults (WHP's defaults omit SPEC_CTRL, IBPB, STIBP, SSBD, and PSFD); sharing costs nothing on Ice Lake and Emerald Rapids and drops PKU, UMIP, FDP_EXCPTN_ONLY, FLUSH_L1D, and the AMD-alias speculation bits on Skylake; ARCH_CAPABILITIES is pinned under the bank-derived mask, presented through the banks on MSHV and WHP; the v1 profiles clear ITS_NO and BHI_CTRL; H2 names the --cpu-fingerprint tool and its openvmm-cpu-fingerprint/v1 output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 44 +++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 8b8676ce9..faf7a8cbf 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -300,6 +300,9 @@ revisions are never deleted, so old snapshots stay restorable. One surface serves all three backends: MSHV and WHP derive their processor feature banks from the profile's CPUID (`cpu_profile::hv_banks`) instead of from the host's, and each backend verifies at partition creation that it supports the profile. +Every backend programs the whole surface explicitly and never relies on +hypervisor defaults; WHP's default processor features, for example, omit +SPEC_CTRL, IBPB, STIBP, and SSBD on Skylake and PSFD on Ice Lake. A shared profile gives the same guest behavior on every backend; it does not make snapshots portable, because cross-backend restore is rejected (`E_BACKEND_MISMATCH`). @@ -321,7 +324,7 @@ document in pretty form. | `cpuid` | A dense table of every leaf and subleaf in `[0, max basic]` and `[0x80000000, max extended]` with a value and a mask per register; mask bit 1 pins the value, mask bit 0 marks a VMM-owned or runtime-owned bit | | `xcr0`, `xss`, `xsave_components` | The XSAVE features the guest may enable, and the size, offset, and flags of every enabled component | | `physical_address_width` | The guest physical address width | -| `msrs` | Pinned MSR values with masks; v1 pins `IA32_ARCH_CAPABILITIES` only | +| `msrs` | Pinned MSR values with masks; v1 pins `IA32_ARCH_CAPABILITIES` only, under the mask of the bits that Hyper-V's feature banks derive plus `ITS_NO`. KVM presents it as a feature MSR; MSHV and WHP present it through their banks' `*_NO` bits, because neither has a register to set or read it back | | `provenance` | The derivation method and the source fingerprints' backends, host counts, and surface digests (informative) | VMM-owned bits are a code table, not profile data: `CPUID.1:EBX[31:16]` @@ -335,17 +338,33 @@ pinned set and, with the hypervisor bit, exempt from host-support verification, because nested Azure hypervisors hide them from fingerprints; host qualification measures them instead. Profiles also pin policy zeros (VMX and SVM, SGX, PT, RDT, PCONFIG, and the other features listed by the -profiles' derivation policy). `IA32_UCODE_REV` is not pinned in v1. The -effective guest CPUID is a pure function of the profile, the VM topology, -and the time ABI's identity leaves. +profiles' derivation policy). The effective guest CPUID is a pure function of +the profile, the VM topology, and the time ABI's identity leaves. + +Informational fields: + +- The brand string (`0x80000002..=0x80000004`) is generic per generation: + `Intel(R) Xeon(R) Processor ()` with the generation's display name + (`Skylake-SP`, `Ice Lake-SP`, or `Emerald Rapids`), zero-padded, without a + frequency. Every host of a generation presents it whatever its SKU, and + derivation needs no common brand across the source hosts. +- The cache leaves (`0x2` and `0x4`, except their VMM-owned counts) come from + the reference hosts; other SKUs of the generation differ, and verification + does not compare them. Address widths are checked as limits. +- `IA32_UCODE_REV` is not pinned: Linux disables its microcode loader and + skips its microcode checks under a hypervisor, so the value only reaches + `/proc/cpuinfo`, and it is outside the effective-CPUID record and every + restore check. Mitigation-relevant bits follow the hardware. A profile sets `ITS_NO` (`IA32_ARCH_CAPABILITIES` bit 62) or `BHI_CTRL` (`CPUID.7.2:EDX[4]`) only if every CPU of its generation is immune or has the control and every backend -can present the value; MSHV cannot intercept `IA32_ARCH_CAPABILITIES`. -Without `ITS_NO`, the hypervisor bit makes Linux enable its ITS mitigation. -NVX keeps every such mitigation enabled, and the kernel must not leave its -thunk pages writable and executable (boot check `K1`). +can present the value; MSHV cannot intercept `IA32_ARCH_CAPABILITIES`. The +v1 profiles clear both: MSHV and WHP cannot present `ITS_NO`, and no host +available to the fleet exposes `BHI_CTRL`. Without `ITS_NO`, the hypervisor +bit makes Linux enable its ITS mitigation. NVX keeps every such mitigation +enabled, and the kernel must not leave its thunk pages writable and +executable (boot check `K1`). **Selection.** A microVM always has a profile. A cold boot uses `--cpu-profile `, or `auto` (the default), which selects the highest @@ -387,6 +406,13 @@ bare-metal hosts (one per backend) and fifteen Azure hosts: | `intel.icelake-sp.v1` | 6/106 | Xeon Platinum 8370C runners | KVM, MSHV, WHP | | `intel.emeraldrapids.v1` | 6/207 | Xeon Platinum 8573C runners | MSHV, WHP (no KVM host exists) | +Sharing costs nothing on Ice Lake-SP and Emerald Rapids, where every backend +offers the same features. On Skylake-SP, used only by the bare-metal +development hosts, the shared profile drops what one backend alone offers: +PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the +AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use +the `7.0` EDX equivalents, which every backend presents). + No profile pins the legacy P6 L2-cache MSRs (`0x88..=0x8a`, `0x116`, `0x118..=0x11b`, and `0x11e`) that the KVM and WHP backends stub for Windows booting from the PCAT BIOS. A microVM boots its kernel directly and never @@ -1166,7 +1192,7 @@ line per check and fail if any check fails. The time checks replace the | ID | Check | | --- | --- | | `H1` | The backend device or API is present and usable | -| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | +| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | | `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d`. The Linux host clocksource is reported as evidence only: MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, and KVM rewrites per-vCPU TSC offsets only on a host that marked its TSC unstable without that clocksource, which `H6` measures | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | From 8d92d2a021837475229d4a576dd5416a15ce23a2 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 10:23:29 -0700 Subject: [PATCH 034/188] doc: fix the time ABI anchor sample count and the hidden-MSR behavior From the kvm agent's requests, with the orchestrator's decisions: - Both anchors take at most 64 samples (KVM uses up to 16) before E_TSC_ANCHOR. - Restore step 14 sets every VP's LAPIC state always after the synchronized set (step 12), because KVM derives one-shot deadlines from the guest TSC when the LAPIC state is set. - Core never calls the legacy clock entry points on a time ABI partition; KVM rejects both there. - IA32_TSC_ADJUST and IA32_TSC_DEADLINE raise #GP on every backend, with each backend's mechanism: KVM's MSR filter; MSHV's hypervisor once the feature-bank bits are cleared (measured on prometheus30: reads and writes of both fail on every CPU, with no unchecked-MSR warning); WHP's exits to OpenVMM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index faf7a8cbf..4f63b0539 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -254,10 +254,10 @@ settled cell on the registered hosts. | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | -| TSC-deadline and `TSC_ADJUST` hidden | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve, and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; the hypervisor raises #GP for `IA32_TSC_ADJUST` | +| TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC fail qualification (`azure-azlinux-2`) | Set by the CPUID override: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`), so those hosts qualify on the warp probe and rate stability (the guest TSC matched the declared rate against host QPC within 0.001 ppm over 118 s) | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | -| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, re-sampled a bounded number of times | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | +| Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | @@ -620,13 +620,15 @@ In the worker, with every VP stopped: `TSC_target` to every instantiated VP at that instant. Per-VP TSC values from saved VP state are omitted from the VP restore and never applied. A restore anchor that pairs a TSC read with host time (KVM's common offset) - is re-sampled a bounded number of times and fails with `E_TSC_ANCHOR` if - no pair is within 100 µs; a pairing error shifts every VP's TSC alike, by - at most that amount. A frozen write needs no such pairing. + takes at most 64 samples and fails with `E_TSC_ANCHOR` if no pair is + within 100 µs; a pairing error shifts every VP's TSC alike, by at most + that amount. A frozen write needs no such pairing. Core never calls the + legacy clock entry points (`set_tsc_frequency_hz`, + `advance_snapshot_time`) on a time ABI partition; KVM rejects both there. 13. Read back: every instantiated VP holds the synchronized value as defined by its backend (`E_TSC_SYNC_READBACK`). 14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every - VP's LAPIC state again after the synchronized set, even when no timer is + VP's LAPIC state again, always after step 12 and even when no timer is armed: KVM derives its timer deadline from the guest TSC when the LAPIC state is set. 15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT @@ -654,7 +656,7 @@ state units are quiesced, capture: PIT channel 0 is not counting periodically (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`, `E_PIT_ACTIVE`); 2. takes the capture anchor: VP 0's TSC paired with a host time sample taken - within 100 µs of it, keeping the tightest of a bounded number of samples + within 100 µs of it, keeping the tightest of at most 64 samples (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); and 3. records the declared rates, the CPU profile, the effective CPUID, and the process's generation counter. @@ -1152,7 +1154,7 @@ code. | `E_HOST_IDENTITY` | Host identity, boot identity, or clocks unavailable | Capture, restore | | `E_DOWNTIME_NEGATIVE` | Downtime below zero | Restore | | `E_DOWNTIME_EXCESSIVE` | Downtime above 30 days | Restore | -| `E_TSC_ANCHOR` | An anchor is unavailable, or no sample pairs within 100 µs after bounded re-sampling | Capture, restore | +| `E_TSC_ANCHOR` | An anchor is unavailable, or none of at most 64 samples pairs within 100 µs | Capture, restore | | `E_TSC_TARGET_OVERFLOW` | `TSC_target` exceeds 64 bits | Restore | | `E_TSC_SYNC_READBACK` | A VP does not hold the synchronized value | Restore | | `E_VP_LATE_CREATION` | A VP was instantiated after the synchronized set | Restore | From 5d8ee7c39556164beba085ccb6ffda1cffe85f07 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 10:25:02 -0700 Subject: [PATCH 035/188] guest: test the exhaustive check's report format The CI conformance scenario parses `nvx-time exhaustive` output, so test the per-check line, its detail escaping, and the summary line against the spec. The summary moves into one helper that both exit paths use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-time.c | 34 +++++++++++++++++++++++++++++----- scripts/test_guest_time.py | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 5 deletions(-) diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 53db1d49f..41fa0d23b 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -1934,6 +1934,13 @@ static void exhaustive_report(struct exhaustive *state, const char *id, int cpu, state->failures++; } +static int exhaustive_summary(const struct exhaustive *state, int online) +{ + printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 status=%s cpus=%d failures=%d\n", + state->failures == 0 ? "ok" : "fail", online, state->failures); + return state->failures == 0 ? 0 : 1; +} + // X1 (every leaf 0x40000006..=0x400000ff) when STEP is 1, or X2 (every base // 0x40000100..=0x4000ff00) when STEP is 0x100, on the current CPU. static void exhaustive_leaves(struct exhaustive *state, const char *id, int cpu, @@ -2110,8 +2117,8 @@ static int exhaustive_run(void) memset(&state, 0, sizeof(state)); if (read_text("/sys/devices/system/cpu/online", text, sizeof(text)) < 0 || (online = parse_cpu_list(text, cpus)) <= 0) { - printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 status=fail cpus=0 failures=1\n"); - return 1; + state.failures = 1; + return exhaustive_summary(&state, 0); } cpuinfo_init(&parse, &state.cpuinfo, cpus, false); (void)for_each_file_line("/proc/cpuinfo", cpuinfo_line, &parse); @@ -2155,9 +2162,7 @@ static int exhaustive_run(void) } if (CPU_COUNT(&original) > 0) (void)sched_setaffinity(0, sizeof(original), &original); - printf("NVX-TIME-ABI-EXHAUSTIVE: v=1 status=%s cpus=%d failures=%d\n", - state.failures == 0 ? "ok" : "fail", online, state.failures); - return state.failures == 0 ? 0 : 1; + return exhaustive_summary(&state, online); } // C6: `tsc` is current, and only tsc, refined-jiffies, or jiffies is @@ -3826,6 +3831,25 @@ static int cmd_test(int argc, char **argv) printf("fail %s\n", detail); return 0; } + if (strcmp(name, "exhaustive-report") == 0 && argc == 4) { + struct exhaustive state; + + memset(&state, 0, sizeof(state)); + exhaustive_report(&state, argv[0], atoi(argv[1]), + strcmp(argv[2], "pass") == 0, argv[3]); + printf("failures=%d\n", state.failures); + return 0; + } + if (strcmp(name, "exhaustive-summary") == 0 && argc == 2) { + struct exhaustive state; + int status; + + memset(&state, 0, sizeof(state)); + state.failures = atoi(argv[1]); + status = exhaustive_summary(&state, atoi(argv[0])); + printf("exit=%d\n", status); + return 0; + } if (strcmp(name, "boot-log") == 0) return test_lines(argc, argv, LINES_BOOT_LOG); if (strcmp(name, "syslog") == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 6d103af75..6edd6e657 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -7,6 +7,7 @@ from __future__ import annotations +import re import shutil import struct import subprocess @@ -545,6 +546,42 @@ def leaf(number: str, value: tuple[str, ...]) -> str: self.assertIn('(signature "KVMKVMKVM...")', signature) self.assertTrue(signature.endswith("or the highest basic leaf"), signature) + def test_exhaustive_report_lines_follow_the_spec(self): + # The spec's exhaustive check prints one line per check and CPU, with + # an escaped detail, then a summary. + line = re.compile( + r"^NVX-TIME-ABI-EXHAUSTIVE: v=1 check=X[1-6] cpu=\d+ " + r'status=(pass|fail) detail="(?:[^"\\]|\\.)*"$' + ) + passed = self.run_test( + "exhaustive-report", "X3", "2", "pass", "ok" + ).splitlines() + self.assertEqual( + passed, + [ + 'NVX-TIME-ABI-EXHAUSTIVE: v=1 check=X3 cpu=2 status=pass detail="ok"', + "failures=0", + ], + ) + failed = self.run_test( + "exhaustive-report", + "X2", + "7", + "fail", + 'base 0x40000100 "KVMKVMKVM" \\ x\ty', + ).splitlines() + self.assertRegex(failed[0], line) + self.assertIn(r'detail="base 0x40000100 \"KVMKVMKVM\" \\ x\x09y"', failed[0]) + self.assertEqual(failed[1], "failures=1") + self.assertEqual( + self.run_test("exhaustive-summary", "4", "0").splitlines(), + ["NVX-TIME-ABI-EXHAUSTIVE: v=1 status=ok cpus=4 failures=0", "exit=0"], + ) + self.assertEqual( + self.run_test("exhaustive-summary", "8", "3").splitlines(), + ["NVX-TIME-ABI-EXHAUSTIVE: v=1 status=fail cpus=8 failures=3", "exit=1"], + ) + def timer_list( self, sections: list[tuple[int, str, int, str, int]], From 6d953b652b28ecafb5ad9cb0e1de3a170073f4fa Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 10:48:47 -0700 Subject: [PATCH 036/188] doc: time ABI ordering, effective CPUID set, and anchor bound rationale Fold in the MSHV agent's requests for core: - Explain the shared 100 us anchor pairing bound: nested hypercall round trips on Azure take about 12 us, so nested MSHV pairs at a median of 6.2 us and WHP on 8573C runners at up to 11 us, and downtime needs only millisecond accuracy. - Restore step 8: preflight runs before any VP runs or any VP state is restored (MSHV writes TimeFreeze and reads VP 0's reset-state CPUID). Step 11: every instantiated VP is bound, which creates it on MSHV, before the synchronized set. - Add the cold-boot sequence: profile selection and clock-token rejection, preflight before any VP runs, declared rates F = F_d and L = L_d, g = 0, and no TSC write. - Define the governed leaf set of the effective CPUID, computed by OpenVMM and compared with each backend's report of VP 0's CPUID under its masks (E_CPU_SURFACE, now also at cold boot). - The fatal-error line leads with the first code in the error chain. - MSHV cannot present BHI_CTRL: no bhi_dis or bhi_no in its banks, and leaf 7 reports maximum subleaf 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 92 +++++++++++++++++++++++++++++++++--------- 1 file changed, 72 insertions(+), 20 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 4f63b0539..274f0a154 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -261,7 +261,7 @@ settled cell on the registered hosts. | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | -| VP instantiation | All `C` VPs exist before the set | The created prefix exists before the set; no VP is created after it | All `C` VPs exist before the set | +| VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | | Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [CPU profiles](#cpu-profiles)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | | Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | @@ -341,6 +341,21 @@ host qualification measures them instead. Profiles also pin policy zeros profiles' derivation policy). The effective guest CPUID is a pure function of the profile, the VM topology, and the time ABI's identity leaves. +**Effective CPUID.** The effective CPUID (`openvmm-effective-cpuid/v1`) +lists exactly the governed leaves: every leaf and subleaf of the profile's +`cpuid` table; the topology leaves `0xB` and `0x1F`, at the subleaves the +VM's topology defines, when they are within the maximum basic leaf; and the +identity leaves `0x40000000..=0x40000005` with the explicit zero leaves. Each +entry holds the four registers and their masks; the per-VP fields hold VP 0's +APIC identity, and the runtime-owned bits are unmasked. OpenVMM computes it, +records it, and recomputes it on restore, so the record never depends on how +a backend enumerates or caches CPUID, and one profile and topology give the +same record on every backend. At preflight, before any VP runs, each backend +reports VP 0's CPUID for exactly the governed leaves (KVM from the +`KVM_SET_CPUID2` table it programmed, MSHV with `get_cpuid_values` on VP 0, +WHP from VP 0's register view; an entry without a subleaf at subleaf 0), and +OpenVMM compares the report with the effective CPUID under its masks. + Informational fields: - The brand string (`0x80000002..=0x80000004`) is generic per generation: @@ -361,10 +376,12 @@ Mitigation-relevant bits follow the hardware. A profile sets `ITS_NO` every CPU of its generation is immune or has the control and every backend can present the value; MSHV cannot intercept `IA32_ARCH_CAPABILITIES`. The v1 profiles clear both: MSHV and WHP cannot present `ITS_NO`, and no host -available to the fleet exposes `BHI_CTRL`. Without `ITS_NO`, the hypervisor -bit makes Linux enable its ITS mitigation. NVX keeps every such mitigation -enabled, and the kernel must not leave its thunk pages writable and -executable (boot check `K1`). +available to the fleet exposes `BHI_CTRL`. MSHV cannot present `BHI_CTRL` at +all: even on 8573C runners its feature banks have neither `bhi_dis` nor +`bhi_no`, and leaf 7 reports a maximum subleaf of 0. Without `ITS_NO`, the +hypervisor bit makes Linux enable its ITS mitigation. NVX keeps every such +mitigation enabled, and the kernel must not leave its thunk pages writable +and executable (boot check `K1`). **Selection.** A microVM always has a profile. A cold boot uses `--cpu-profile `, or `auto` (the default), which selects the highest @@ -396,6 +413,8 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the recorded one (`E_CPU_SURFACE`). The effective-CPUID record replaces the exact-equality CPU contract. +5. The backend presents the effective CPUID: VP 0's CPUID for every governed + leaf equals it under its masks (`E_CPU_SURFACE`). The catalog has three profiles, derived from the fingerprints of three bare-metal hosts (one per backend) and fifteen Azure hosts: @@ -558,6 +577,17 @@ across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive part came from a host sample taken before the per-VP TSC save, which the paired capture anchor removes. +**Anchor pairing bound.** An anchor that pairs a TSC read with a host time +sample (the capture anchor on every backend, and KVM's restore anchor) +brackets one with the other, and its pairing error is half the bracket. Every +backend uses one bound, 100 µs (`MAX_ANCHOR_PAIRING_NS` in +`virt::time_abi`): it keeps the tightest of at most 64 samples, never accepts +a looser pair, and otherwise fails with `E_TSC_ANCHOR`. A nested hypercall +round trip on Azure takes about 12 µs, so nested MSHV pairs at a median of +6.2 µs and WHP on 8573C runners at up to 11 µs; a 10 µs bound would leave +about 1.5× margin over the median and fail restores at random. Downtime needs +only millisecond accuracy, so a 100 µs pairing error is negligible. + ## Restore algorithm Restore runs these steps in order. Any failure rejects the restore before a @@ -590,11 +620,13 @@ Before worker construction: In the worker, with every VP stopped: 8. Create the partition with the profile CPUID, the identity leaves, and the - time platform declaring `F = F_s` and `L = L_s`. Preflight the backend: - profile support and effective CPUID (`E_PROFILE_UNSUPPORTED`, - `E_CPU_SURFACE`), identity routing (`E_IDENTITY_ROUTING`), the - synchronized-set primitive (`E_TSC_SYNC_UNSUPPORTED`), and no scaling - (`E_TSC_SCALING_ACTIVE`). + time platform declaring `F = F_s` and `L = L_s`. Preflight the backend + before any VP runs and before any VP state is restored: profile support + and effective CPUID (`E_PROFILE_UNSUPPORTED`, `E_CPU_SURFACE`), identity + routing (`E_IDENTITY_ROUTING`), the synchronized-set primitive + (`E_TSC_SYNC_UNSUPPORTED`), and no scaling (`E_TSC_SCALING_ACTIVE`). + MSHV's preflight writes `TimeFreeze` to probe the primitive and reads + VP 0's CPUID at its reset state. 9. Read `F_d` and `L_d` and apply the [rate policy](#tsc-rate-policy-and-lapic-rate-rule) (`E_TSC_RATE_UNAVAILABLE`, `E_TSC_RATE_IMPLAUSIBLE`, @@ -605,9 +637,10 @@ In the worker, with every VP stopped: partition, and every VP. 11. Assert the saved timers: no armed periodic or TSC-deadline LAPIC timer on any VP (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`), and PIT channel 0 not - counting in a periodic mode (`E_PIT_ACTIVE`). Freeze the instantiated VP - set; creating a VP after this point is an internal error - (`E_VP_LATE_CREATION`). + counting in a periodic mode (`E_PIT_ACTIVE`). Every instantiated VP was + bound before step 10 restored its state, and MSHV creates a VP only when + it is bound, so the instantiated set is complete. Freeze it; creating a + VP after this point is an internal error (`E_VP_LATE_CREATION`). 12. Synchronized TSC set. The backend takes the restore anchor (one host instant) and passes its host sample to the orchestrator, which selects the downtime source, computes `D` and checks its bounds, and returns @@ -641,6 +674,23 @@ In the worker, with every VP stopped: The guest then repairs its clocks (see [Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. +**Cold boot.** A cold boot runs the partition steps without a snapshot, in +this order: + +1. Select the CPU profile (see [CPU profiles](#cpu-profiles)) and reject + `tsc_early_khz=` and `lapic_timer_hz=` in the command line + (`E_CMDLINE_CLOCK_TOKEN`); OpenVMM injects neither. +2. Create the partition and preflight the backend as in step 8, before any + VP runs. +3. Read `F_d` and `L_d`, check them (`E_TSC_RATE_UNAVAILABLE`, + `E_TSC_RATE_IMPLAUSIBLE`, `E_LAPIC_RATE_UNAVAILABLE`, + `E_LAPIC_RATE_MISMATCH`), and declare `F = F_d` and `L = L_d`. +4. Set `g = 0`, start the state units, and release the VPs. + +A cold boot writes no TSC: every backend starts the VPs' TSCs together (the +spikes measured at most 63 ns of skew), and skew enforcement points 2 to 4 +apply. + ## Snapshot manifest New snapshots use manifest version 6 with format magic @@ -697,7 +747,7 @@ reused. It adds two required fields: | 1 | `id` | `string` | Profile ID | | 2 | `sha256` | `bytes` | Profile digest, 32 bytes | | 3 | `profile` | `bytes` | Canonical profile encoding, at most 1 MiB | -| 4 | `effective_cpuid` | `bytes` | The effective guest CPUID in its canonical encoding, `openvmm-effective-cpuid/v1` (compact canonical JSON): the profile's pinned values, the topology fields, and the identity leaves | +| 4 | `effective_cpuid` | `bytes` | The [effective CPUID](#cpu-profiles) in its canonical encoding, `openvmm-effective-cpuid/v1` (compact canonical JSON): every governed leaf with its values and masks | | 5 | `effective_cpuid_sha256` | `bytes` | Digest of `effective_cpuid`, 32 bytes | | 6 | `capture_cpu_signature` | `u32` | CPUID.1:EAX of the capture host, for diagnostics | @@ -1125,11 +1175,13 @@ Workloads that need prompt notice of a step can also arm a `timerfd` with Codes are stable. OpenVMM errors put the code in brackets at the start of the message, for example `[E_TSC_RATE_TOLERANCE] destination TSC rate ...`, and -the code survives error wrapping across the worker boundary. A rejected cold -boot or restore exits the OpenVMM process with status 1, the existing -fatal-error status; tests and orchestrators match the bracketed code. A -rejected capture is rollback-safe: the guest continues, and OpenVMM logs the -code. +the code survives every error-wrapping layer, including the worker boundary: +the process's fatal-error line leads with the first code in the chain, +`fatal error: [E_TSC_RATE_TOLERANCE] `, followed by the +full cause chain. A rejected cold boot or restore exits the OpenVMM process +with status 1, the existing fatal-error status; tests and orchestrators match +the bracketed code at the start of that line. A rejected capture is +rollback-safe: the guest continues, and OpenVMM logs the code. | VMM code | Condition | Detected at | | --- | --- | --- | @@ -1142,7 +1194,7 @@ code. | `E_PROFILE_TIME_BITS` | The profile is invalid or violates the CPU time bits | Cold boot, restore | | `E_CPU_GENERATION` | Host CPU vendor, family, model, or stepping not in the profile | Cold boot, restore | | `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, MSR value, or feature-bank bit of the profile, or the host is not qualified | Cold boot, restore | -| `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one | Restore | +| `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one, or VP 0's CPUID differs from the effective CPUID under its masks | Cold boot, restore | | `E_IDENTITY_ROUTING` | Backend cannot deliver the identity CPUID or MSRs | Cold boot, restore | | `E_TSC_SYNC_UNSUPPORTED` | Backend lacks the synchronized TSC set | Cold boot, restore | | `E_TSC_SCALING_ACTIVE` | The guest TSC would be scaled | Cold boot, restore | From dd637ee0c8361da5b0f0ca2b772930508c670817 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 11:25:50 -0700 Subject: [PATCH 037/188] doc: clarify the time ABI effective CPUID masks Say that the runtime-owned bits have mask 0, rather than "unmasked", and that a backend reads an entry without a subleaf at subleaf 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 274f0a154..d18d4e92a 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -347,14 +347,15 @@ lists exactly the governed leaves: every leaf and subleaf of the profile's VM's topology defines, when they are within the maximum basic leaf; and the identity leaves `0x40000000..=0x40000005` with the explicit zero leaves. Each entry holds the four registers and their masks; the per-VP fields hold VP 0's -APIC identity, and the runtime-owned bits are unmasked. OpenVMM computes it, +APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, records it, and recomputes it on restore, so the record never depends on how a backend enumerates or caches CPUID, and one profile and topology give the same record on every backend. At preflight, before any VP runs, each backend reports VP 0's CPUID for exactly the governed leaves (KVM from the `KVM_SET_CPUID2` table it programmed, MSHV with `get_cpuid_values` on VP 0, -WHP from VP 0's register view; an entry without a subleaf at subleaf 0), and -OpenVMM compares the report with the effective CPUID under its masks. +WHP from VP 0's register view), reading an entry without a subleaf at +subleaf 0, and OpenVMM compares the report with the effective CPUID under its +masks. Informational fields: From 4f8bc084cc561c232b0cd2950bd3b579032c41f3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 12:13:53 -0700 Subject: [PATCH 038/188] doc: finalize the time ABI host qualification from the WHP results Apply the orchestrator's qualification decision and fold in the WHP spike and backend results: - Rate stability (H4): a host probe against CLOCK_MONOTONIC or QueryPerformanceCounter, with sleep-separated samples (3 at 1 s in validate-runner, 13 at 10 s in doctor). It must be conclusive at 0.25 ppm per interval, interval rates agree within 1 ppm, and |r - F_d| is within 100 ppm, with the rationale for each bound. - Warp schedules: doctor's H6 runs five probes with idle gaps of 0.1, 1, 5, and 1 s; CI runs two with a 1 s gap after boot and every restore, so idle-triggered TSC corrections (#265) are exercised. - Qualification gates on measured properties only; the host invariant-TSC bit and clocksource are evidence. azure-azlinux-2 is unqualified because our account cannot run guests there, not because of the rule. - WHP: pinned leaves through CpuidResultList2 and CPUID exits only for the identity and topology leaves (about 25 us per exit-list entry), the frozen write's rationale, the Azure invariant-TSC evidence, and the hv1 capability rule. - Azure withholds the SPEC_CTRL family and MD_CLEAR from every backend; explicit banks enable them on bare-metal Skylake-SP. - Performance: the gate is measured on the CI Azure platforms; add the measured WHP restore, cold-start, timestamp-read, and fixed-cost effects. - Add the #292 conformance case, scope the removals to microVM paths (leaf 0x15 synthesis stays for standard-machine VMs), and require backends to derive their supported surface without a probe partition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 151 +++++++++++++++++++++++++++++++---------- 1 file changed, 114 insertions(+), 37 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index d18d4e92a..0a296363f 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -249,20 +249,20 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), plus CPUID exits for leaves 1, 6, 7, `0xA`, `0x15`, `0x16`, and `0x80000007` | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), and the profile's pinned leaves as `CpuidResultList2` results (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the per-VP topology leaves (`1`, `4`, `0xB`, and `0x1F` on Intel): each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | | TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | -| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Hosts without an invariant TSC fail qualification (`azure-azlinux-2`) | Set by the CPUID override: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`), so those hosts qualify on the warp probe and rate stability (the guest TSC matched the declared rate against host QPC within 0.001 ppm over 118 s) | +| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`). There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | | Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime`; `TscVirtualOffset` is unusable (writes fail) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime` before any VP runs. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | | Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | -| Partition capabilities | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | +| Partition capabilities, so that the identity leaves never enable the Hyper-V emulator or its saved-state elements | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | | Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [CPU profiles](#cpu-profiles)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | | Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | @@ -302,7 +302,12 @@ from the profile's CPUID (`cpu_profile::hv_banks`) instead of from the host's, and each backend verifies at partition creation that it supports the profile. Every backend programs the whole surface explicitly and never relies on hypervisor defaults; WHP's default processor features, for example, omit -SPEC_CTRL, IBPB, STIBP, and SSBD on Skylake and PSFD on Ice Lake. +SPEC_CTRL, IBPB, STIBP, and SSBD on Skylake and PSFD on Ice Lake. With +explicit banks, Linux in a bare-metal Skylake-SP WHP guest enables IBRS, +IBPB, SSBD, and CPU buffer clearing. On Azure, the L0 hypervisor withholds +the SPEC_CTRL family and MD_CLEAR from every backend on both generations +(each reports `CPUID.7.0:EDX` = `0x20000010`), and the Ice Lake-SP and +Emerald Rapids profiles pin that value. A shared profile gives the same guest behavior on every backend; it does not make snapshots portable, because cross-backend restore is rejected (`E_BACKEND_MISMATCH`). @@ -408,7 +413,10 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: widths) is within the backend's, every enabled XSAVE component has the same size, offset, and flags, XCR0 and XSS are subsets, and every pinned MSR value can be presented (an `ARCH_CAPABILITIES` immunity the host lacks - is a violation). The time policy bits are exempt. + is a violation). The time policy bits are exempt. Each backend derives + its supported surface from capability queries and the host's CPUID, + without a probe partition, which would add 2.6 to 5.5 ms per cold boot + and restore on WHP; the full fingerprint stays in `--cpu-fingerprint`. 4. On restore only: this OpenVMM pins a profile with the same ID and digest, and the recorded document hashes to it (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the @@ -500,9 +508,11 @@ at most 1 µs, that is `F_s / 1,000,000` cycles. Linux no longer checks this VP at one host instant and verifies it by read-back before any VP runs (`E_TSC_SYNC_READBACK`). The VMM introduces no skew of its own. 2. **Host qualification.** `nvx.py doctor` runs a host TSC skew probe (`H5`) - and the guest warp probe (`H6`); a host above 1 µs is not qualified. + and the guest warp probe on an idle-inducing schedule (`H6`); a host above + 1 µs is not qualified. 3. **CI warp probe.** Every CI microVM boot and restore scenario, and the conformance, restore-matrix, and soak runs, execute the guest warp probe + on the CI schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) after boot and after every restore and fail above 1 µs. 4. **Backend live skew.** Each backend shows that live skew stays within the bound after release. The spikes measured at most 63 ns of ping-pong @@ -1236,57 +1246,117 @@ failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. ## Host qualification -`nvx.py doctor --backend ` qualifies a host interactively by running -every check. The `validate-runner` action runs `H1`, `H2`, and `H4` before -every CI job and `H3` in microVM jobs, and every CI microVM boot and restore -scenario runs the `H6` warp schedule; `H5` and `H7` run in `doctor` only. -Both tools print one `NVX-DOCTOR: check= status= detail=...` -line per check and fail if any check fails. The time checks replace the -`nonstop_tsc` check. +`nvx.py doctor --backend ` qualifies a host by running every check, +with `H4` and `H6` on their long schedules. The `validate-runner` action runs +`H1`, `H2`, and the short `H4` before every CI job and `H3` in microVM jobs, +and every CI microVM boot and restore scenario runs the CI warp schedule; +`H5` and `H7` run in `doctor` only. Both tools print one +`NVX-DOCTOR: check= status= detail=...` line per check and +fail if any check fails. The time checks replace the `nonstop_tsc` check. | ID | Check | | --- | --- | | `H1` | The backend device or API is present and usable | | `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | -| `H4` | Host TSC rate stability: two 1 s measurements of the TSC against host monotonic time agree within 1 ppm and are within 100 ppm of `F_d`. The Linux host clocksource is reported as evidence only: MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, and KVM rewrites per-vCPU TSC offsets only on a host that marked its TSC unstable without that clocksource, which `H6` measures | +| `H4` | TSC rate stability against the host's monotonic clock (see [Rate stability](#rate-stability-h4)) | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | -| `H6` | Guest warp probe on an idle-inducing schedule: a microVM with the host's largest supported vCPU count up to 8 boots, passes the boot check, and runs the probe twice with every vCPU halted for 1 s between the runs (once with one vCPU); every run reports `max_backward_ns` and `max_abs_offset_ns` at most 1,000, no stall, and a conclusive verdict | +| `H6` | Guest warp probe on the qualification schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | +### Rate stability (`H4`) + +A host probe compares the TSC with the host's monotonic clock: +`CLOCK_MONOTONIC` on Linux, which is never stepped and runs at the rate time +synchronization disciplines, and `QueryPerformanceCounter` on Windows. A +sample reads the TSC between two reads of that clock, keeping the tightest +of up to 64 such brackets; its uncertainty is half the bracket plus half the +clock's resolution. The probe sleeps between samples, so the host's CPUs can +idle: 3 samples 1 s apart in `validate-runner`, and 13 samples 10 s apart +(120 s) in `doctor`. For each interval between consecutive samples it +computes the rate `r_i`, in TSC cycles per second of the clock, and its +uncertainty `u_i`, the two samples' uncertainties divided by the interval; +`r` is the rate over the whole window. The check passes if: + +- every `u_i` is at most 0.25 ppm of `r`; otherwise the measurement is + inconclusive and fails; +- the interval rates agree within 1 ppm of `r` (largest minus smallest); and +- where `F_d` is known (in `doctor`, and in the `validate-runner` jobs that + run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. + +The detail reports `r`, its deviation from `F_d` in ppm, the agreement, the +largest `u_i`, and the clock. An invariant TSC agrees with the host clock far +inside 1 ppm: 0.001 to 0.089 ppm over 1 s windows on the eleven CI runners, +and 0.000 ppm with a residual of at most 0.07 µs over 118 s on the WHP hosts. +A TSC that stops, slows, or is rescaled across idle misses the bound by +orders of magnitude. The 100 ppm bound keeps the guest's wall-clock +discipline, which steers at most 500 ppm, clear of the TSC's error plus a +restore's rate deviation of up to 250 ppm. On Windows the clock derives from +the same TSC, so the deviation compares `F_d` with the host's own +calibration. + +### Warp schedules (`H6` and CI) + +Each run of the guest warp probe executes +`/sbin/nvx-time-probe warp --bound-ns 1000` over every online vCPU and must +report `max_backward_ns` and `max_abs_offset_ns` at most 1,000, no stall, and +a conclusive verdict. Between runs the guest only sleeps, so every vCPU halts +and the host applies any idle-triggered TSC correction it makes, the source +of the warps in #265: + +- `H6`: a microVM with the host's largest supported vCPU count up to 8 boots, + passes the boot check, and runs the probe five times, with idle gaps of + 0.1, 1, 5, and 1 s; a 1-vCPU microVM then boots and runs it once. +- CI: every microVM boot and restore scenario runs the probe twice, with a + 1 s idle gap, after boot and after every restore. + +### Qualification gates + Qualification gates alike on every backend, on measured properties only: the -CPU profile (`H2` and `H3`), rate stability (`H4`), and the idle-scheduled -warp probe (`H6`). Hosts whose OS sees no invariant TSC (Azure WHP and nested -MSHV partitions) still expose the CPUID invariant-TSC bit through the -profile, so `H4` and `H6` are the evidence for TSC invariance there; a host -that fails them, such as `azure-azlinux-2`, is not qualified. +CPU profile (`H2`, `H3`), rate stability (`H4`), and the idle-scheduled warp +probe (`H6`, and the CI schedule in every microVM job). The host OS's +invariant-TSC bit and the Linux host clocksource are recorded as evidence +and never gate. Azure's WHP and nested MSHV hosts hide the bit from their +guests, and the profile exposes it anyway, so `H4` and `H6` measure what the +bit promises. MSHV roots and Azure KVM hosts run +`hyperv_clocksource_tsc_page`, and KVM rewrites per-vCPU TSC offsets only on +a host that marked its TSC unstable without that clocksource, which `H6` +measures. `azure-azlinux-2` (8370C), whose host OS lacks the bit and which +showed the #265 warps, is out of rotation and unqualified because our +account cannot run guests there, not because of this rule; its host-level +warp probe saw backward steps of at most 2.1 ns. + +### Generations and runner placement Generation names used in logs, reports, and job summaries are `skylake-sp` (family 6, model 85), `icelake-sp` (6/106), and `emeraldrapids` (6/207). `validate-runner` and `nvx.py doctor` detect the generation at run time and -report it, together with the selected profile, `F_d`, `L`, and the skew -metrics, in their log and the job summary; an unknown generation fails -qualification explicitly. Runner labels are not used and runners are not -re-registered: per-PR CI captures and restores on the same runner, and -same-generation cross-VM restore is validated by the fleet restore matrix -(`p6-restore-matrix`) on the hosts our account can use. Routing CI jobs by -generation labels is optional future work. +report it, together with the selected profile, `F_d`, `L`, the `H4` rate +metrics, and the skew metrics, in their log and the job summary; an unknown +generation fails qualification explicitly. Runner labels are not used and +runners are not re-registered: per-PR CI captures and restores on the same +runner, and same-generation cross-VM restore is validated by the fleet +restore matrix (`p6-restore-matrix`) on the hosts our account can use. +Routing CI jobs by generation labels is optional future work. ## Performance expectations and acceptance gate **Gate.** Per backend and vCPU count, no p50 is worse than the base-branch median by more than `max(5%, 2 ms)`. The gate covers all 36 one-vCPU metrics -and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. +and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. It is measured on +the CI Azure platforms; bare-metal results are informational. Expected effects: | Change | Expected effect | | --- | --- | -| WHP restored SMP without RDTSC emulation | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50) and the 45 to 70 µs cost of every guest timestamp read after an SMP restore | +| WHP restored SMP without RDTSC emulation | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50): SMP restores measured 8 to 18 ms faster in the spike and 14 to 31 ms faster on 8370C runners at 2 to 8 vCPUs. A guest timestamp read after an SMP restore drops from 41 to 73 µs to 10 to 16 ns | | Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits, most visible on WHP | | Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | | No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | -| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP; KVM unchanged | +| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | +| Invariant TSC from the profile where the host hides it (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | +| Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Added cold-boot cost, reported as the first run's `elapsed_us`; budget 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP; the gate is authoritative | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | @@ -1322,7 +1392,10 @@ all 18 registered hosts, plus the exhaustive CI check and the warp probe on every backend. The fleet runs them on the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on -the other KVM and MSHV runners, which only CI jobs exercise. +the other KVM and MSHV runners, which only CI jobs exercise. On the Azure WHP +runners, a cold boot with an empty command line, which logs to the console, +keeps `tsc` as its clocksource (#292: 6 of 6 runs with the time ABI, against +0 of 6 before it, which fell back from `tsc-early` to `refined-jiffies`). **Restore matrix.** Every case runs with zero violations; rejected cases must fail with the listed code. Per-PR CI runs the same-host cases on one runner; @@ -1340,7 +1413,7 @@ can use as the fleet restore matrix. | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | | Across VMs of one generation | `azure-windows-1` to `-2`; `azure-windows-3` to `-4`. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | `azure-windows-1` (8370C) to `-3` (8573C); prometheus32 to `azure-kvm-5` (KVM); prometheus30 to `azure-azlinux-5` (MSHV) | `E_CPU_GENERATION` | -| Host without invariant TSC at the host OS | `azure-azlinux-2` (out of CI rotation) | Qualified only if `H4` and `H6` pass; otherwise `nvx.py doctor` fails | +| Backend that hides invariant TSC from its guests | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | @@ -1375,7 +1448,8 @@ packet and in every time sample. ## Removed mechanisms and migration -Removed from OpenVMM: +Removed from OpenVMM's microVM paths; standard-machine VMs keep their +behavior: - Cold-boot clock tokens: `prepare_cold_boot_command_line` no longer injects `tsc_early_khz=` or `lapic_timer_hz=`, and their propagation, parsing, and @@ -1383,7 +1457,9 @@ Removed from OpenVMM: - The exact-equality CPU contract (`CpuCompatibilityContract`, including its `tsc_deadline` and `kvm_clock` fields) and host-derived CPUID, replaced by CPU profiles. -- Leaf `0x15` synthesis (`tsc_frequency_cpuid_leaves`) on every backend. +- Leaf `0x15` synthesis on every backend. `tsc_frequency_cpuid_leaves` and + WHP's `add_frequency_leaves` stay for standard-machine VMs, such as WHP VMs + with `--hv`. - Per-backend downtime paths (`advance_snapshot_time`), per-VP TSC advancement (`advance_tsc`), TSC-deadline advancement, periodic LAPIC advancement, wall-clock-only downtime (`calculate_snapshot_downtime`), and @@ -1429,8 +1505,9 @@ Migration impact: - The guest identifies as Hyper-V on every backend, including KVM. - CPUID becomes profile-defined, so guests can lose host features that no profile of their generation pins. -- Hosts that fail qualification, such as `azure-azlinux-2`, cannot run - microVMs until replaced. +- Hosts that fail qualification cannot run microVMs until replaced. + `azure-azlinux-2` stays out of rotation and unqualified because our + account cannot run guests there. - Per-PR CI captures and restores on the same runner. Same-generation cross-VM restore is validated by the fleet restore matrix on WHP only; no usable KVM or MSHV pair of one generation exists, so the simulated host From c4973aca117e19b1cd4fe26a9543bad4e99006c4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 12:17:58 -0700 Subject: [PATCH 039/188] doc: state the time ABI legacy L2-cache MSR rule in the MSR section Record the orchestrator's decision where the guest-visible MSR contract lives: on time ABI partitions, every backend raises #GP for the legacy P6 L2-cache MSRs (0x88..=0x8a, 0x116, 0x118..=0x11b, 0x11e), as MSHV's hypervisor already does. KVM and WHP keep their stubs, which exist only for Windows booting from the PCAT BIOS, in standard-machine partitions. Linux 6.18 never accesses them, and the boot check (C4) and the violation watcher (G_UNCHECKED_MSR) reject any unchecked MSR access error. Rename the section to "MSRs", since it also covers the architectural timer MSRs, and point the CPU profiles section and the obligations table at it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 0a296363f..4533eeb63 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -112,7 +112,7 @@ No leaf returns Hyper-V feature data, a `VS#1` interface signature at (step `0x100`) carries a KVM, Xen, VMware, or other hypervisor signature, so Linux selects only the Hyper-V platform. -### Synthetic MSRs +### MSRs OpenVMM serves the identity MSR range `0x40000000..=0x400001ff` identically on every backend: @@ -134,6 +134,14 @@ The CPU time bits hide two architectural timer MSRs, and every backend makes them raise #GP: `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`). +The legacy P6 L2-cache MSRs (`0x88..=0x8a`, `0x116`, `0x118..=0x11b`, and +`0x11e`) also raise #GP on every backend, as MSHV's hypervisor already +makes them do; no profile pins them. KVM and WHP stub them only for Windows +booting from the PCAT BIOS, and standard-machine partitions keep those +stubs. Linux 6.18 never accesses them. A fault on an MSR that the guest +kernel accesses unchecked logs `unchecked MSR access error`, which the boot +check (`C4`) and the violation watcher (`G_UNCHECKED_MSR`) reject. + ### CPU time bits Every CPU profile fixes these bits. A profile that violates them is rejected. @@ -263,7 +271,7 @@ settled cell on the registered hosts. | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities, so that the identity leaves never enable the Hyper-V emulator or its saved-state elements | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | -| Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [CPU profiles](#cpu-profiles)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | +| Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [MSRs](#msrs)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | | Removed | `KVM_GET_CLOCK`/`KVM_SET_CLOCK` in the microVM downtime path, kvmclock MSR state, leaf `0x15` synthesis, `KVM_SET_TSC_KHZ`, restore-time `IA32_TSC` writes | BSP-copy TSC alignment, exact-rate equality, leaf `0x15` synthesis | 1 GHz request and fallback, `RestoredTsc` and its RDTSC, RDTSCP, and `IA32_TSC` exits, leaf `0x15` synthesis | **KVM common offset.** Restoring each VP's `IA32_TSC` is unreliable on KVM. @@ -441,11 +449,8 @@ PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use the `7.0` EDX equivalents, which every backend presents). -No profile pins the legacy P6 L2-cache MSRs (`0x88..=0x8a`, `0x116`, -`0x118..=0x11b`, and `0x11e`) that the KVM and WHP backends stub for Windows -booting from the PCAT BIOS. A microVM boots its kernel directly and never -probes them, so on a time ABI partition they raise #GP on every backend, as -MSHV's hypervisor already does. +No profile pins the legacy P6 L2-cache MSRs; they raise #GP on every +backend (see [MSRs](#msrs)). ## TSC rate policy and LAPIC rate rule From d2cf575a307dcf376cad3bc31bd7acc500b18143 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 12:13:14 -0700 Subject: [PATCH 040/188] scripts: name OpenVMM's fatal time ABI code in harness exit errors OpenVMM leads its fatal-error message with the time ABI code that its error chain carries ("fatal error: [E_CODE] ", openvmm a90f9d450), so the harness and job logs can match it. The harness still reported such an exit as "OpenVMM exited with status 1" and left the code in the output tail. The time ABI monitor now records the first such line, and the harness's exit errors for cold boots, measured restores, snapshot captures, device snapshots, and OpenvmmProcess end with it, so the error line names the failure code. The line only explains an exit and never changes how it is classified: expected failures, such as the invalid-restore checks, still return their status. A capture source that exits nonzero before its snapshot request now reports that status and its fatal error instead of the generic missing-request message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/nvx_tools/benchmark.py | 30 +++++--------- scripts/nvx_tools/openvmm_process.py | 2 +- scripts/nvx_tools/time_abi.py | 22 ++++++++++ scripts/test_time_abi.py | 60 ++++++++++++++++++++++++++++ 4 files changed, 93 insertions(+), 21 deletions(-) diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index 64166fc8e..6f1152c0b 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -1831,9 +1831,7 @@ def finish_output( if process.poll() is not None: drain_exited_output(output, monitor) monitor.check_exit(process.returncode) - raise RuntimeError( - f"OpenVMM exited with status {process.returncode}" - ) from None + raise monitor.exit_error(process.returncode) from None continue if stream == "stderr": if profile is not None and chunk is not None: @@ -1843,7 +1841,7 @@ def finish_output( monitor.finish() returncode = exit_status_after_eof(process) monitor.check_exit(returncode) - raise RuntimeError(f"OpenVMM exited with status {returncode}") + raise monitor.exit_error(returncode) monitor.feed(chunk) console = output.console if failure_marker is not None: @@ -1903,9 +1901,7 @@ def finish_output( # finish_output() would drain them. drain_exited_output(output, monitor) monitor.check_exit(returncode) - raise RuntimeError( - f"OpenVMM exited with status {returncode} during teardown" - ) + raise monitor.exit_error(returncode, when="during teardown") finish_output(marker_reached, readiness_counters) return elapsed_ms, peak_bytes, teardown_ms, wall_ms except GuestFailureReported: @@ -2519,7 +2515,7 @@ def run_guest_script( returncode = process.wait() monitor.check_exit(returncode) if teardown_mode == "guest-exit" and returncode != 0: - raise RuntimeError(f"OpenVMM exited with status {returncode}") + raise monitor.exit_error(returncode) if not input_sent: raise RuntimeError("guest exited before its boot marker") if not completed: @@ -2608,7 +2604,7 @@ def capture_automatic_snapshot( returncode = process.wait() monitor.check_exit(returncode) if returncode != 0: - raise RuntimeError(f"snapshot source exited with status {returncode}") + raise monitor.exit_error(returncode, "snapshot source") for marker in required_markers: if marker not in output: raise RuntimeError( @@ -2698,9 +2694,7 @@ def capture_device_restore_snapshot( returncode = process.wait() monitor.check_exit(returncode) if returncode != 0: - raise RuntimeError( - f"{device}/{mode} snapshot source exited with status {returncode}" - ) + raise monitor.exit_error(returncode, f"{device}/{mode} snapshot source") markers = _device_restore_markers( output.decode("utf-8", "replace"), device, mode ) @@ -2816,9 +2810,7 @@ def observe_chunk(chunk: bytes, observed_ns: int) -> None: returncode = process.wait() monitor.check_exit(returncode) if returncode != 0: - raise RuntimeError( - f"{device}/{mode} restore exited with status {returncode}" - ) + raise monitor.exit_error(returncode, f"{device}/{mode} restore") for phase in ("restore-ready", "trigger", "io-success"): if phase not in phase_times: raise RuntimeError(f"{device}/{mode} restore is missing {phase!r}") @@ -4790,9 +4782,7 @@ def request_snapshot() -> None: if process.poll() is not None and snapshot_published_ns is None: drain_exited_output(output, monitor) monitor.check_exit(process.returncode) - raise RuntimeError( - f"OpenVMM exited with status {process.returncode}" - ) from None + raise monitor.exit_error(process.returncode) from None continue if chunk is None: if output.closed: @@ -4841,10 +4831,10 @@ def request_snapshot() -> None: monitor.check_exit(returncode) if snapshot_published_ns is None and snapshot_path.is_dir(): snapshot_published_ns = source_exited_ns + if returncode != 0: + raise monitor.exit_error(returncode, "snapshot source") if not snapshot_requested: raise RuntimeError("source guest exited before its snapshot request") - if returncode != 0: - raise RuntimeError(f"snapshot source exited with status {returncode}") if not snapshot_path.is_dir(): raise RuntimeError(f"snapshot was not published at {snapshot_path}") if snapshot_guest_dispatched_ns is None: diff --git a/scripts/nvx_tools/openvmm_process.py b/scripts/nvx_tools/openvmm_process.py index 2d97d90cf..a9e8635a8 100644 --- a/scripts/nvx_tools/openvmm_process.py +++ b/scripts/nvx_tools/openvmm_process.py @@ -122,7 +122,7 @@ def _exited(self, before: str) -> Exception: self._monitor.check_exit(returncode) except TimeAbiFailure as error: return error - return RuntimeError(f"OpenVMM exited with status {returncode} before {before}") + return self._monitor.exit_error(returncode, when=f"before {before}") def wait_for_time_abi(self, phase: str, timeout: float) -> dict[str, str]: """Wait for a passing ``NVX-TIME-ABI`` marker of a conformance phase.""" diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 589b410f7..32ed42f68 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -15,6 +15,10 @@ MARKER_PREFIX = "NVX-TIME-ABI: " VIOLATION_PREFIX = "NVX-TIME-ABI-VIOLATION: " REPORT_ONLY_PREFIX = "NVX-TIME-REPORT" +# OpenVMM's exit message leads with the time ABI code that its error chain +# carries: "fatal error: [E_CODE] ". +OPENVMM_FATAL_PREFIX = "fatal error: " +OPENVMM_FATAL_CODE_PREFIX = f"{OPENVMM_FATAL_PREFIX}[E_" ABI_VERSION = "1" STATUS_CLASSES: Mapping[int, str] = { 193: "conformance", @@ -240,6 +244,7 @@ def __init__(self, command: Sequence[str] = ()) -> None: self.restores: list[dict[str, str]] = [] self.uncertain: list[dict[str, str]] = [] self.violation: str | None = None + self.fatal: str | None = None self.report_only = False self.guest_booted = False self._pending = bytearray() @@ -265,6 +270,12 @@ def finish(self) -> None: def _line(self, line: str) -> None: line = _clean_line(line) + # A guest shell prompt without a newline can precede OpenVMM's line. + fatal = line.find(OPENVMM_FATAL_CODE_PREFIX) + if fatal >= 0: + if self.fatal is None: + self.fatal = line[fatal + len(OPENVMM_FATAL_PREFIX) :] + return if VIOLATION_PREFIX in line: try: event = parse_violation(line) @@ -313,6 +324,17 @@ def check_exit(self, returncode: int | None) -> None: event = self.violation or "no NVX-TIME-ABI-VIOLATION event was observed" raise TimeAbiFailure(f"{description}: {event}") + def exit_error( + self, returncode: int | None, subject: str = "OpenVMM", when: str = "" + ) -> RuntimeError: + """Return the error for a failed exit, led by its time ABI code if any.""" + message = f"{subject} exited with status {returncode}" + if when: + message += f" {when}" + if self.fatal is not None: + message += f": {self.fatal}" + return RuntimeError(message) + def require_boot( self, context: str, diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index e2e3d259d..18f0214c9 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -5,6 +5,7 @@ import contextlib import io import queue +import re import shutil import socket import subprocess @@ -25,6 +26,9 @@ ) KVM_BOOT = ["openvmm", "--hypervisor", "kvm", "--kernel", "vmlinux"] MSHV_RESTORE = ["openvmm", "--hypervisor", "mshv", "--restore-snapshot", "snap"] +# OpenVMM's openvmm_entry fatal_error_message, after a guest prompt. +FATAL_DETAIL = "[E_TSC_SYNC_UNSUPPORTED] failed to launch vm worker" +FATAL = f"~ # fatal error: {FATAL_DETAIL}\r\n\r\nCaused by:\r\n".encode() def warp_output( @@ -240,6 +244,22 @@ def test_bounds_an_unterminated_line(self): monitor.feed(b"\n" + BOOT_LINE.encode()) self.assertIsNotNone(monitor.boot) + def test_records_the_first_fatal_time_abi_error_for_exit_errors(self): + monitor = TimeAbiMonitor(KVM_BOOT) + monitor.feed(b"fatal error: failed to open the kernel\r\n") + self.assertIsNone(monitor.fatal) + self.assertEqual(str(monitor.exit_error(1)), "OpenVMM exited with status 1") + # A guest prompt without a newline can precede OpenVMM's line. + monitor.feed(FATAL + b" 0: [E_TSC_SYNC_UNSUPPORTED] no synchronized set\r\n") + monitor.feed(b"fatal error: [E_TEST_HOOK] later\r\n") + self.assertEqual(monitor.fatal, FATAL_DETAIL) + # The line only explains the exit; it never changes its classification. + monitor.check_exit(1) + self.assertEqual( + str(monitor.exit_error(1, "snapshot source", "during teardown")), + f"snapshot source exited with status 1 during teardown: {FATAL_DETAIL}", + ) + class WarpProbeTests(unittest.TestCase): def test_accepts_conclusive_passing_measurements(self): @@ -449,6 +469,46 @@ def test_measure_once_classifies_a_time_abi_power_off(self): with self.assertRaisesRegex(RuntimeError, "OpenVMM exited with status 1"): self._measure([b"restoring\n"], 1, MSHV_RESTORE) + def test_exit_errors_lead_with_openvmm_fatal_time_abi_code(self): + expected = f"exited with status 1: {re.escape(FATAL_DETAIL)}\n" + with self.assertRaisesRegex(RuntimeError, f"^OpenVMM {expected}"): + self._measure([b"restoring\n", FATAL], 1, MSHV_RESTORE) + interaction = FakeInteraction([b"booting\n", FATAL], 1) + with tempfile.TemporaryDirectory() as temporary: + with patch.object( + benchmark, "InteractiveProcess", return_value=interaction + ): + with self.assertRaisesRegex( + RuntimeError, f"^snapshot source {expected}" + ): + benchmark.capture_snapshot( + KVM_BOOT, + Path(temporary) / "snapshot", + backend="kvm", + timeout=5, + ) + log_path = Path(temporary) / "process.log" + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([FATAL], 1), + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + with self.assertRaisesRegex( + RuntimeError, + f"^OpenVMM exited with status 1 before .*: " + f"{re.escape(FATAL_DETAIL)}\n", + ): + process.wait_for(b"NEVER", 1) + # An expected failure still returns its status and output. + with patch.object( + openvmm_process, + "InteractiveProcess", + return_value=FakeInteraction([FATAL], 1), + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + self.assertEqual(process.wait(1).returncode, 1) + def test_measure_once_classifies_a_power_off_during_teardown(self): interaction = FakeInteraction([benchmark.RESTORE_MARKER + b"\n"], 194) with ( From bb5d2a46f7f657727fe32a10ecaf5a174e731aa5 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 12:21:46 -0700 Subject: [PATCH 041/188] scripts: check the CPU profile with OpenVMM's fingerprint tool in doctor H2 The spec's H2 (doc/design/time-abi.md, "Host qualification") runs `openvmm --hypervisor --cpu-fingerprint `, which writes the host's CPU fingerprint, checks it against the generation's profile, and prints one NVX-CPU-PROFILE: line (openvmm 07c22d21e). H2 so far only mapped the host's CPUID signature to a generation and its profile name. H2 now runs the tool. It requires exit status 0 and status=pass, cross-checks the backend, the generation, and the profile revision against its own detection, reports the profile and CPU surface digests in the log and the job summary, and otherwise fails with OpenVMM's code, for example E_PROFILE_UNSUPPORTED naming every unsupported CPUID bit. The fingerprint goes to nvx-cpu-fingerprint-.json in the probe directory, or where --cpu-fingerprint says. validate-runner qualifies a runner before its job has an OpenVMM binary, so it passes the new --no-openvmm: H2 then checks the CPU identity and generation only, and H3 and H6 cannot run. The microVM jobs will run the full H2 together with H3 once they verify the downloaded time ABI OpenVMM. H3 no longer accepts OpenVMM's interim host profile. OpenVMM selects the generation's catalog profile by default since fa64e3b37, so an interim one now means a regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 7 +- doc/ci.md | 28 ++-- scripts/nvx_tools/doctor.py | 186 ++++++++++++++++++--- scripts/test_nvx_tools.py | 7 +- scripts/test_time_abi.py | 174 ++++++++++++++++++- 5 files changed, 362 insertions(+), 40 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index b307e0856..ce9de285d 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -120,8 +120,9 @@ runs: # Time ABI host qualification (doc/design/time-abi.md, "Host # qualification"): the backend, the CPU fingerprint and generation, and # the TSC rate stability. It replaces the nonstop_tsc check, which is now - # recorded as evidence only. The guest warp probe runs in the microVM - # scenarios, which boot the time ABI. + # recorded as evidence only. Jobs qualify before they have an OpenVMM + # binary, so H2 does not run OpenVMM's CPU profile check. The guest warp + # probe runs in the microVM scenarios, which boot the time ABI. - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash @@ -129,6 +130,7 @@ runs: python3 scripts/nvx.py doctor --backend "${{ inputs.backend }}" --checks H1 H2 H4 + --no-openvmm --summary "${GITHUB_STEP_SUMMARY}" - name: Qualify host time on Windows @@ -138,6 +140,7 @@ runs: python scripts\nvx.py doctor --backend "${{ inputs.backend }}" --checks H1 H2 H4 + --no-openvmm --summary "$env:GITHUB_STEP_SUMMARY" - name: Validate KVM diff --git a/doc/ci.md b/doc/ci.md index 184fcee20..6003cb988 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -56,9 +56,10 @@ exposes an invariant TSC. The control log is kept as failure; the restore still fails. Every job that uses the `validate-runner` action first qualifies its runner -for the time ABI with `nvx.py doctor --checks H1 H2 H4` (see +for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm` (see [Host qualification](#host-qualification)): the backend, the CPU fingerprint -and generation, and the TSC rate stability. This replaces the earlier +and generation, and the TSC rate stability. The job has no OpenVMM binary yet, +so H2 skips OpenVMM's CPU profile check. This replaces the earlier `nonstop_tsc` check and takes a few seconds. It reports the CPU generation, the CPU profile that `auto` selects, and the measured TSC rate in the log and the job summary, and fails the job with a stable code when the runner is not @@ -179,8 +180,8 @@ brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. | Check | Implementation | | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | -| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | -| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names; OpenVMM's interim `interim.host..v1` profile is accepted until it selects catalog profiles. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | +| H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. Then `openvmm --hypervisor --cpu-fingerprint ` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line. H2 requires exit status 0, `status=pass`, the same backend and generation, and a revision of the generation's profile, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | +| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | | H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | | H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker, and the idle-inducing warp schedule stays within 1,000 ns: two rounds of `nvx-time-probe warp` over every CPU pair with all vCPUs halted for 1 s between them, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265) | @@ -190,19 +191,22 @@ H2, H4, and H5 use a dependency-free host probe, [`host_time_probe.rs`](../scripts/nvx_tools/host_time_probe.rs), which the doctor builds with `rustc` once per source version into `$RUNNER_TOOL_CACHE/nvx-host-time-probe` (or `build/host-time-probe` outside -CI). H3 and H6 need the OpenVMM binary and the guest artifacts; `--openvmm`, -`--kernel`, and `--initrd` override their default build paths. +CI). H2's profile check, H3, and H6 need the OpenVMM binary, and H3 and H6 +also need the guest artifacts; `--openvmm`, `--kernel`, and `--initrd` +override their default build paths. `--no-openvmm` qualifies a host without +OpenVMM: H2 then checks the CPU identity and generation only, and H3 and H6 +cannot run. Qualification gates on measured properties, alike on every backend: the guest warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource are evidence only. In CI, `validate-runner` runs the cheap host-level checks -H1, H2, and H4 before every job; jobs download OpenVMM and the guest artifacts -only later. The guest warp probe needs a time ABI boot, so it belongs in the -microVM boot and restore scenarios, which assert its verdict once OpenVMM -boots the time ABI by default. Without H3, H4 checks the rate stability -without comparing it against the backend's rate. H5 and H7 remain available -for interactive qualification. +H1, H2, and H4 before every job, with `--no-openvmm`, because jobs download +OpenVMM and the guest artifacts only later. The guest warp probe needs a time +ABI boot, so it belongs in the microVM boot and restore scenarios, which +assert its verdict once OpenVMM boots the time ABI by default. Without H3, H4 +checks the rate stability without comparing it against the backend's rate. H5 +and H7 remain available for interactive qualification. ## Rust crate diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 0123fb819..25580552c 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -33,6 +33,7 @@ WARP_BOUND_NS, WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_IDLE_SECONDS, + CpuGeneration, TimeAbiFailure, TimeAbiMonitor, check_warp_probe, @@ -45,17 +46,21 @@ CHECK_IDS = ("H1", "H2", "H3", "H4", "H5", "H6", "H7") CHECK_TITLES: Mapping[str, str] = { "H1": "backend device or API", - "H2": "CPU fingerprint and generation", + "H2": "CPU fingerprint, generation, and profile", "H3": "OpenVMM time ABI preflight", "H4": "host TSC rate stability", "H5": "host cross-CPU TSC skew", "H6": "guest warp probe", "H7": "host UTC synchronization", } +# Checks that boot OpenVMM; H2 also runs it unless qualification has none. +OPENVMM_CHECKS = ("H3", "H6") DOCTOR_PREFIX = "NVX-DOCTOR: " PROBE_SOURCE = Path(__file__).with_name("host_time_probe.rs") PROBE_PREFIX = "NVX-HOST-TIME-PROBE " VERIFY_PREFIX = "NVX-TIME-ABI-VERIFY:" +CPU_PROFILE_PREFIX = "NVX-CPU-PROFILE:" +RUST_ESCAPE = re.compile(r"\\(u\{[0-9a-fA-F]{1,6}\}|.)") RATE_MEASURE_MS = 1000 RATE_AGREEMENT_PPM = 1.0 RATE_TOLERANCE_PPM = 100.0 @@ -101,6 +106,9 @@ class DoctorContext: timeout: float facts: dict[str, str] = field(default_factory=dict[str, str]) probe: Path | None = None + # Where H2 writes OpenVMM's CPU fingerprint; None skips the profile check + # when qualification runs without OpenVMM. + fingerprint: Path | None = None def host_is_windows() -> bool: @@ -347,17 +355,129 @@ def check_cpu(context: DoctorContext) -> CheckResult: f"microcode={info.get('microcode', 'unknown')} os={info['os']} " f"invariant_tsc={info['invariant_tsc']} brand={info['brand']}" ) + problems: list[str] = [] if generation is None: - return CheckResult( - "H2", - False, + problems.append( f"[E_PROFILE_HOST_UNKNOWN] {signature} is not a time ABI generation " - "(skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids " - f"6/207); {detail}", + "(skylake-sp 6/85 steppings 0-4, icelake-sp 6/106, emeraldrapids 6/207)" ) + if context.fingerprint is None: + detail += "; CPU profile not checked: qualification runs without OpenVMM" + else: + profile_detail, profile_problems = _check_cpu_profile(context, generation) + detail += f"; {profile_detail}" + problems.extend(profile_problems) + if problems: + return CheckResult("H2", False, "; ".join(problems) + f"; {detail}") return CheckResult("H2", True, detail) +def parse_cpu_profile_line(line: str) -> dict[str, str]: + """Parse OpenVMM's ``NVX-CPU-PROFILE:`` line; ``detail`` is a Rust string.""" + text = line.strip().removeprefix(CPU_PROFILE_PREFIX) + head, quoted, detail = text.partition(' detail="') + fields = parse_fields(head) + if quoted: + fields["detail"] = RUST_ESCAPE.sub(_rust_unescape, detail.removesuffix('"')) + return fields + + +def _rust_unescape(match: re.Match[str]) -> str: + escape = match.group(1) + if escape.startswith("u{"): + return chr(int(escape[2:-1], 16)) + return {"n": "\n", "r": "\r", "t": "\t", "0": "\0"}.get(escape, escape) + + +def _same_profile_lineage(selected: str, expected: str) -> bool: + """Whether ``selected`` is a revision of the profile ``expected`` names.""" + return selected.startswith(expected.rpartition(".v")[0] + ".v") + + +def _check_cpu_profile( + context: DoctorContext, generation: CpuGeneration | None +) -> tuple[str, list[str]]: + """Fingerprint the host with OpenVMM and check its generation's profile.""" + assert context.fingerprint is not None + if not context.openvmm.is_file(): + return "CPU profile not checked", [ + f"OpenVMM was not found at {context.openvmm}; H2 checks the CPU " + "profile with its --cpu-fingerprint tool (or pass --no-openvmm)" + ] + context.fingerprint.parent.mkdir(parents=True, exist_ok=True) + environment = os.environ.copy() + environment["OPENVMM_LOG"] = "off" + completed = subprocess.run( + [ + os.fspath(context.openvmm), + "--hypervisor", + context.backend, + "--cpu-fingerprint", + os.fspath(context.fingerprint), + ], + capture_output=True, + text=True, + timeout=context.timeout, + env=environment, + check=False, + ) + output = f"{completed.stdout}\n{completed.stderr}" + line = next( + ( + line.strip() + for line in output.splitlines() + if line.strip().startswith(CPU_PROFILE_PREFIX) + ), + None, + ) + if line is None: + last = next( + (line.strip() for line in reversed(output.splitlines()) if line.strip()), + "no output", + ) + if "--cpu-fingerprint" in output and "unexpected argument" in output: + last = "this OpenVMM predates the --cpu-fingerprint tool" + return "CPU profile not checked", [ + f"OpenVMM's CPU fingerprint exited {completed.returncode} without " + f"an {CPU_PROFILE_PREFIX} line: {last}" + ] + fields = parse_cpu_profile_line(line) + for name in ("profile", "profile_digest", "surface_digest"): + if fields.get(name, "none") != "none": + context.facts[name] = fields[name] + context.facts["cpu_fingerprint"] = os.fspath(context.fingerprint) + detail = ( + f"CPU profile check status={fields.get('status')} " + f"profile={fields.get('profile')} " + f"profile_digest={fields.get('profile_digest', 'none')} " + f"surface_digest={fields.get('surface_digest')} " + f"fingerprint={context.fingerprint}" + ) + problems: list[str] = [] + if completed.returncode != 0 or fields.get("status") != "pass": + code = fields.get("code", "") + prefix = f"[{code}] " if code.startswith("E_") else "" + problems.append( + f"{prefix}OpenVMM's CPU profile check failed (exit " + f"{completed.returncode}): {fields.get('detail', line)}" + ) + return detail, problems + if fields.get("backend") != context.backend: + problems.append(f"OpenVMM fingerprinted the {fields.get('backend')} backend") + if generation is not None: + if fields.get("generation") != generation.name: + problems.append( + f"OpenVMM maps the host to generation {fields.get('generation')}, " + f"not {generation.name}" + ) + if not _same_profile_lineage(fields.get("profile", ""), generation.profile_id): + problems.append( + f"OpenVMM selected CPU profile {fields.get('profile')}, not a " + f"revision of {generation.profile_id}" + ) + return detail, problems + + def _guest_vcpus() -> int: host_cpus = os.cpu_count() or 1 return next(count for count in GUEST_VCPU_COUNTS if count <= host_cpus) @@ -447,20 +567,14 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: ) expected_profile = context.facts.get("profile", "none") selected_profile = fields.get("cpu_profile", "") - interim = f"interim.host.{context.backend}.v1" - if expected_profile != "none" and selected_profile != interim: - # Later revisions of the same generation are fine. - lineage = expected_profile.rpartition(".v")[0] + ".v" - if not selected_profile.startswith(lineage): - problems.append( - f"OpenVMM selected CPU profile {selected_profile}, but H2 maps the " - f"host to {expected_profile}" - ) + if expected_profile != "none" and not _same_profile_lineage( + selected_profile, expected_profile + ): + problems.append( + f"OpenVMM selected CPU profile {selected_profile}, but H2 maps the " + f"host to {expected_profile}" + ) detail = " ".join(f"{name}={value}" for name, value in fields.items()) - if selected_profile == interim: - # OpenVMM uses an interim host profile until it selects catalog - # profiles; core-design, "Changes". - detail += f"; interim CPU profile, expected {expected_profile} later" context.facts.update( { "tsc_hz": fields.get("tsc_hz", ""), @@ -716,6 +830,8 @@ def summary_markdown( ("microcode", "Microcode"), ("os", "Host OS"), ("profile", "Profile"), + ("profile_digest", "Profile digest"), + ("surface_digest", "CPU surface digest"), ("tsc_hz", "Declared TSC rate (Hz)"), ("lapic_hz", "LAPIC rate (Hz)"), ("measured_tsc_hz", "Measured TSC rate (Hz)"), @@ -736,14 +852,27 @@ def run(args: argparse.Namespace) -> int: if args.backend not in OPENVMM_TEST_BACKENDS: raise ScriptError(f"unsupported backend {args.backend!r}") checks = tuple(dict.fromkeys(args.checks or CHECK_IDS)) + if args.no_openvmm: + booting = [check for check in checks if check in OPENVMM_CHECKS] + if booting: + raise ScriptError( + f"--no-openvmm cannot run {' and '.join(booting)}, which boot OpenVMM" + ) + probe_directory = args.probe_dir or default_probe_directory() + fingerprint: Path | None = None + if not args.no_openvmm: + fingerprint = args.cpu_fingerprint or ( + probe_directory / f"nvx-cpu-fingerprint-{args.backend}.json" + ) context = DoctorContext( backend=args.backend, openvmm=args.openvmm or openvmm_binary_path(), kernel=args.kernel or artifact_path(KernelBuildConstants.BINARY_NAME), initrd=args.initrd or artifact_path(AlpineBuildConstants.INITRAMFS_NAME), openvmm_args=tuple(args.openvmm_arg), - probe_directory=args.probe_dir or default_probe_directory(), + probe_directory=probe_directory, timeout=args.timeout, + fingerprint=fingerprint, ) results = run_checks(context, checks) failed = [result.check for result in results if not result.passed] @@ -782,9 +911,24 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: metavar="ID", help="checks to run, in spec order (default: H1 to H7)", ) - parser.add_argument("--openvmm", type=Path, help="OpenVMM binary for H3 and H6") + parser.add_argument( + "--openvmm", type=Path, help="OpenVMM binary for H2, H3, and H6" + ) parser.add_argument("--kernel", type=Path, help="guest kernel for H6") parser.add_argument("--initrd", type=Path, help="guest initramfs for H6") + openvmm = parser.add_mutually_exclusive_group() + openvmm.add_argument( + "--cpu-fingerprint", + type=Path, + help="where H2 writes OpenVMM's CPU fingerprint " + "(default: nvx-cpu-fingerprint-.json in the probe directory)", + ) + openvmm.add_argument( + "--no-openvmm", + action="store_true", + help="qualify without an OpenVMM binary: H2 checks the CPU identity and " + "generation but not the CPU profile, and H3 and H6 cannot run", + ) parser.add_argument( "--probe-dir", type=Path, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index f081c1598..99a5725fb 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3190,7 +3190,8 @@ def test_runners_qualify_their_host_time_before_every_job(self): ).read_text(encoding="utf-8") # The doctor replaces the nonstop_tsc check: the backend, the CPU - # fingerprint and generation, and the TSC rate stability. + # fingerprint and generation, and the TSC rate stability. Jobs have no + # OpenVMM binary yet, so H2 skips OpenVMM's CPU profile check. self.assertNotIn("Validate host TSC", validate_runner) for name, shell, command, summary in ( ( @@ -3213,11 +3214,13 @@ def test_runners_qualify_their_host_time_before_every_job(self): self.assertIn(command, step) self.assertIn('--backend "${{ inputs.backend }}"', step) self.assertIn("--checks H1 H2 H4\n", step) + self.assertIn("--no-openvmm\n", step) self.assertIn(summary, step) args = nvx.parse_args( - ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4"] + ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "--no-openvmm"] ) self.assertEqual(args.checks, ["H1", "H2", "H4"]) + self.assertTrue(args.no_openvmm) self.assertLess( validate_runner.index("Validate Linux toolchain"), validate_runner.index("Qualify host time on Linux"), diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 18f0214c9..9b591fece 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -646,6 +646,12 @@ def completed(stdout: str, returncode: int = 0, stderr: str = ""): "os": "Linux 6.6.150.1-1.azl3", "invariant_tsc": "yes", } +# OpenVMM's cpu_profile FingerprintCheck::summary_line, printed on stderr. +PROFILE_PASS = ( + "NVX-CPU-PROFILE: status=pass backend=kvm generation=icelake-sp " + f"profile=intel.icelake-sp.v1 profile_digest=sha256:{'a3' * 32} " + "surface_digest=sha256:6286956acbef host_invariant_tsc=yes\n" +) class DoctorTests(unittest.TestCase): @@ -720,6 +726,130 @@ def test_cpu_check_names_the_generation_and_records_invariant_tsc_as_evidence( context.facts["invariant_tsc"], "no (missing nonstop_tsc)" ) + def test_cpu_check_verifies_the_profile_with_openvmm_fingerprint(self): + fingerprint = self.root / "out" / "fingerprint.json" + + def context_with_openvmm(backend: str = "kvm") -> doctor.DoctorContext: + context = doctor_context(self.root, backend) + context.openvmm.write_bytes(b"") + context.fingerprint = fingerprint + return context + + def check( + result: subprocess.CompletedProcess[str], + cpu: dict[str, str] = CPU, + backend: str = "kvm", + ) -> tuple[doctor.CheckResult, doctor.DoctorContext]: + context = context_with_openvmm(backend) + with ( + patch.object(doctor, "host_cpu", return_value=dict(cpu)), + patch.object(doctor.subprocess, "run", return_value=result), + ): + return doctor.check_cpu(context), context + + context = context_with_openvmm() + with ( + patch.object(doctor, "host_cpu", return_value=dict(CPU)), + patch.object( + doctor.subprocess, "run", return_value=completed("", 0, PROFILE_PASS) + ) as run, + ): + result = doctor.check_cpu(context) + self.assertTrue(result.passed, result.detail) + self.assertEqual( + run.call_args.args[0], + [ + str(context.openvmm), + "--hypervisor", + "kvm", + "--cpu-fingerprint", + str(fingerprint), + ], + ) + self.assertEqual(run.call_args.kwargs["env"]["OPENVMM_LOG"], "off") + self.assertTrue(fingerprint.parent.is_dir()) + self.assertIn("surface_digest=sha256:6286956acbef", result.detail) + self.assertEqual(context.facts["profile"], "intel.icelake-sp.v1") + self.assertEqual(context.facts["profile_digest"], f"sha256:{'a3' * 32}") + self.assertEqual(context.facts["surface_digest"], "sha256:6286956acbef") + # A later revision of the generation's profile is fine. + revised = PROFILE_PASS.replace("icelake-sp.v1", "icelake-sp.v2") + self.assertTrue(check(completed("", 0, revised))[0].passed) + + unsupported = ( + "NVX-CPU-PROFILE: status=fail backend=whp generation=icelake-sp " + "profile=intel.icelake-sp.v1 profile_digest=sha256:a3 " + "surface_digest=sha256:d67f host_invariant_tsc=no " + 'code=E_PROFILE_UNSUPPORTED detail="CPUID 0x7.0 EDX bits 26, 27 are ' + 'not supported; leaf \\"7\\"\\u{a0}subleaf"\n' + ) + result, context = check(completed("", 1, unsupported), backend="whp") + self.assertFalse(result.passed) + self.assertTrue( + result.detail.startswith( + "[E_PROFILE_UNSUPPORTED] OpenVMM's CPU profile check failed (exit 1): " + 'CPUID 0x7.0 EDX bits 26, 27 are not supported; leaf "7"\xa0subleaf;' + ), + result.detail, + ) + unknown = ( + "NVX-CPU-PROFILE: status=fail backend=kvm generation=none profile=none " + "surface_digest=sha256:1 host_invariant_tsc=yes " + 'code=E_PROFILE_HOST_UNKNOWN detail="GenuineIntel family 6 model 143"\n' + ) + result, context = check(completed("", 1, unknown), dict(CPU, model="143")) + self.assertFalse(result.passed) + self.assertTrue(result.detail.startswith("[E_PROFILE_HOST_UNKNOWN] ")) + self.assertIn("[E_PROFILE_HOST_UNKNOWN] OpenVMM's CPU profile", result.detail) + self.assertEqual(context.facts["profile"], "none") + for output, message in ( + ( + completed("", 2, "error: unexpected argument '--cpu-fingerprint'"), + "predates the --cpu-fingerprint tool", + ), + (completed("", 1, "fatal error: no backend"), "fatal error: no backend"), + ( + completed("", 0, PROFILE_PASS.replace("=kvm", "=mshv")), + "fingerprinted the mshv backend", + ), + ( + completed( + "", + 0, + PROFILE_PASS.replace( + "generation=icelake-sp", "generation=emeraldrapids" + ), + ), + "generation emeraldrapids, not icelake-sp", + ), + ( + completed( + "", 0, PROFILE_PASS.replace("intel.icelake", "intel.skylake") + ), + "profile intel.skylake-sp.v1, not a revision", + ), + ): + with self.subTest(message=message): + result = check(output)[0] + self.assertFalse(result.passed) + self.assertIn(message, result.detail) + missing = doctor_context(self.root / "none") + missing.fingerprint = fingerprint + with patch.object(doctor, "host_cpu", return_value=dict(CPU)): + result = doctor.check_cpu(missing) + self.assertFalse(result.passed) + self.assertIn("was not found", result.detail) + self.assertIn("--no-openvmm", result.detail) + # Without OpenVMM, H2 checks the identity and generation only. + with ( + patch.object(doctor, "host_cpu", return_value=dict(CPU)), + patch.object(doctor.subprocess, "run") as run, + ): + result = doctor.check_cpu(doctor_context(self.root)) + self.assertTrue(result.passed) + self.assertIn("CPU profile not checked", result.detail) + run.assert_not_called() + def test_reads_the_first_processor_from_cpuinfo(self): path = self.root / "cpuinfo" path.write_text( @@ -802,9 +932,8 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: for selected, passed in ( ("intel.icelake-sp.v2", True), ("intel.skylake-sp.v1", False), - # OpenVMM's interim host profile until it selects catalog profiles. - ("interim.host.kvm.v1", True), - ("interim.host.mshv.v1", False), + # OpenVMM selects catalog profiles, so an interim one is a regression. + ("interim.host.kvm.v1", False), ): with self.subTest(selected=selected): context = context_with_files() @@ -1034,6 +1163,45 @@ def broken(context: doctor.DoctorContext) -> doctor.CheckResult: self.assertIn("**fail** | rustc is required", text) self.assertIn("Generation: `emeraldrapids`", text) + def test_run_selects_the_cpu_fingerprint_or_runs_without_openvmm(self): + seen: list[Path | None] = [] + + def cpu(context: doctor.DoctorContext) -> doctor.CheckResult: + seen.append(context.fingerprint) + context.facts["surface_digest"] = "sha256:62" + return doctor.CheckResult("H2", True, "ok") + + summary = self.root / "summary.md" + common = ["--backend", "kvm", "--checks", "H2", "--summary", str(summary)] + common += ["--probe-dir", str(self.root)] + with ( + patch.dict(doctor.CHECKS, {"H2": cpu}), + contextlib.redirect_stdout(io.StringIO()), + ): + for extra in ( + [], + ["--cpu-fingerprint", str(self.root / "fingerprint.json")], + ["--no-openvmm"], + ): + arguments = doctor_parser().parse_args([*common, *extra]) + self.assertEqual(doctor.run(arguments), 0) + self.assertEqual( + seen, + [ + self.root / "nvx-cpu-fingerprint-kvm.json", + self.root / "fingerprint.json", + None, + ], + ) + self.assertIn("CPU surface digest: `sha256:62`", summary.read_text("utf-8")) + arguments = doctor_parser().parse_args(["--backend", "kvm", "--no-openvmm"]) + with self.assertRaisesRegex(doctor.ScriptError, "cannot run H3 and H6"): + doctor.run(arguments) + with contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + doctor_parser().parse_args( + ["--backend", "kvm", "--no-openvmm", "--cpu-fingerprint", "x"] + ) + def test_builds_the_host_probe_once_per_source_version(self): target = doctor.probe_path(self.root) self.assertRegex(target.name, r"^nvx-host-time-probe-[0-9a-f]{16}(\.exe)?$") From 45de5382f8284440d892e3d6d323f544a9e4577e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:07:15 -0700 Subject: [PATCH 042/188] doc: cover management-endpoint microVMs and --restore-entropy in the time ABI Every microVM uses the time ABI, including microVMs created through OpenVMM's management endpoint. With restore packet v4 on every restore, --restore-entropy has no effect: OpenVMM still accepts it, and the harness stops passing it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 4533eeb63..aa1ae7bc4 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -29,8 +29,10 @@ behavior. Every check in this document is a hard check. Goals: -- One time ABI on every backend: the same CPUID identity, synthetic MSRs, - CPU time bits, clocksource, tick, timers, and restore semantics. +- One time ABI on every backend and for every microVM, whether launched from + the command line or through OpenVMM's management endpoint: the same CPUID + identity, synthetic MSRs, CPU time bits, clocksource, tick, timers, and + restore semantics. - No guest kernel patches and no clock command-line tokens. - No implicit fallbacks. A host or snapshot that cannot meet the contract is rejected with a stable error code. @@ -1477,6 +1479,9 @@ behavior: - WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` with its RDTSC, RDTSCP, and `IA32_TSC` exits. - Restore packets v1 to v3 and manifest versions 2 to 5. +- The effect of `--restore-entropy`: every restore exposes packet v4, which + carries fresh entropy. OpenVMM still accepts the option without effect, + and the harness stops passing it. Removed from NVX: From a9d7c5c0a12bf4c9f45a1d5e79af3cf441b2dd23 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:29:19 -0700 Subject: [PATCH 043/188] doc: make the host clocksource evidence only in time ABI qualification The host clocksource never gates qualification on any backend; H4 reports it on Linux hosts as evidence. MSHV roots and Azure KVM hosts run hyperv_clocksource_tsc_page, the guest has no kvmclock, and the downtime uses host monotonic time and UTC, whose accuracy does not depend on the clocksource. KVM's TSC offset rewrites on hosts with an unstable TSC are covered by the H6 warp probe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index aa1ae7bc4..f4b479c5c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1266,7 +1266,7 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | `H1` | The backend device or API is present and usable | | `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | -| `H4` | TSC rate stability against the host's monotonic clock (see [Rate stability](#rate-stability-h4)) | +| `H4` | TSC rate stability against the host's monotonic clock (see [Rate stability](#rate-stability-h4)). On a Linux host the detail also reports the host clocksource, as evidence only | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | | `H6` | Guest warp probe on the qualification schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | @@ -1292,7 +1292,8 @@ uncertainty `u_i`, the two samples' uncertainties divided by the interval; run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. The detail reports `r`, its deviation from `F_d` in ppm, the agreement, the -largest `u_i`, and the clock. An invariant TSC agrees with the host clock far +largest `u_i`, the clock, and, on Linux, the host clocksource, which never +gates. An invariant TSC agrees with the host clock far inside 1 ppm: 0.001 to 0.089 ppm over 1 s windows on the eleven CI runners, and 0.000 ppm with a residual of at most 0.07 µs over 118 s on the WHP hosts. A TSC that stops, slows, or is rescaled across idle misses the bound by @@ -1322,16 +1323,25 @@ of the warps in #265: Qualification gates alike on every backend, on measured properties only: the CPU profile (`H2`, `H3`), rate stability (`H4`), and the idle-scheduled warp probe (`H6`, and the CI schedule in every microVM job). The host OS's -invariant-TSC bit and the Linux host clocksource are recorded as evidence -and never gate. Azure's WHP and nested MSHV hosts hide the bit from their -guests, and the profile exposes it anyway, so `H4` and `H6` measure what the -bit promises. MSHV roots and Azure KVM hosts run -`hyperv_clocksource_tsc_page`, and KVM rewrites per-vCPU TSC offsets only on -a host that marked its TSC unstable without that clocksource, which `H6` -measures. `azure-azlinux-2` (8370C), whose host OS lacks the bit and which -showed the #265 warps, is out of rotation and unqualified because our -account cannot run guests there, not because of this rule; its host-level -warp probe saw backward steps of at most 2.1 ns. +invariant-TSC bit and the host clocksource are recorded as evidence and never +gate, on any backend: + +- Azure's WHP and nested MSHV hosts hide the invariant-TSC bit from their + guests, and the profile exposes it anyway, so `H4` and `H6` measure what the + bit promises. +- MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, not + `tsc`. The guest has no kvmclock, so no guest clock derives from the host + clocksource, and the downtime comes from host monotonic time and UTC, whose + accuracy does not depend on it. +- KVM rewrites a vCPU's TSC offset whenever it loads the vCPU on a host that + marked its TSC unstable, unless the host clocksource is + `hyperv_clocksource_tsc_page`. `H6` measures the warps that this would + cause. + +`azure-azlinux-2` (8370C), whose host OS lacks the bit and which showed the +#265 warps, is out of rotation and unqualified because our account cannot run +guests there, not because of this rule; its host-level warp probe saw +backward steps of at most 2.1 ns. ### Generations and runner placement From bc889560240c76691efc79340032f91843652b4c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:33:11 -0700 Subject: [PATCH 044/188] doc: resume partition time at the read-back on every time ABI backend Guest time runs from the restore anchor on every backend: KVM's TSC never stops, and MSHV and WHP resume partition time right after a successful read-back, inside the synchronized TSC set, rather than at the first VP run. Restore steps 13 and 17 and the backend obligations say so, and the downtime section records MSHV's guest monotonic error with the early resume (-0.05 to -0.55 ms, against -0.9 to -1.6 ms before), at parity with KVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index f4b479c5c..6d5d9af8c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -268,8 +268,8 @@ settled cell on the registered hosts. | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`). There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | | Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and resume partition time right after the read-back (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume explicitly with `WHvResumePartitionTime` before any VP runs. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | -| Read-back before release | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | +| Read-back before any VP runs | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities, so that the identity leaves never enable the Hyper-V emulator or its saved-state elements | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | @@ -584,8 +584,11 @@ the guest and tests can tell the two paths apart. The restore anchor is the instant of the synchronized TSC set, and the guest TSC runs from that instant on every backend: KVM never stops it, and MSHV and -WHP resume partition time right after the read-back. The guest monotonic -advance across a restore therefore lies in `[D, D + R]`, up +WHP resume partition time right after a successful read-back, inside the +synchronized set (see step 13 of the +[restore algorithm](#restore-algorithm)). No backend defers the resume to the +first VP run. The guest monotonic advance across a restore therefore lies in +`[D, D + R]`, up to the two anchors' pairing errors (each at most 100 µs), where `R` is the time from the restore anchor to the first restored instruction; it falls short of `D + R` only by the frozen write (at most about 0.5 ms). Wall-clock @@ -593,7 +596,9 @@ repair absorbs `R` and the pairing errors for `CLOCK_REALTIME`. The KVM spike kept guest `CLOCK_MONOTONIC` within −0.09 to +2.8 ms of the host interval across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive part came from a host sample taken before the per-VP TSC save, which the -paired capture anchor removes. +paired capture anchor removes. MSHV measured −0.05 to −0.55 ms with the +resume at the read-back, at parity with KVM, against −0.9 to −1.6 ms when the +first VP run thawed time; warps and restore latency were unchanged. **Anchor pairing bound.** An anchor that pairs a TSC read with a host time sample (the capture anchor on every backend, and KVM's restore anchor) @@ -677,7 +682,12 @@ In the worker, with every VP stopped: legacy clock entry points (`set_tsc_frequency_hz`, `advance_snapshot_time`) on a time ABI partition; KVM rejects both there. 13. Read back: every instantiated VP holds the synchronized value as defined - by its backend (`E_TSC_SYNC_READBACK`). + by its backend (`E_TSC_SYNC_READBACK`). A backend that froze partition + time for the set resumes it right after a successful read-back, before + the synchronized set returns: MSHV clears `TimeFreeze`, and WHP calls + `WHvResumePartitionTime`; KVM's TSC never stops. Guest time therefore + runs from the restore anchor on every backend. A failed read-back leaves + time frozen and fails the restore. 14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every VP's LAPIC state again, always after step 12 and even when no timer is armed: KVM derives its timer deadline from the guest TSC when the LAPIC @@ -687,7 +697,9 @@ In the worker, with every VP stopped: 16. Seal the time fields of the restore packet: `D`, its source, the rate deviation, `g`, and the test-hook flag. 17. Start the state units, with host input gated when the restore requires an - acknowledgement; publish the readiness event; release the VPs. + acknowledgement; publish the readiness event; release the VPs. Partition + time has run since step 13, so releasing the VPs does not start it, and + no backend resumes time at the first VP run. The guest then repairs its clocks (see [Snapshot agent](#snapshot-agent)) before acknowledging a gated restore. From 7a01d7be6743ec1318067f26484ca73fd7ef4348 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:53:03 -0700 Subject: [PATCH 045/188] doc: scope time ABI lineages to one capture and its restores OpenVMM cannot capture a restored microVM: --snapshot-destination and --restore-snapshot are exclusive, and so are the management endpoint's capture and restore paths. A lineage is therefore one capture and its restores. The rate rule says so and stays defined for a future re-capture. The restore matrix replaces its undrivable capture-restore chain row with repeated restores, and leaves longer chains to the unit tests of the generation arithmetic. The downtime section also records the KVM agent's comparison: the legacy path left guest monotonic time about 400 ms behind on azure-kvm-5, where the time ABI stays within 1.5 ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6d5d9af8c..7d174ccb2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -180,7 +180,11 @@ measured invariance (see [Host qualification](#host-qualification)). - **TSC.** The guest TSC runs at the destination's native rate `F_d` and is never scaled. The declared rate `F` is fixed for the life of a snapshot lineage: a cold boot declares its native rate, and every restored process - keeps `F = F_s`, including when it is captured again. The kernel computes + keeps `F = F_s`. OpenVMM does not capture a restored process + (`--snapshot-destination` and `--restore-snapshot` are exclusive, and so + are the management endpoint's capture and restore paths), so a lineage is + one capture and its restores; if re-capture is added, the new snapshot + records `F_s` and the restored process's `g`. The kernel computes `tsc_khz = floor(F / 1000)` once and never recalibrates. The [rate policy](#tsc-rate-policy-and-lapic-rate-rule) bounds `|F_d - F| / F`. @@ -598,7 +602,11 @@ across restores with 0 s and 30 s of downtime at 1 to 8 vCPUs; its positive part came from a host sample taken before the per-VP TSC save, which the paired capture anchor removes. MSHV measured −0.05 to −0.55 ms with the resume at the read-back, at parity with KVM, against −0.9 to −1.6 ms when the -first VP run thawed time; warps and restore latency were unchanged. +first VP run thawed time; warps and restore latency were unchanged. The +legacy path, which stopped guest time during restore setup, left guest +`CLOCK_MONOTONIC` about 400 ms behind the host on every restore on +`azure-kvm-5` with 512 MiB (7 to 12 ms on prometheus32); the time ABI stays +within 1.5 ms there. **Anchor pairing bound.** An anchor that pairs a TSC read with a host time sample (the capture anchor on every backend, and KVM's restore anchor) @@ -1446,7 +1454,7 @@ can use as the fleet restore matrix. | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | -| Every tier and an untiered snapshot; capture-restore chains | All backends | Restored; `g` increments by one per restore | +| Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | No test reboots a host. The orchestrator asks the user before any real reboot. From 91109d3532d50ee9b2fe20b2cf2abd252be78cd0 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:56:21 -0700 Subject: [PATCH 046/188] doc: say what Azure backends lack in invariant TSC The host OS sees an invariant TSC on every runner except azure-azlinux-2. What Azure's WHP and nested MSHV lack is the ability to offer it to guests through their feature banks, which is why the profile forces the bit through a CPUID override. Correct the CPU time bits, the profile verification exemption, the backend table, the qualification gates, the performance table, and the restore matrix, which said that these hosts hide the bit. This matches the ci agent's 4994dba. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 7d174ccb2..93448bb41 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -169,11 +169,12 @@ Linux trusts the TSC because of `AccessTscInvariantControls`, not because of CPUID `0x80000007` EDX[8]; that bit only adds the `nonstop_tsc` flag and spares each AP a delay-loop calibration of about 150 ms (on Azure MSHV, an 8-vCPU cold boot takes 1,385 ms without the bit and 299 ms with it). The time -bits override the fingerprint: the bit is policy, not a fingerprint feature, -so profiles set it even where a nested hypervisor hides it from host -fingerprints (Azure's MSHV and WHP L1 partitions), every backend exposes it -(WHP through its CPUID override), and host qualification backs it with -measured invariance (see [Host qualification](#host-qualification)). +bits override the fingerprint: the bit is policy, not a fingerprint feature. +Profiles set it even where the backend cannot offer it to guests through its +feature banks, as on Azure's WHP and nested MSHV, whose host OS sees an +invariant TSC. Every backend exposes it (MSHV and WHP through a CPUID +override), and host qualification backs it with measured invariance (see +[Host qualification](#host-qualification)). ### Rates @@ -269,7 +270,7 @@ settled cell on the registered hosts. | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | | TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | -| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile: Azure's nested MSHV does not pass the bit through, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: Azure WHP hosts cannot expose it through the feature banks (bank 1 lacks `TscInvariant`). There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | +| Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile through a CPUID intercept result: Azure's nested MSHV cannot offer the bit to its guests, although the host OS sees an invariant TSC, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: on Azure, WHP cannot offer it through the feature banks (bank 1 lacks `TscInvariant`), although the host OS sees an invariant TSC. There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | | Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | @@ -354,8 +355,10 @@ state: `OSXSAVE`, `OSPKE`, and the XSAVE sizes for the current XCR0 and XSS. The time ABI owns `0x40000000..=0x4fffffff`, `0x15`, and `0x16`, and every profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and ARAT are pinned set and, with the hypervisor bit, exempt from host-support -verification, because nested Azure hypervisors hide them from fingerprints; -host qualification measures them instead. Profiles also pin policy zeros +verification, because a backend's fingerprint can lack them: WHP's feature +banks cannot express ARAT, and on Azure, WHP and nested MSHV cannot offer +invariant TSC to guests through their feature banks although the host OS +sees it. Host qualification measures them instead. Profiles also pin policy zeros (VMX and SVM, SGX, PT, RDT, PCONFIG, and the other features listed by the profiles' derivation policy). The effective guest CPUID is a pure function of the profile, the VM topology, and the time ABI's identity leaves. @@ -1346,9 +1349,10 @@ probe (`H6`, and the CI schedule in every microVM job). The host OS's invariant-TSC bit and the host clocksource are recorded as evidence and never gate, on any backend: -- Azure's WHP and nested MSHV hosts hide the invariant-TSC bit from their - guests, and the profile exposes it anyway, so `H4` and `H6` measure what the - bit promises. +- On Azure, WHP and nested MSHV cannot offer the invariant-TSC bit to their + guests through their feature banks, although the host OS sees an invariant + TSC on every runner except `azure-azlinux-2`. The profile exposes the bit + through CPUID anyway, so `H4` and `H6` measure what it promises. - MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, not `tsc`. The guest has no kvmclock, so no guest clock derives from the host clocksource, and the downtime comes from host monotonic time and UTC, whose @@ -1392,7 +1396,7 @@ Expected effects: | Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | | No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | | `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | -| Invariant TSC from the profile where the host hides it (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | +| Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Added cold-boot cost, reported as the first run's `elapsed_us`; budget 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP; the gate is authoritative | @@ -1450,7 +1454,7 @@ can use as the fleet restore matrix. | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | | Across VMs of one generation | `azure-windows-1` to `-2`; `azure-windows-3` to `-4`. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | `azure-windows-1` (8370C) to `-3` (8573C); prometheus32 to `azure-kvm-5` (KVM); prometheus30 to `azure-azlinux-5` (MSHV) | `E_CPU_GENERATION` | -| Backend that hides invariant TSC from its guests | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | +| Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | From 31b3371711f4347bcb8e2756c6c98a5744a20948 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 15:05:35 -0700 Subject: [PATCH 047/188] doc: keep the time ABI quiet on the console in production Console bytes cost one or two port exits each, so the boot marker (about 110 bytes) and the restore marker (about 117 bytes) cost 1 to 5 ms on the measured paths and can fail the gate on azure-kvm-5. - Production prints no time ABI marker or status line. Checks record their result in the state file, which gains the check fields, and /sbin/nvx-time status prints the boot, capture, or restore marker and a runtime line on demand. It waits for pending checks for at most 30 s. CI scenarios and the matrix driver run it over the console. - Violations still print their event and power off with their status. - Before shell-ready, only the initial time sample and clock step (C12) remain. The other boot checks and the daemon start run asynchronously afterwards and stay fail-fast. - Before the 0x605 acknowledgement, only the packet read, the clock set, CPU and memory activation, and entropy and identity repair remain. The restore checks, the RCU release, and the deferred C7 run asynchronously afterwards. - Update the uncertainty bounds, the failure-code text, the performance table and attribution, and the conformance and restore matrices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 156 ++++++++++++++++++++++++++--------------- 1 file changed, 98 insertions(+), 58 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 93448bb41..98435635e 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -908,8 +908,8 @@ otherwise, and then fails with a stable code: | Use | Bound | Attempts | When every attempt exceeds the bound | | --- | --- | --- | --- | | Restore repair (packet bracket, then time samples) | 1 ms | 3 | `G_REPAIR_SAMPLE`: event and power-off with status 195 | -| Initial synchronization at boot (time samples) | 1 ms | 3 | `G_CONFORMANCE_C12`: event and power-off with status 193 | -| Discipline poll (time samples) | 50 µs | 3 | `G_SAMPLE_UNCERTAIN`: the poll is skipped and the code is recorded in the state file and on the console; never fatal | +| Initial synchronization at boot, before shell-ready (time samples) | 1 ms | 3 | `G_CONFORMANCE_C12`: event and power-off with status 193 | +| Discipline poll (time samples) | 50 µs | 3 | `G_SAMPLE_UNCERTAIN`: the poll is skipped and the code is recorded in the state file; never fatal | `epsilon` is half the round trip of one PMIO write exit plus two `CLOCK_REALTIME` reads, so it stays far below both bounds on every backend. @@ -953,14 +953,31 @@ retries, and a poll whose three attempts all fail is skipped without harm The guest's time component (`nvx-time` in `guest/common`) performs the conformance checks, runs the violation watcher and the wall-clock discipline -as one daemon, and executes the snapshot agent's time steps. Every guest mode -starts it: init's interactive and test paths, `nvx_exec`, the sandbox agent, -and the managed agent. +as one daemon, executes the snapshot agent's time steps, and reports status +on demand. Every guest mode starts it: init's interactive and test paths, +`nvx_exec`, the sandbox agent, and the managed agent. + +Production paths keep the console quiet: the time component prints no +marker or status line, and only a [violation event](#violation-watcher) +reaches the console. Each console byte costs one or two port exits, about 10 +to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure, so a +110-byte boot marker and a 117-byte restore marker would cost 1 to 5 ms on +the measured paths. State goes to the [state file](#wall-clock-discipline), +and `nvx-time status` prints it on demand. ### Conformance checks and the `NVX-TIME-ABI` marker -Init runs the boot check immediately after mounting `/proc`, `/sys`, and -`/dev`, before any other guest work, and starts the daemon only if it passes. +Before shell-ready, init does only what boot correctness needs. Right after +mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the +[uncertainty bound](#uncertainty-bounds) and steps the clock to host UTC +(`C12`), so the workload starts on host time; this step is not counted as a +discontinuity. Init then reports shell-ready (or starts the workload, in +modes without a shell) and runs every other boot check asynchronously. The +checks start the daemon when they pass. Every check stays fail-fast: a +failure powers the guest off with status 193, even if the workload is +running. A guest that powers off before its checks finish skips them, which +is why CI takes its evidence from `nvx-time status`, which waits for them. + CPUID is executed on every online CPU (the checker pins itself to each CPU in turn, or runs one thread pinned to each CPU), and MSRs are read through `/dev/cpu//msr`. A CPU's VP index is its Linux CPU number, because CPUs @@ -981,7 +998,7 @@ partition-wide on every backend, so `C3` reads them on CPU 0 only. | `C9` | `/proc/cmdline` contains none of `tsc_early_khz=`, `lapic_timer_hz=`, `notsc`, `nolapic`, `nolapic_timer`, `tsc=unstable`, `hpet=force`, or `clocksource=` with a value other than `tsc` | boot | | `C10` | The time daemon is running and has recorded no violation; `/sys/kernel/rcu_stall_count` is 0; at boot, the state file is published and the daemon started | boot, capture, restore | | `C11` | Debug kernel only: `/proc/sys/kernel/soft_watchdog` is 1 and `/proc/sys/kernel/hung_task_timeout_secs` is nonzero | boot | -| `C12` | A time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | +| `C12` | Before shell-ready: a time sample within the [uncertainty bound](#uncertainty-bounds) is obtained, and the clock is stepped to host UTC | boot | | `K1` | Kernel integrity, not a clock property: the kernel's boot-time W+X audit logged no `Found insecure W+X mapping` record | boot | Capture and restore checks never wait. The boot `C7` check and the deferred @@ -1010,33 +1027,33 @@ NVX-TIME-ABI-EXHAUSTIVE: v=1 check= cpu= status= detail=" cpus= failures= ``` -On success the check prints one line to the console: +**Status command.** Checks print nothing on success; they record their +result in the [state file](#wall-clock-discipline). `/sbin/nvx-time status` +prints it on demand: CI scenarios and the matrix driver run it over the +console when they need evidence, and production paths never do. It waits +until no check is pending (the asynchronous boot checks, or a restore's +deferred work), for at most 30 s. It then prints the marker of the last +completed check and the runtime line: ```text NVX-TIME-ABI: v=1 phase= status=ok cpus= tsc_hz= lapic_hz= generation= elapsed_us= +NVX-TIME-ABI: v=1 phase=runtime status= generation= discontinuities= offset_ns= uncertainty_ns= rejected_samples= last_sample_error= ``` -`elapsed_us` covers the checks and, at boot, the daemon start. The line is -printed after it is taken, synchronously and before any workload starts, so -a harness sees the marker before workload output and an early power-off -cannot lose it; its console cost (one port exit per byte) is part of the -cold-boot cost the performance gate measures. - -On failure it prints `status=fail check= detail=""` in the same -format, emits a violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` -for `K1`), and powers -off with status 193. The boot step of `C12` steps the clock before any -workload starts and is not counted as a discontinuity. The only other line -with this prefix is the non-fatal `phase=runtime status=uncertain` line of -the [wall-clock discipline](#wall-clock-discipline). +It exits with status 0. If a check is still pending after 30 s, the first +line reports `status=pending`, without `elapsed_us`, and the exit status is +1. `elapsed_us` covers the checks and, at boot, the daemon start. A failed +check prints no marker: it emits the violation event with code +`G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193. **Report-only mode (test only).** The kernel token `nvx_time_abi=report-only` makes the checks and the watcher report instead of powering off, so the guest -can be evaluated under an OpenVMM that does not implement the time ABI. It -prints `NVX-TIME-REPORT` and `NVX-TIME-REPORT-VIOLATION` lines with the same -fields, and its marker ends with ` failures=`. These prefixes are reserved -for this mode; production images never set the token, and harnesses never -accept a report-only boot as conformant. +can be evaluated under an OpenVMM that does not implement the time ABI. +Violations print `NVX-TIME-REPORT-VIOLATION` lines with the event's fields, +and `nvx-time status` prints its marker with the `NVX-TIME-REPORT` prefix and +` failures=` appended. These prefixes are reserved for this mode; +production images never set the token, and harnesses never accept a +report-only boot as conformant. ### Violation watcher @@ -1077,10 +1094,11 @@ NVX-TIME-ABI-VIOLATION: v=1 code= source= phas ``` The guest writes it to `/dev/kmsg` at priority 2, to the portb data port -`0xe9` (through `/dev/port` or `outb`), and to the console. Consumers treat -identical lines as one event. The guest then writes the status to the -shutdown port `0x604` (directly, or through `nvx-exit`); its first byte -becomes the OpenVMM process exit status. +`0xe9` (through `/dev/port` or `outb`), and to the console, on every path: +violations are rare, and a diagnosable failure is worth the console bytes. +Consumers treat identical lines as one event. The guest then writes the +status to the shutdown port `0x604` (directly, or through `nvx-exit`); its +first byte becomes the OpenVMM process exit status. ### Snapshot agent @@ -1090,7 +1108,8 @@ builds the soft-lockup and hung-task detectors. Before the capture request: -1. Run the capture checks (`C6`, `C7`, `C10`). They do not wait. +1. Run the capture checks (`C6`, `C7`, `C10`). They do not wait and print + nothing. 2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. 3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and `/proc/sys/kernel/hung_task_timeout_secs`. @@ -1099,7 +1118,8 @@ Before the capture request: If the write returns without a restore (no destination or a rejected capture), the agent removes the barriers and then restores the values saved -in steps 2 and 3. Otherwise, in the restored process: +in steps 2 and 3. Otherwise, in the restored process, only these steps run +before the acknowledgement: 6. Read the status from `0xea`. Bit 1 is always set after a restore. 7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, @@ -1118,14 +1138,19 @@ in steps 2 and 3. Otherwise, in the restored process: 10. Run the existing entropy and identity repair for the tier. The RTC-based wall-clock refresh is removed; `instance-checkpoint` restores also get step 8. -11. Run the restore checks (`C1`, `C2`, and `C5` on newly onlined CPUs; `C3` - for CPU 0; `C6`; `C10`). They do not wait. -12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. -13. Signal the daemon to finish the restore off the latency path: wait for - the [grace period release](#rcu-grace-period-release); restore the - values saved in steps 2 and 3 (`G_REPAIR_SUPPRESSION`); run the deferred - `C7` check; print `NVX-TIME-ABI: v=1 phase=restore status=ok`; and restart - the discipline at the fast cadence. +11. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. +12. After the acknowledgement, or after step 10 when none is required, the + daemon finishes the restore asynchronously, and fail-fast: + - it runs the restore checks (`C1`, `C2`, and `C5` on newly onlined + CPUs; `C3` for CPU 0; `C6`; `C10`); + - it waits for the [grace period release](#rcu-grace-period-release); + - it restores the values saved in steps 2 and 3 + (`G_REPAIR_SUPPRESSION`); + - it runs the deferred `C7` check and records the restore check in the + state file; + - it restarts the discipline at the fast cadence. + + It prints nothing unless a check fails. Steps 6, 7, and 8 run in one helper process. Repair failures emit a violation event and power off with status 195. @@ -1163,12 +1188,9 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: - **Cadence.** A poll every 16 s for the first four polls after the boot check or a restore, then every 64 s. - **Sample.** Up to three time samples per poll; keep the first with - `epsilon <= 50 µs`. Otherwise skip the poll, count a rejected sample, set - `last_sample_error=G_SAMPLE_UNCERTAIN`, and print - `NVX-TIME-ABI: v=1 phase=runtime status=uncertain code=G_SAMPLE_UNCERTAIN - epsilon_ns=` on the console, with `epsilon_ns=none` when no - attempt produced a valid sample. The next accepted sample resets - `last_sample_error` to `none`. + `epsilon <= 50 µs`. Otherwise skip the poll, count a rejected sample, and + set `last_sample_error=G_SAMPLE_UNCERTAIN` in the state file; nothing is + printed. The next accepted sample resets `last_sample_error` to `none`. A sample whose `g` differs from the recorded `g` is discarded: a restore happened, and restore repair resets the discipline. - **Step.** If `|theta| >= 128 ms`, apply `ADJ_SETOFFSET | ADJ_NANO` with @@ -1186,8 +1208,9 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. -**Discontinuity state.** The daemon publishes `/run/nvx/time/state`, replaced -atomically with `rename(2)` after every change. The sandbox agent bind-mounts +**State file.** The time component publishes `/run/nvx/time/state` +from the start of the boot checks, replaced atomically with `rename(2)` after +every change, and `nvx-time status` reads it. The sandbox agent bind-mounts `/run/nvx/time` read-only into the container at the same path. The file holds `key=value` lines: @@ -1195,6 +1218,11 @@ atomically with `rename(2)` after every change. The sandbox agent bind-mounts | --- | --- | | `version` | 1 | | `generation` | `g` | +| `check_phase` | `boot`, `capture`, or `restore`: the last conformance check | +| `check_status` | `pending` while it runs, then `ok`; a failed check powers the guest off | +| `check_cpus` | Online CPUs the check covered | +| `check_elapsed_us` | Its duration, with the daemon start at boot | +| `tsc_hz`, `lapic_hz` | `F` and `L` | | `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | | `last_discontinuity` | `none`, `restore`, or `step` | | `last_step_ns` | Signed size of the last step | @@ -1269,7 +1297,12 @@ signals, and 255): | 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | `G_SAMPLE_UNCERTAIN` is the only non-fatal guest code: the discipline records -it and continues (see [Uncertainty bounds](#uncertainty-bounds)). +it and continues (see [Uncertainty bounds](#uncertainty-bounds)). Every fatal +guest code prints its violation event before the power-off, on production +paths too. No other time ABI line reaches the console unless a test runs +`nvx-time status` or `nvx-time exhaustive`. A boot or restore check that +fails after shell-ready or after the acknowledgement still powers off with +193, while the workload may be running. A workload can exit with any 8-bit status, so a harness classifies a time failure by the status together with its `NVX-TIME-ABI-VIOLATION` event. @@ -1399,9 +1432,10 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Added cold-boot cost, reported as the first run's `elapsed_us`; budget 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP; the gate is authoritative | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their `elapsed_us`, budgeted at 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| Restore repair and checks before the acknowledgement | Run in one helper process; the RCU release and deferred checks run after the acknowledgement | +| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` run asynchronously after the acknowledgement | +| No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. @@ -1412,9 +1446,9 @@ profile records three exclusive time ABI restore phases: and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that selection to the arrival of the `0x605` acknowledgement). The `restore.guest_repair_gate` milestone still spans from the VP release to the -release of the acknowledgement boundary. The guest reports its own repair -checks as the restore marker's `elapsed_us`, which covers ungated restores -too. +release of the acknowledgement boundary. The guest's restore checks run +after the acknowledgement, outside these phases; `nvx-time status` reports +their duration as `elapsed_us`. ## Test matrix @@ -1430,7 +1464,11 @@ parser; four-byte portb reads; and the generation counter. **Conformance.** The boot and restore checks pass at 1, 2, 4, and 8 vCPUs on all 18 registered hosts, plus the exhaustive CI check and the warp probe on -every backend. The fleet runs them on the hosts our SSH account can use, +every backend. Production paths print no marker, so CI scenarios and the +matrix driver run `/sbin/nvx-time status` over the console after shell-ready +and after every restore, and require its `phase=boot` or `phase=restore` +marker with `status=ok` and the expected `generation`. The fleet runs them on +the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on the other KVM and MSHV runners, which only CI jobs exercise. On the Azure WHP @@ -1439,7 +1477,9 @@ keeps `tsc` as its clocksource (#292: 6 of 6 runs with the time ABI, against 0 of 6 before it, which fell back from `tsc-early` to `refined-jiffies`). **Restore matrix.** Every case runs with zero violations; rejected cases must -fail with the listed code. Per-PR CI runs the same-host cases on one runner; +fail with the listed code. A restored case's evidence is the +`nvx-time status` output after the restore, which waits for the deferred +checks. Per-PR CI runs the same-host cases on one runner; the full matrix, including the cross-VM cases, runs on the hosts our account can use as the fleet restore matrix. @@ -1451,7 +1491,7 @@ can use as the fleet restore matrix. | Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | -| Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored, with `G_SAMPLE_UNCERTAIN` recorded and the guest running; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | +| Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | | Across VMs of one generation | `azure-windows-1` to `-2`; `azure-windows-3` to `-4`. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | `azure-windows-1` (8370C) to `-3` (8573C); prometheus32 to `azure-kvm-5` (KVM); prometheus30 to `azure-azlinux-5` (MSHV) | `E_CPU_GENERATION` | | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | From be570a39c53d04a3dc3396d49a4f073820bad202 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 15:09:05 -0700 Subject: [PATCH 048/188] doc: run the deferred time ABI work at SCHED_IDLE in one helper Following the guest agent's restore-latency request on top of 1efcd28: - Say explicitly that no restore check runs before the acknowledgement, and list the post-acknowledgement checks: C1, C2, C3, C5, C6, C7, and C10. - Let the helper that runs steps 6 to 8 run every other step before the acknowledgement and signal the daemon, so that a restore without shell work needs no second helper. - Run the deferred restore work and the asynchronous boot checks at SCHED_IDLE. The watcher and the discipline stay at normal priority. - Allow rep insl for the packet read. OpenVMM serves string port reads on every backend: KVM in one exit, MSHV and WHP through the instruction emulator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 98435635e..756337496 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -972,10 +972,10 @@ mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the [uncertainty bound](#uncertainty-bounds) and steps the clock to host UTC (`C12`), so the workload starts on host time; this step is not counted as a discontinuity. Init then reports shell-ready (or starts the workload, in -modes without a shell) and runs every other boot check asynchronously. The -checks start the daemon when they pass. Every check stays fail-fast: a -failure powers the guest off with status 193, even if the workload is -running. A guest that powers off before its checks finish skips them, which +modes without a shell) and runs every other boot check asynchronously, at +`SCHED_IDLE`. The checks start the daemon when they pass. Every check stays +fail-fast: a failure powers the guest off with status 193, even if the +workload is running. A guest that powers off before its checks finish skips them, which is why CI takes its evidence from `nvx-time status`, which waits for them. CPUID is executed on every online CPU (the checker pins itself to each CPU in @@ -1123,7 +1123,9 @@ before the acknowledgement: 6. Read the status from `0xea`. Bit 1 is always set after a restore. 7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, - and read the packet with four-byte reads. Validate the magic, version, + and read the packet with four-byte reads: `inl`, or `rep insl`, which + KVM serves in one exit and MSHV and WHP through OpenVMM's instruction + emulator. Validate the magic, version, reserved bits, counts, `g`, and the generation ID (`G_REPAIR_PACKET`, `G_REPAIR_GENERATION`). 8. Set the wall clock. Compute `theta` and `epsilon` from the bracket. If @@ -1152,7 +1154,16 @@ before the acknowledgement: It prints nothing unless a check fails. -Steps 6, 7, and 8 run in one helper process. Repair failures emit a violation +No restore check runs before the acknowledgement. The post-acknowledgement +checks are `C1`, `C2`, `C3`, `C5`, `C6`, `C7`, and `C10`. Steps 6, 7, and 8 +run in one helper process, which may also run every other step before the +acknowledgement and signal the daemon, so that a restore without shell work +(for example, untiered with no processor or memory target) needs no second +helper. Step 12 and the asynchronous boot checks run at `SCHED_IDLE`, so on +few vCPUs they never take a CPU from the restored or starting workload; the +watcher and the discipline run at normal priority. Under a CPU-bound +workload they still progress, more slowly, and stall suppression stays set +until the release. Repair failures emit a violation event and power off with status 195. ### RCU grace period release From 73f0dc8437dba7c8bb0a1a6acc5b76336457f398 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 15:49:50 -0700 Subject: [PATCH 049/188] doc: check the restored timers where restore sets them Restore step 11 only freezes the VP set: the PIT's restore in step 10 already rejects a periodic channel 0, and step 14's LAPIC pass, which sets every VP's timer state anyway, rejects an armed periodic or TSC-deadline timer before any VP runs. OpenVMM therefore takes no separate pass over the VPs to check the timers. Name the restore clock phases that OpenVMM logs, so the restore.time_abi_clock attribution can be broken down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 756337496..0af8b8747 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -669,12 +669,11 @@ In the worker, with every VP stopped: 10. Restore every state unit while stopped: VM time, chipset and virtio devices, the time platform (`HV_X64_MSR_TSC_INVARIANT_CONTROL`), the partition, and every VP. -11. Assert the saved timers: no armed periodic or TSC-deadline LAPIC timer on - any VP (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`), and PIT channel 0 not - counting in a periodic mode (`E_PIT_ACTIVE`). Every instantiated VP was - bound before step 10 restored its state, and MSHV creates a VP only when - it is bound, so the instantiated set is complete. Freeze it; creating a - VP after this point is an internal error (`E_VP_LATE_CREATION`). +11. Freeze the instantiated VP set. Every instantiated VP was bound before + step 10 restored its state, and MSHV creates a VP only when it is bound, + so the set is complete; creating a VP after this point is an internal + error (`E_VP_LATE_CREATION`). The PIT's restore in step 10 already + rejected a channel 0 counting in a periodic mode (`E_PIT_ACTIVE`). 12. Synchronized TSC set. The backend takes the restore anchor (one host instant) and passes its host sample to the orchestrator, which selects the downtime source, computes `D` and checks its bounds, and returns @@ -702,7 +701,9 @@ In the worker, with every VP stopped: 14. Advance every VP's counting-mode LAPIC timer by `D` at `L`, and set every VP's LAPIC state again, always after step 12 and even when no timer is armed: KVM derives its timer deadline from the guest TSC when the LAPIC - state is set. + state is set. The same pass rejects an armed periodic or TSC-deadline + timer (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`) before any VP runs, + so restore takes no separate pass over the VPs to check them. 15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT catches up from its saved VM-time cursor; it is idle. 16. Seal the time fields of the restore packet: `D`, its source, the rate @@ -1452,7 +1453,9 @@ Expected wins are tracked separately and do not relax the gate. **Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle profile records three exclusive time ABI restore phases: -`restore.time_abi_clock` (restore steps 11 to 16), `restore.guest_resume` +`restore.time_abi_clock` (restore steps 12 to 16, whose phases OpenVMM also +logs as `time ABI restore clock` with `tsc_set_us`, `lapic_us`, +`vm_time_us`, and `total_us`), `restore.guest_resume` (from the VP release to the guest's first selection of the restore packet), and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that selection to the arrival of the `0x605` acknowledgement). The From 30f0d245b169755d68897f7f0be5e84adfadc993 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 12:48:00 -0700 Subject: [PATCH 050/188] scripts: qualify H4 and H6 on the spec's rate and warp schedules Spec 0d07017 ("Rate stability (H4)", "Warp schedules") finalizes both measured gates from the WHP results. H4: the host probe reads the TSC between two reads of a host clock, keeps the tightest of 64 brackets, and sleeps between samples so the host's CPUs idle: 13 samples 10 s apart in doctor, 3 samples 1 s apart in validate-runner. Each interval between consecutive samples must be conclusive within 0.25 ppm (half the brackets plus half the clock's resolution, over the interval), the interval rates must agree within 1 ppm, and the rate over the whole window must lie within 100 ppm of the backend's when H3 ran. Before, H4 compared two independent 1 s measurements against Rust's Instant, without an uncertainty bound. Two clocks: the interval stability is judged against a clock that time synchronization never steers, CLOCK_MONOTONIC_RAW on Linux and QueryPerformanceCounter on Windows, while the whole-window rate and its deviation from the backend's use the disciplined CLOCK_MONOTONIC on Linux. chrony's frequency updates move CLOCK_MONOTONIC's rate between seconds: on the Azure MSHV runners the short schedule's interval rates against it differed by up to 12.2 ppm (3 of 9 runs failed the 1 ppm bound) and on the KVM runners by up to 0.85 ppm, while against CLOCK_MONOTONIC_RAW the TSC agreed within 0.07 ppm over 30 one-second intervals on the same runners. The clock's resolution is the larger of the reported one and the smallest step between consecutive reads when two reads return the same value: Hyper-V's reference TSC page, the clocksource of MSHV roots and the Azure MSHV runners, advances the Linux clocks in 100 ns steps although clock_getres reports 1 ns. The detail and the job summary report the rate, its deviation, the agreement, the largest uncertainty, and the clocks. H6: the largest guest runs the warp probe five times with idle gaps of 0.1, 1, 5, and 1 s, and a 1-vCPU guest then runs it once, instead of two runs 1 s apart. warp_probe_script() and warp_rounds() take the gaps, and their defaults keep CI's schedule: two runs 1 s apart after every boot and restore. doctor runs both checks on the long schedules; --ci-schedule selects the short ones, which validate-runner passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 11 +- doc/ci.md | 40 +-- scripts/nvx_tools/doctor.py | 208 ++++++++++--- scripts/nvx_tools/host_time_probe.rs | 225 +++++++++++--- scripts/nvx_tools/time_abi.py | 48 +-- scripts/test_nvx_tools.py | 1 + scripts/test_time_abi.py | 340 +++++++++++++++++---- 7 files changed, 692 insertions(+), 181 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index ce9de285d..5cf0e156d 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -119,10 +119,11 @@ runs: # Time ABI host qualification (doc/design/time-abi.md, "Host # qualification"): the backend, the CPU fingerprint and generation, and - # the TSC rate stability. It replaces the nonstop_tsc check, which is now - # recorded as evidence only. Jobs qualify before they have an OpenVMM - # binary, so H2 does not run OpenVMM's CPU profile check. The guest warp - # probe runs in the microVM scenarios, which boot the time ABI. + # the TSC rate stability on its short CI schedule. It replaces the + # nonstop_tsc check, which is now recorded as evidence only. Jobs qualify + # before they have an OpenVMM binary, so H2 does not run OpenVMM's CPU + # profile check. The guest warp probe runs in the microVM scenarios, + # which boot the time ABI. - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash @@ -131,6 +132,7 @@ runs: --backend "${{ inputs.backend }}" --checks H1 H2 H4 --no-openvmm + --ci-schedule --summary "${GITHUB_STEP_SUMMARY}" - name: Qualify host time on Windows @@ -141,6 +143,7 @@ runs: --backend "${{ inputs.backend }}" --checks H1 H2 H4 --no-openvmm + --ci-schedule --summary "$env:GITHUB_STEP_SUMMARY" - name: Validate KVM diff --git a/doc/ci.md b/doc/ci.md index 6003cb988..03d5f13a8 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -56,14 +56,15 @@ exposes an invariant TSC. The control log is kept as failure; the restore still fails. Every job that uses the `validate-runner` action first qualifies its runner -for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm` (see -[Host qualification](#host-qualification)): the backend, the CPU fingerprint -and generation, and the TSC rate stability. The job has no OpenVMM binary yet, -so H2 skips OpenVMM's CPU profile check. This replaces the earlier -`nonstop_tsc` check and takes a few seconds. It reports the CPU generation, -the CPU profile that `auto` selects, and the measured TSC rate in the log and -the job summary, and fails the job with a stable code when the runner is not -qualified, for example `E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation. +for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm +--ci-schedule` (see [Host qualification](#host-qualification)): the backend, +the CPU fingerprint and generation, and the TSC rate stability on its short +schedule. The job has no OpenVMM binary yet, so H2 skips OpenVMM's CPU profile +check. This replaces the earlier `nonstop_tsc` check and takes a few seconds. +It reports the CPU generation, the CPU profile that `auto` selects, and the +measured TSC rate in the log and the job summary, and fails the job with a +stable code when the runner is not qualified, for example +`E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation. Qualification gates only on measured properties, alike on every backend: the host OS's invariant-TSC flags and clocksource are recorded as evidence, and the guest warp probe in the microVM scenarios measures the skew that a host @@ -173,18 +174,20 @@ the [time ABI](design/time-abi.md#host-qualification). It runs checks H1 to H7 in order, prints one `NVX-DOCTOR: check= status= detail="..."` line per check, and exits with status 1 if any check fails. `--checks` selects a subset, and `--summary` appends a Markdown table with the CPU generation, -profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. A -failure that matches a time ABI failure code starts its detail with the code in -brackets, for example `[E_PROFILE_HOST_UNKNOWN]`. +profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. H4 +and H6 run on the spec's long qualification schedules, which take about three +minutes; `--ci-schedule` selects CI's short ones. A failure that matches a time +ABI failure code starts its detail with the code in brackets, for example +`[E_PROFILE_HOST_UNKNOWN]`. | Check | Implementation | | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | | H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. Then `openvmm --hypervisor --cpu-fingerprint ` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line. H2 requires exit status 0, `status=pass`, the same backend and generation, and a revision of the generation's profile, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | -| H4 | Two 1 s measurements of the TSC against host monotonic time agree within 1 ppm, and lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | +| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, `CLOCK_MONOTONIC` on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | -| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker, and the idle-inducing warp schedule stays within 1,000 ns: two rounds of `nvx-time-probe warp` over every CPU pair with all vCPUs halted for 1 s between them, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265) | +| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | | H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows | H2, H4, and H5 use a dependency-free host probe, @@ -202,11 +205,12 @@ warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource are evidence only. In CI, `validate-runner` runs the cheap host-level checks H1, H2, and H4 before every job, with `--no-openvmm`, because jobs download -OpenVMM and the guest artifacts only later. The guest warp probe needs a time -ABI boot, so it belongs in the microVM boot and restore scenarios, which -assert its verdict once OpenVMM boots the time ABI by default. Without H3, H4 -checks the rate stability without comparing it against the backend's rate. H5 -and H7 remain available for interactive qualification. +OpenVMM and the guest artifacts only later, and with `--ci-schedule`. The +guest warp probe needs a time ABI boot, so it belongs in the microVM boot and +restore scenarios, which run CI's warp schedule after every boot and restore +and assert its verdict once OpenVMM boots the time ABI by default. Without H3, +H4 checks the rate stability without comparing it against the backend's rate. +H5 and H7 remain available for interactive qualification. ## Rust crate diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 25580552c..5b54a370c 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -12,6 +12,7 @@ import argparse import ctypes import hashlib +import itertools import os import platform import re @@ -27,12 +28,13 @@ from .ci import OPENVMM_TEST_BACKENDS from .common import ScriptError, artifact_path, openvmm_binary_path from .time_abi import ( + CI_WARP_GAPS, LAPIC_HZ, MAX_TSC_HZ, MIN_TSC_HZ, + QUALIFICATION_WARP_GAPS, WARP_BOUND_NS, WARP_PROBE_COMPLETION_MARKER, - WARP_PROBE_IDLE_SECONDS, CpuGeneration, TimeAbiFailure, TimeAbiMonitor, @@ -61,12 +63,28 @@ VERIFY_PREFIX = "NVX-TIME-ABI-VERIFY:" CPU_PROFILE_PREFIX = "NVX-CPU-PROFILE:" RUST_ESCAPE = re.compile(r"\\(u\{[0-9a-fA-F]{1,6}\}|.)") -RATE_MEASURE_MS = 1000 +# doc/design/time-abi.md, "Rate stability (H4)". +RATE_UNCERTAINTY_PPM = 0.25 RATE_AGREEMENT_PPM = 1.0 RATE_TOLERANCE_PPM = 100.0 SKEW_PAIR_DURATION_MS = 5 GUEST_VCPU_COUNTS = (8, 4, 2, 1) GUEST_MEMORY_MIB = 128 + + +@dataclass(frozen=True) +class Schedule: + """How long H4 samples the TSC rate and how often H6 runs the warp probe.""" + + rate_samples: int + rate_interval_ms: int + warp_gaps: tuple[str, ...] + + +# doc/design/time-abi.md, "Rate stability (H4)" and "Warp schedules": doctor +# qualifies on the long schedules, and CI on the short ones. +QUALIFICATION_SCHEDULE = Schedule(13, 10_000, QUALIFICATION_WARP_GAPS) +CI_SCHEDULE = Schedule(3, 1_000, CI_WARP_GAPS) # Qualification gates only on measured properties, alike on every backend: # the guest warp probe, the TSC rate stability, and the CPU profile. The host # OS's invariant-TSC flags and clocksource are recorded as evidence only, @@ -109,6 +127,7 @@ class DoctorContext: # Where H2 writes OpenVMM's CPU fingerprint; None skips the profile check # when qualification runs without OpenVMM. fingerprint: Path | None = None + schedule: Schedule = QUALIFICATION_SCHEDULE def host_is_windows() -> bool: @@ -590,31 +609,97 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: return CheckResult("H3", True, detail) +def _rate_points( + samples: Sequence[Mapping[str, str]], clock: Mapping[str, str], role: str +) -> list[tuple[int, int, float]]: + """Return each sample's (TSC, bracket midpoint in half clock ticks, + uncertainty in clock ticks: half the bracket plus half the resolution).""" + half_resolution = float(clock["resolution_ns"]) * int(clock["hz"]) / 2e9 + return [ + ( + int(sample[f"{role}_tsc"]), + 2 * int(sample[f"{role}_clock"]) + int(sample[f"{role}_bracket"]), + int(sample[f"{role}_bracket"]) / 2 + half_resolution, + ) + for sample in samples + ] + + +def _window_rate(points: Sequence[tuple[int, int, float]], clock_hz: int) -> float: + seconds = (points[-1][1] - points[0][1]) / (2 * clock_hz) + if seconds <= 0: + raise ScriptError("the host clock did not advance between samples") + return (points[-1][0] - points[0][0]) / seconds + + def check_rate(context: DoctorContext) -> CheckResult: clocksource = "" if not host_is_windows(): clocksource = host_clocksource() context.facts["host_clocksource"] = clocksource + schedule = context.schedule records = run_probe( - context, "rate", "--measure-ms", str(RATE_MEASURE_MS), "--count", "2" + context, + "rate", + "--samples", + str(schedule.rate_samples), + "--interval-ms", + str(schedule.rate_interval_ms), ) - rates = [float(fields["tsc_hz"]) for kind, fields in records if kind == "rate"] - if len(rates) != 2: - raise ScriptError(f"host time probe printed {len(rates)} rate records, not 2") - measured = sum(rates) / len(rates) - spread_ppm = (max(rates) - min(rates)) / min(rates) * 1e6 + clocks = {fields.get("role"): fields for kind, fields in records if kind == "clock"} + for role in ("rate", "stability"): + if role not in clocks: + raise ScriptError(f"host time probe printed no {role} clock") + samples = [fields for kind, fields in records if kind == "sample"] + if len(samples) != schedule.rate_samples: + raise ScriptError( + f"host time probe printed {len(samples)} samples, not " + f"{schedule.rate_samples}" + ) + # The rate clock, which time synchronization disciplines, gives the true + # rate. The stability clock, which it never steers, shows whether the TSC + # keeps its rate across idle: chrony's frequency updates move + # CLOCK_MONOTONIC's rate by several ppm between seconds. + rate_clock, stability_clock = clocks["rate"], clocks["stability"] + measured = _window_rate( + _rate_points(samples, rate_clock, "rate"), int(rate_clock["hz"]) + ) + clock_hz = int(stability_clock["hz"]) + points = _rate_points(samples, stability_clock, "stability") + rates: list[float] = [] + uncertainties_ppm: list[float] = [] + for (tsc0, mid0, error0), (tsc1, mid1, error1) in itertools.pairwise(points): + if mid1 <= mid0: + raise ScriptError("the host clock did not advance between samples") + seconds = (mid1 - mid0) / (2 * clock_hz) + rates.append((tsc1 - tsc0) / seconds) + uncertainties_ppm.append((error0 + error1) / clock_hz / seconds * 1e6) + stable_rate = _window_rate(points, clock_hz) + agreement_ppm = (max(rates) - min(rates)) / stable_rate * 1e6 + uncertainty_ppm = max(uncertainties_ppm) context.facts["measured_tsc_hz"] = f"{measured:.0f}" - context.facts["rate_spread_ppm"] = f"{spread_ppm:.3f}" + context.facts["rate_agreement_ppm"] = f"{agreement_ppm:.3f}" + context.facts["rate_uncertainty_ppm"] = f"{uncertainty_ppm:.3f}" detail = ( - f"tsc_hz={measured:.0f} two {RATE_MEASURE_MS} ms measurements differ by " - f"{spread_ppm:.3f} ppm" + f"tsc_hz={measured:.0f} against {rate_clock['name']} from {len(samples)} " + f"samples {schedule.rate_interval_ms / 1000:g} s apart; against " + f"{stability_clock['name']} (resolution " + f"{float(stability_clock['resolution_ns']):g} ns) the interval rates agree " + f"within {agreement_ppm:.3f} ppm, largest interval uncertainty " + f"{uncertainty_ppm:.3f} ppm" ) problems: list[str] = [] if not MIN_TSC_HZ <= measured <= MAX_TSC_HZ: problems.append(f"[E_TSC_RATE_IMPLAUSIBLE] measured {measured:.0f} Hz") - if spread_ppm > RATE_AGREEMENT_PPM: + if uncertainty_ppm > RATE_UNCERTAINTY_PPM: problems.append( - f"the TSC rate is unstable: {spread_ppm:.3f} ppm > {RATE_AGREEMENT_PPM} ppm" + f"the measurement is inconclusive: an interval's uncertainty is " + f"{uncertainty_ppm:.3f} ppm > {RATE_UNCERTAINTY_PPM} ppm" + ) + if agreement_ppm > RATE_AGREEMENT_PPM: + problems.append( + f"the TSC rate is unstable: interval rates differ by " + f"{agreement_ppm:.3f} ppm > {RATE_AGREEMENT_PPM} ppm" ) declared = context.facts.get("native_tsc_hz") or context.facts.get("tsc_hz") if declared is None: @@ -623,6 +708,7 @@ def check_rate(context: DoctorContext) -> CheckResult: detail += "; not compared with the backend's rate (H3 did not run)" else: deviation_ppm = abs(measured - int(declared)) / int(declared) * 1e6 + context.facts["rate_deviation_ppm"] = f"{deviation_ppm:.1f}" detail += f"; {deviation_ppm:.1f} ppm from the backend's {declared} Hz" if deviation_ppm > RATE_TOLERANCE_PPM: problems.append( @@ -664,15 +750,10 @@ def check_host_skew(context: DoctorContext) -> CheckResult: return CheckResult("H5", True, detail) -def check_guest_warp(context: DoctorContext) -> CheckResult: - for path, description in ( - (context.openvmm, "OpenVMM"), - (context.kernel, "guest kernel"), - (context.initrd, "guest initramfs"), - ): - if not path.is_file(): - return CheckResult("H6", False, f"{description} was not found at {path}") - vcpus = _guest_vcpus() +def _warp_guest( + context: DoctorContext, vcpus: int, gaps: Sequence[str] +) -> tuple[dict[str, str], list[dict[str, str]]]: + """Boot a guest, check its boot marker, and run a warp schedule in it.""" command = workload_boot_command( context.openvmm, context.backend, @@ -683,38 +764,66 @@ def check_guest_warp(context: DoctorContext) -> CheckResult: processors=vcpus, ) command.extend(context.openvmm_args) - try: - result = run_guest_script( - command, - warp_probe_script() + "nvx-exit 0\n", - WARP_PROBE_COMPLETION_MARKER, - timeout=context.timeout, - ) - monitor = TimeAbiMonitor(command) - monitor.feed(result["text"].encode()) - monitor.finish() - boot = monitor.require_boot("the guest boot marker", online_cpus=vcpus) - rounds = check_warp_probe( - result["text"], cpus=vcpus, context="H6", rounds=warp_rounds(vcpus) + result = run_guest_script( + command, + warp_probe_script(gaps) + "nvx-exit 0\n", + WARP_PROBE_COMPLETION_MARKER, + timeout=context.timeout, + ) + monitor = TimeAbiMonitor(command) + monitor.feed(result["text"].encode()) + monitor.finish() + boot = monitor.require_boot("the guest boot marker", online_cpus=vcpus) + rounds = check_warp_probe( + result["text"], cpus=vcpus, context="H6", rounds=warp_rounds(vcpus, gaps) + ) + return boot, rounds + + +def check_guest_warp(context: DoctorContext) -> CheckResult: + for path, description in ( + (context.openvmm, "OpenVMM"), + (context.kernel, "guest kernel"), + (context.initrd, "guest initramfs"), + ): + if not path.is_file(): + return CheckResult("H6", False, f"{description} was not found at {path}") + vcpus = _guest_vcpus() + # The largest guest runs the schedule, then a 1-vCPU guest runs the + # probe once. + guests: list[tuple[int, tuple[str, ...]]] = [] + if vcpus > 1: + guests.append((vcpus, context.schedule.warp_gaps)) + guests.append((1, ())) + boots: list[dict[str, str]] = [] + rounds: list[dict[str, str]] = [] + runs: list[str] = [] + for count, count_gaps in guests: + try: + boot, count_rounds = _warp_guest(context, count, count_gaps) + except (RuntimeError, TimeAbiFailure) as error: + first = str(error).splitlines()[0] + return CheckResult("H6", False, f"vcpus={count}: {first}") + boots.append(boot) + rounds.extend(count_rounds) + runs.append( + f"vcpus={count} rounds={len(count_rounds)} " + f"idle_gaps_s={','.join(count_gaps) or 'none'} " + f"boot_elapsed_us={boot.get('elapsed_us', '?')}" ) - except (RuntimeError, TimeAbiFailure) as error: - first = str(error).splitlines()[0] - return CheckResult("H6", False, f"vcpus={vcpus}: {first}") def worst(name: str) -> int: return max(int(warp[name]) for warp in rounds) context.facts["guest_warp_ns"] = str(worst("max_abs_offset_ns")) - context.facts.setdefault("tsc_hz", boot["tsc_hz"]) - context.facts.setdefault("lapic_hz", boot["lapic_hz"]) + context.facts.setdefault("tsc_hz", boots[0]["tsc_hz"]) + context.facts.setdefault("lapic_hz", boots[0]["lapic_hz"]) return CheckResult( "H6", True, - f"vcpus={vcpus} rounds={len(rounds)} idle_s={WARP_PROBE_IDLE_SECONDS} " - f"max_backward_ns={worst('max_backward_ns')} " + "; ".join(runs) + f"; max_backward_ns={worst('max_backward_ns')} " f"max_abs_offset_ns={worst('max_abs_offset_ns')} " - f"max_uncertainty_ns={worst('max_uncertainty_ns')} boot_elapsed_us=" - f"{boot.get('elapsed_us', '?')}", + f"max_uncertainty_ns={worst('max_uncertainty_ns')}", ) @@ -835,6 +944,9 @@ def summary_markdown( ("tsc_hz", "Declared TSC rate (Hz)"), ("lapic_hz", "LAPIC rate (Hz)"), ("measured_tsc_hz", "Measured TSC rate (Hz)"), + ("rate_deviation_ppm", "Rate deviation from the backend (ppm)"), + ("rate_agreement_ppm", "Interval rate agreement (ppm)"), + ("rate_uncertainty_ppm", "Largest interval uncertainty (ppm)"), ("host_skew_ns", "Host skew (ns)"), ("guest_warp_ns", "Guest warp offset (ns)"), ("invariant_tsc", "Host invariant TSC (evidence)"), @@ -873,6 +985,7 @@ def run(args: argparse.Namespace) -> int: probe_directory=probe_directory, timeout=args.timeout, fingerprint=fingerprint, + schedule=CI_SCHEDULE if args.ci_schedule else QUALIFICATION_SCHEDULE, ) results = run_checks(context, checks) failed = [result.check for result in results if not result.passed] @@ -929,6 +1042,13 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: help="qualify without an OpenVMM binary: H2 checks the CPU identity and " "generation but not the CPU profile, and H3 and H6 cannot run", ) + parser.add_argument( + "--ci-schedule", + action="store_true", + help="run H4 and H6 on CI's short schedules: H4 takes 3 samples 1 s " + "apart instead of 13 samples 10 s apart, and H6 runs the warp probe " + "twice instead of five times", + ) parser.add_argument( "--probe-dir", type=Path, diff --git a/scripts/nvx_tools/host_time_probe.rs b/scripts/nvx_tools/host_time_probe.rs index 80d013cf4..a92102530 100644 --- a/scripts/nvx_tools/host_time_probe.rs +++ b/scripts/nvx_tools/host_time_probe.rs @@ -2,10 +2,10 @@ //! //! `nvx.py doctor` builds this file with `rustc` and runs it for the host //! checks of doc/design/time-abi.md: H2 (the CPU identity and the host's -//! invariant-TSC bit), H4 (the TSC rate against host monotonic time), and H5 -//! (cross-CPU TSC skew between pinned threads). It has no dependencies and -//! prints `NVX-HOST-TIME-PROBE` lines of `key=value` fields; the doctor -//! applies the bounds. +//! invariant-TSC bit), H4 (sleep-separated samples of the TSC against the +//! host's monotonic clocks), and H5 (cross-CPU TSC skew between pinned +//! threads). It has no dependencies and prints `NVX-HOST-TIME-PROBE` lines of +//! `key=value` fields; the doctor applies the bounds. use std::process::ExitCode; use std::sync::Arc; @@ -123,6 +123,121 @@ mod affinity { } } +/// The host clocks that H4 compares the TSC with: the rate clock, which time +/// synchronization disciplines to the true rate, and the stability clock, +/// which it never steers. chrony's frequency updates move CLOCK_MONOTONIC's +/// rate by several ppm between seconds, which says nothing about the TSC. +#[cfg(target_os = "linux")] +mod clock { + #[repr(C)] + struct Timespec { + tv_sec: i64, + tv_nsec: i64, + } + + unsafe extern "C" { + fn clock_gettime(clock: i32, time: *mut Timespec) -> i32; + fn clock_getres(clock: i32, time: *mut Timespec) -> i32; + } + + #[derive(Clone, Copy)] + pub struct Clock { + pub name: &'static str, + id: i32, + } + + pub const RATE: Clock = Clock { + name: "CLOCK_MONOTONIC", + id: 1, + }; + pub const STABILITY: Clock = Clock { + name: "CLOCK_MONOTONIC_RAW", + id: 4, + }; + + impl Clock { + /// Returns the clock's ticks per second. + pub fn hz(self) -> Result { + Ok(1_000_000_000) + } + + /// Returns the clock's reported resolution in nanoseconds. + pub fn resolution_ns(self) -> Result { + let mut time = Timespec { tv_sec: 0, tv_nsec: 0 }; + // SAFETY: time points to a valid timespec. + if unsafe { clock_getres(self.id, &mut time) } != 0 { + return Err(format!( + "clock_getres({}) failed: {}", + self.name, + std::io::Error::last_os_error() + )); + } + Ok(time.tv_sec as f64 * 1e9 + time.tv_nsec as f64) + } + + /// Returns the clock in ticks. + #[inline(always)] + pub fn now(self) -> u64 { + let mut time = Timespec { tv_sec: 0, tv_nsec: 0 }; + // SAFETY: time points to a valid timespec, and both clocks always + // exist. + unsafe { clock_gettime(self.id, &mut time) }; + time.tv_sec as u64 * 1_000_000_000 + time.tv_nsec as u64 + } + } +} + +/// The host clock that H4 compares the TSC with. Time synchronization never +/// steers QueryPerformanceCounter, so it serves both roles. +#[cfg(windows)] +mod clock { + #[link(name = "kernel32")] + unsafe extern "system" { + fn QueryPerformanceCounter(count: *mut i64) -> i32; + fn QueryPerformanceFrequency(frequency: *mut i64) -> i32; + } + + #[derive(Clone, Copy)] + pub struct Clock { + pub name: &'static str, + } + + pub const RATE: Clock = Clock { + name: "QueryPerformanceCounter", + }; + pub const STABILITY: Clock = RATE; + + impl Clock { + /// Returns the clock's ticks per second. + pub fn hz(self) -> Result { + let mut frequency = 0i64; + // SAFETY: frequency points to a valid i64. + if unsafe { QueryPerformanceFrequency(&mut frequency) } == 0 || frequency <= 0 { + return Err(format!( + "QueryPerformanceFrequency failed: {}", + std::io::Error::last_os_error() + )); + } + Ok(frequency as u64) + } + + /// Returns the clock's reported resolution in nanoseconds: one tick. + pub fn resolution_ns(self) -> Result { + Ok(1e9 / self.hz()? as f64) + } + + /// Returns the clock in ticks. + #[inline(always)] + pub fn now(self) -> u64 { + let mut count = 0i64; + // SAFETY: count points to a valid i64; the call cannot fail on + // Windows XP and later. + unsafe { QueryPerformanceCounter(&mut count) }; + count as u64 + } + } +} + fn option<'a>(arguments: &'a [String], name: &str) -> Result, String> { let mut value = None; let mut index = 0; @@ -242,43 +357,84 @@ fn cpu() -> Result<(), String> { Ok(()) } -/// Pairs a TSC reading with host monotonic time; returns the narrowest of -/// several brackets as (TSC at the midpoint, instant, bracket width). -fn paired_sample() -> (u64, Instant, u64) { - let mut best: Option<(u64, Instant, u64)> = None; +/// Reads the TSC between two reads of `clock` and returns the tightest of 64 +/// such brackets as (TSC, clock ticks before the read, clock ticks between +/// the two clock reads). +fn clock_sample(clock: clock::Clock) -> (u64, u64, u64) { + let mut best: Option<(u64, u64, u64)> = None; for _ in 0..64 { - let before = tsc(); - let now = Instant::now(); - let after = tsc(); - let width = after.wrapping_sub(before); - if best.is_none_or(|(_, _, narrowest)| width < narrowest) { - best = Some((before.wrapping_add(width / 2), now, width)); + let before = clock.now(); + let tsc = tsc(); + let after = clock.now(); + let bracket = after.saturating_sub(before); + if best.is_none_or(|(_, _, narrowest)| bracket < narrowest) { + best = Some((tsc, before, bracket)); } } best.expect("at least one sample") } +/// Measures `clock` with consecutive reads and returns the smallest positive +/// step between them in clock ticks, and whether two reads ever returned the +/// same value. Such a clock is coarser than one read, and its smallest step +/// is its resolution, although it may report a finer one: Hyper-V's +/// reference TSC page clocksource advances in 100 ns steps while +/// `clock_getres` reports 1 ns. A clock that resolves every read keeps its +/// reported resolution; its smallest step is the time one read takes. +fn clock_step(clock: clock::Clock) -> Result<(u64, bool), String> { + let mut smallest = u64::MAX; + let mut repeated = false; + let mut previous = clock.now(); + for _ in 0..100_000 { + let now = clock.now(); + let step = now.saturating_sub(previous); + if step == 0 { + repeated = true; + } else if step < smallest { + smallest = step; + } + previous = now; + } + if smallest == u64::MAX { + return Err(format!("{} did not advance", clock.name)); + } + Ok((smallest, repeated)) +} + fn rate(arguments: &[String]) -> Result<(), String> { - check_options(arguments, &["--measure-ms", "--count"])?; - let measure_ms: u64 = number(arguments, "--measure-ms", 1000)?; - let count: u32 = number(arguments, "--count", 2)?; - if measure_ms == 0 || count == 0 { - return Err("--measure-ms and --count must be positive".into()); + check_options(arguments, &["--samples", "--interval-ms"])?; + let samples: u32 = number(arguments, "--samples", 3)?; + let interval_ms: u64 = number(arguments, "--interval-ms", 1000)?; + if samples < 2 || interval_ms == 0 { + return Err("--samples must be at least 2 and --interval-ms positive".into()); } - // One CPU's TSC against host monotonic time; cross-CPU skew is H5's. + // One CPU's TSC against the host clocks; cross-CPU skew is H5's. The + // sleeps between samples let the host's CPUs idle. let first = *affinity::allowed()?.first().ok_or("no CPU is allowed")?; affinity::pin(first)?; - for index in 1..=count { - let (start_tsc, start, start_width) = paired_sample(); - thread::sleep(Duration::from_millis(measure_ms)); - let (end_tsc, end, end_width) = paired_sample(); - let elapsed_ns = end.duration_since(start).as_nanos(); - let cycles = end_tsc.wrapping_sub(start_tsc); - let hz = cycles as f64 * 1e9 / elapsed_ns as f64; + for (role, clock) in [("rate", clock::RATE), ("stability", clock::STABILITY)] { + let hz = clock.hz()?; + let reported_ns = clock.resolution_ns()?; + let (step, repeated) = clock_step(clock)?; + let step_ns = step as f64 * 1e9 / hz as f64; + let resolution_ns = if repeated { reported_ns.max(step_ns) } else { reported_ns }; + println!( + "{PREFIX} clock role={role} name={} hz={hz} resolution_ns={resolution_ns:.3} \ + reported_resolution_ns={reported_ns:.3} step_ns={step_ns:.3} repeated={}", + clock.name, + u8::from(repeated), + ); + } + for index in 0..samples { + if index > 0 { + thread::sleep(Duration::from_millis(interval_ms)); + } + let rate = clock_sample(clock::RATE); + let stability = clock_sample(clock::STABILITY); println!( - "{PREFIX} rate index={index} cpu={first} tsc_hz={hz:.3} elapsed_ns={elapsed_ns} \ - cycles={cycles} bracket_cycles={}", - start_width.max(end_width), + "{PREFIX} sample index={index} cpu={first} rate_tsc={} rate_clock={} rate_bracket={} \ + stability_tsc={} stability_clock={} stability_bracket={}", + rate.0, rate.1, rate.2, stability.0, stability.1, stability.2, ); } Ok(()) @@ -433,10 +589,11 @@ fn skew(arguments: &[String]) -> Result<(), String> { .map_err(|_| format!("--tsc-hz has an invalid value {text:?}"))?, None => { affinity::pin(cpus[0])?; - let (start_tsc, start, _) = paired_sample(); + let (start_tsc, start, _) = clock_sample(clock::RATE); thread::sleep(Duration::from_millis(100)); - let (end_tsc, end, _) = paired_sample(); - end_tsc.wrapping_sub(start_tsc) as f64 * 1e9 / end.duration_since(start).as_nanos() as f64 + let (end_tsc, end, _) = clock_sample(clock::RATE); + end_tsc.wrapping_sub(start_tsc) as f64 * clock::RATE.hz()? as f64 + / end.saturating_sub(start).max(1) as f64 } }; let to_ns = |cycles: f64| cycles * 1e9 / tsc_hz; @@ -484,7 +641,7 @@ fn skew(arguments: &[String]) -> Result<(), String> { fn usage() -> String { format!( - "usage: {} cpu\n {} rate [--measure-ms N] [--count K]\n \ + "usage: {} cpu\n {} rate [--samples N] [--interval-ms MS]\n \ {} skew [--duration-ms N] [--cpus LIST] [--tsc-hz HZ] [--bound-ns N]", "nvx-host-time-probe", "nvx-host-time-probe", "nvx-host-time-probe" ) diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 32ed42f68..bb57a53cf 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -9,6 +9,7 @@ from __future__ import annotations +import re from collections.abc import Mapping, Sequence from dataclasses import dataclass @@ -42,6 +43,12 @@ # The idle-inducing warp schedule: probe rounds separated by halted vCPUs. WARP_PROBE_ROUNDS = 2 WARP_PROBE_IDLE_SECONDS = 1 +# Idle gaps between probe rounds, in seconds (doc/design/time-abi.md, "Warp +# schedules"): CI's schedule after boot and every restore, and H6's. +CI_WARP_GAPS: tuple[str, ...] = (str(WARP_PROBE_IDLE_SECONDS),) * ( + WARP_PROBE_ROUNDS - 1 +) +QUALIFICATION_WARP_GAPS: tuple[str, ...] = ("0.1", "1", "5", "1") PROFILE_VENDORS: Mapping[str, str] = {"GenuineIntel": "intel"} @@ -371,33 +378,39 @@ def warp_probe_command(*, cpus: str | None = None) -> str: return command if cpus is None else f"{command} --cpus {cpus}" -def warp_rounds(cpus: int) -> int: - """Return how many probe rounds the warp schedule runs on ``cpus`` CPUs.""" - return 1 if cpus < 2 else WARP_PROBE_ROUNDS +def warp_rounds(cpus: int, gaps: Sequence[str] = CI_WARP_GAPS) -> int: + """Return how many probe rounds a warp schedule runs on ``cpus`` CPUs.""" + return 1 if cpus < 2 else len(gaps) + 1 -def warp_probe_script() -> str: - """Return a guest shell fragment that runs the idle-inducing warp schedule. +def warp_probe_script(gaps: Sequence[str] = CI_WARP_GAPS) -> str: + """Return a guest shell fragment that runs an idle-inducing warp schedule. - The probe runs over every online CPU in WARP_PROBE_ROUNDS rounds with all - vCPUs halted for WARP_PROBE_IDLE_SECONDS between rounds: a host without an - invariant TSC corrects guest TSC timing when an idle host CPU wakes, which - busy probe rounds alone never trigger (#265). The CPU list is read when - the fragment runs, so one post-restore script serves every restore target. - A failed round prints WARP_PROBE_FAILURE_MARKER and powers the guest off - with WARP_PROBE_FAILURE_STATUS; otherwise the fragment prints + The probe runs over every online CPU, and again after each idle gap in + ``gaps`` (seconds), during which every vCPU halts: a host without an + invariant TSC corrects guest TSC timing when an idle host CPU wakes, + which busy probe rounds alone never trigger (#265). A single-CPU guest + runs it once, because it has no CPU pair to compare across a gap. The + CPU list is read when the fragment runs, so one post-restore script + serves every restore target. A failed round prints + WARP_PROBE_FAILURE_MARKER and powers the guest off with + WARP_PROBE_FAILURE_STATUS; otherwise the fragment prints WARP_PROBE_COMPLETION_MARKER. """ + if any(not re.fullmatch(r"[0-9]+(\.[0-9]+)?", gap) for gap in gaps): + raise ValueError(f"invalid warp probe idle gaps: {gaps!r}") probe = warp_probe_command(cpus='"$warp_cpus"') return ( 'warp_cpus="$(cat /sys/devices/system/cpu/online)"\n' - f"warp_rounds={WARP_PROBE_ROUNDS}\n" + f'warp_gaps="{" ".join(gaps)}"\n' 'case "$warp_cpus" in\n' " *[-,]*) ;;\n" - " *) warp_rounds=1 ;;\n" + " *) warp_gaps= ;;\n" "esac\n" - "warp_round=1\n" - "while :; do\n" + "warp_round=0\n" + "for warp_gap in 0 $warp_gaps; do\n" + ' [ "$warp_round" -eq 0 ] || sleep "$warp_gap"\n' + " warp_round=$((warp_round + 1))\n" " warp_status=0\n" f" {probe} || warp_status=$?\n" ' if [ "$warp_status" -ne 0 ]; then\n' @@ -406,9 +419,6 @@ def warp_probe_script() -> str: f" nvx-exit {WARP_PROBE_FAILURE_STATUS}\n" f" exit {WARP_PROBE_FAILURE_STATUS}\n" " fi\n" - ' [ "$warp_round" -lt "$warp_rounds" ] || break\n' - f" sleep {WARP_PROBE_IDLE_SECONDS}\n" - " warp_round=$((warp_round + 1))\n" "done\n" f"echo {WARP_PROBE_COMPLETION_MARKER.decode()}\n" ) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 99a5725fb..aac9fd9a2 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3215,6 +3215,7 @@ def test_runners_qualify_their_host_time_before_every_job(self): self.assertIn('--backend "${{ inputs.backend }}"', step) self.assertIn("--checks H1 H2 H4\n", step) self.assertIn("--no-openvmm\n", step) + self.assertIn("--ci-schedule\n", step) self.assertIn(summary, step) args = nvx.parse_args( ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "--no-openvmm"] diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 9b591fece..d82437fca 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -304,6 +304,16 @@ def test_counts_rounds_and_names_the_failing_round(self): [time_abi.warp_rounds(cpus) for cpus in (1, 2, 8)], [1, time_abi.WARP_PROBE_ROUNDS, time_abi.WARP_PROBE_ROUNDS], ) + self.assertEqual(time_abi.CI_WARP_GAPS, ("1",)) + self.assertEqual( + [ + time_abi.warp_rounds(cpus, time_abi.QUALIFICATION_WARP_GAPS) + for cpus in (1, 8) + ], + [1, 5], + ) + with self.assertRaisesRegex(ValueError, "invalid warp probe idle gaps"): + time_abi.warp_probe_script(("1; reboot",)) two = warp_output() + warp_output(offset=30) results = time_abi.check_warp_probe(two, cpus=4, context="boot", rounds=2) self.assertEqual( @@ -332,6 +342,32 @@ def test_warp_fragment_idles_between_rounds_and_powers_off_on_failure(self): .replace(time_abi.WARP_PROBE_PATH, "probe") .replace("nvx-exit", "nvx_exit") ) + + def run_fragment( + fragment: str, online: str, failing_round: int = 0 + ) -> subprocess.CompletedProcess[str]: + assert shell is not None + return subprocess.run( + [shell], + input=( + f"cat() {{ printf '%s\\n' '{online}'; }}\n" + "calls=0\n" + "probe() {\n" + " calls=$((calls + 1))\n" + ' printf "probe %s\\n" "$*"\n' + f' [ "$calls" -ne {failing_round} ]\n' + "}\n" + 'sleep() { printf "SLEEP %s\\n" "$1"; }\n' + 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' + + fragment + + "echo after\n" + ), + text=True, + capture_output=True, + timeout=10, + check=False, + ) + for online, failing_round, calls, sleeps in ( ("0-3", 0, 2, 1), ("0-3", 2, 2, 1), @@ -339,26 +375,7 @@ def test_warp_fragment_idles_between_rounds_and_powers_off_on_failure(self): ("0", 0, 1, 0), ): with self.subTest(online=online, failing_round=failing_round): - result = subprocess.run( - [shell], - input=( - f"cat() {{ printf '%s\\n' '{online}'; }}\n" - "calls=0\n" - "probe() {\n" - " calls=$((calls + 1))\n" - ' printf "probe %s\\n" "$*"\n' - f' [ "$calls" -ne {failing_round} ]\n' - "}\n" - 'sleep() { printf "SLEEP %s\\n" "$1"; }\n' - 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' - + fragment - + "echo after\n" - ), - text=True, - capture_output=True, - timeout=10, - check=False, - ) + result = run_fragment(fragment, online, failing_round) lines = result.stdout.splitlines() self.assertEqual( lines.count(f"probe warp --bound-ns 1000 --cpus {online}"), @@ -380,6 +397,36 @@ def test_warp_fragment_idles_between_rounds_and_powers_off_on_failure(self): self.assertEqual(result.returncode, 0, result.stderr) self.assertIn("NVX-WARP-PROBE-OK", lines) self.assertIn("after", lines) + # H6's qualification schedule idles for each gap in order. + qualification = fragment.replace( + 'warp_gaps="1"', + f'warp_gaps="{" ".join(time_abi.QUALIFICATION_WARP_GAPS)}"', + ) + self.assertEqual( + qualification, + time_abi.warp_probe_script(time_abi.QUALIFICATION_WARP_GAPS) + .replace(time_abi.WARP_PROBE_PATH, "probe") + .replace("nvx-exit", "nvx_exit"), + ) + result = run_fragment(qualification, "0-7") + probe = "probe warp --bound-ns 1000 --cpus 0-7" + self.assertEqual( + result.stdout.splitlines(), + [ + probe, + "SLEEP 0.1", + probe, + "SLEEP 1", + probe, + "SLEEP 5", + probe, + "SLEEP 1", + probe, + "NVX-WARP-PROBE-OK", + "after", + ], + result.stderr, + ) VIOLATION = ( @@ -949,12 +996,40 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: self.assertFalse(missing.passed) self.assertIn("was not found", missing.detail) - def test_rate_check_gates_on_stability_and_the_backend_rate_only(self): - def records(first: float, second: float): - return [ - ("rate", {"tsc_hz": f"{first:.3f}"}), - ("rate", {"tsc_hz": f"{second:.3f}"}), - ] + def test_rate_check_applies_the_spec_bounds_to_sleep_separated_samples(self): + def records( + rates: list[float], + *, + disciplined: list[float] | None = None, + interval_s: float = 10.0, + bracket: int = 40, + clock_hz: int = 1_000_000_000, + resolution_ns: float = 1.0, + names: tuple[str, str] = ("CLOCK_MONOTONIC", "CLOCK_MONOTONIC_RAW"), + ) -> list[tuple[str, dict[str, str]]]: + """Probe records whose consecutive samples measure ``rates`` against + the stability clock and ``disciplined`` (default ``rates``) against + the rate clock.""" + result: list[tuple[str, dict[str, str]]] = [] + for role, name in zip(("rate", "stability"), names, strict=True): + clock = {"role": role, "name": name, "hz": str(clock_hz)} + clock["resolution_ns"] = f"{resolution_ns:.3f}" + result.append(("clock", clock)) + series = {"rate": disciplined or rates, "stability": rates} + state = {role: (10**12, 5 * clock_hz) for role in series} + for index in range(len(rates) + 1): + sample = {"index": str(index)} + for role, role_rates in series.items(): + tsc, ticks = state[role] + if index: + tsc += round(role_rates[index - 1] * interval_s) + ticks += round(interval_s * clock_hz) + state[role] = (tsc, ticks) + sample[f"{role}_tsc"] = str(tsc) + sample[f"{role}_clock"] = str(ticks) + sample[f"{role}_bracket"] = str(bracket) + result.append(("sample", sample)) + return result def check( context: doctor.DoctorContext, @@ -963,27 +1038,113 @@ def check( windows: bool = False, ) -> doctor.CheckResult: with ( - patch.object(doctor, "run_probe", return_value=probe_records), + patch.object(doctor, "run_probe", return_value=probe_records) as run, patch.object(doctor, "host_clocksource", return_value=clocksource), patch.object(doctor, "host_is_windows", return_value=windows), ): - return doctor.check_rate(context) + result = doctor.check_rate(context) + self.assertEqual( + run.call_args.args[1:], + ( + "rate", + "--samples", + str(context.schedule.rate_samples), + "--interval-ms", + str(context.schedule.rate_interval_ms), + ), + ) + return result + def ci_context(backend: str = "kvm") -> doctor.DoctorContext: + context = doctor_context(self.root, backend) + context.schedule = doctor.CI_SCHEDULE + return context + + steady = [2793437000.0 + jitter for jitter in (0, 200, -150, 100) * 3] context = doctor_context(self.root) context.facts["tsc_hz"] = "2793437000" - result = check(context, records(2793437500, 2793437800)) + result = check(context, records(steady)) self.assertTrue(result.passed, result.detail) + self.assertIn( + "against CLOCK_MONOTONIC from 13 samples 10 s apart; against " + "CLOCK_MONOTONIC_RAW (resolution 1 ns) the interval rates agree within " + "0.125 ppm, largest interval uncertainty 0.004 ppm", + result.detail, + ) + self.assertIn("0.0 ppm from the backend's 2793437000 Hz", result.detail) self.assertIn("host clocksource tsc (evidence)", result.detail) + self.assertEqual(context.facts["rate_agreement_ppm"], "0.125") + self.assertEqual(context.facts["rate_deviation_ppm"], "0.0") + # chrony steers CLOCK_MONOTONIC's rate by ppm between seconds, which + # says nothing about the TSC: the undisciplined clock judges stability, + # and the disciplined one the rate. + steered = [2300000100.0 + jitter for jitter in (0, 6900, -4600)] * 4 + context = doctor_context(self.root, "mshv") + context.facts["native_tsc_hz"] = "2300000000" + result = check(context, records([2300000100.0] * 12, disciplined=steered)) + self.assertTrue(result.passed, result.detail) + self.assertIn("agree within 0.000 ppm", result.detail) + self.assertEqual(context.facts["measured_tsc_hz"], "2300000867") + self.assertIn("0.4 ppm from the backend's 2300000000 Hz", result.detail) + # CI's short schedule; Windows reads QueryPerformanceCounter, which time + # synchronization never steers, in both roles. Its 100 ns tick keeps a + # 1 s interval conclusive at 0.1 ppm. + qpc = records( + [2300000100.0, 2300000200.0], + interval_s=1.0, + bracket=0, + clock_hz=10_000_000, + resolution_ns=100.0, + names=("QueryPerformanceCounter", "QueryPerformanceCounter"), + ) + windows = check(ci_context("whp"), qpc, windows=True) + self.assertTrue(windows.passed, windows.detail) + self.assertIn( + "against QueryPerformanceCounter from 3 samples 1 s apart", windows.detail + ) + self.assertIn("uncertainty 0.100 ppm", windows.detail) + self.assertNotIn("clocksource", windows.detail) + # Hyper-V's reference TSC page advances CLOCK_MONOTONIC_RAW in 100 ns + # steps: the resolution, not the empty brackets, bounds the measurement. + tsc_page = records( + [2300000100.0, 2300000200.0], interval_s=1.0, bracket=0, resolution_ns=100.0 + ) + result = check(ci_context("mshv"), tsc_page, "hyperv_clocksource_tsc_page") + self.assertTrue(result.passed, result.detail) + self.assertIn("CLOCK_MONOTONIC_RAW (resolution 100 ns)", result.detail) + self.assertIn("uncertainty 0.100 ppm", result.detail) + coarse = records( + [2300000100.0, 2300000200.0], interval_s=1.0, bracket=0, resolution_ns=300.0 + ) + result = check(ci_context("mshv"), coarse) + self.assertFalse(result.passed) + self.assertIn( + "inconclusive: an interval's uncertainty is 0.300 ppm", result.detail + ) for probe_records, message in ( - (records(2793437000, 2793440000), "unstable"), - (records(2794000000, 2794000001), "ppm from the measured"), + ( + records([2793437000.0, 2793440000.0] * 6), + "unstable: interval rates differ by 1.074 ppm > 1.0 ppm", + ), + ( + records([2793437000.0] * 2, interval_s=1.0, bracket=400), + "inconclusive: an interval's uncertainty is 0.401 ppm > 0.25 ppm", + ), + (records([2794000000.0] * 12), "201.5 ppm from the measured rate"), + (records([100_000_000.0] * 12), "[E_TSC_RATE_IMPLAUSIBLE]"), ): with self.subTest(message=message): context = doctor_context(self.root) + if "inconclusive" in message: + context = ci_context() context.facts["tsc_hz"] = "2793437000" result = check(context, probe_records) self.assertFalse(result.passed) self.assertIn(message, result.detail) + with self.assertRaisesRegex(doctor.ScriptError, "printed 3 samples, not 13"): + check(doctor_context(self.root), records([2793437000.0] * 2)) + with self.assertRaisesRegex(doctor.ScriptError, "no stability clock"): + check(ci_context(), records([2793437000.0] * 2, interval_s=1.0)[:1]) # The clocksource is evidence on every backend. for backend, clocksource in ( ("kvm", "hpet"), @@ -992,26 +1153,16 @@ def check( with self.subTest(backend=backend, clocksource=clocksource): context = doctor_context(self.root, backend) context.facts["native_tsc_hz"] = "2300000000" - result = check(context, records(2300000100, 2300000200), clocksource) + result = check(context, records([2300000100.0] * 12), clocksource) self.assertTrue(result.passed, result.detail) self.assertIn( f"host clocksource {clocksource} (evidence)", result.detail ) self.assertEqual(context.facts["host_clocksource"], clocksource) - windows = check( - doctor_context(self.root, "whp"), - records(2300000100, 2300000200), - windows=True, - ) - self.assertTrue(windows.passed, windows.detail) - self.assertNotIn("clocksource", windows.detail) # Without H3 (validate-runner), H4 still checks stability. - standalone = check(doctor_context(self.root), records(2300000100, 2300000200)) + standalone = check(ci_context(), records([2300000100.0] * 2, interval_s=1.0)) self.assertTrue(standalone.passed, standalone.detail) self.assertIn("not compared with the backend's rate", standalone.detail) - unstable = check(doctor_context(self.root), records(2300000100, 2300009100)) - self.assertFalse(unstable.passed) - self.assertIn("unstable", unstable.detail) def test_host_skew_check_applies_the_bound(self): summary = { @@ -1042,46 +1193,101 @@ def test_host_skew_check_applies_the_bound(self): self.assertFalse(result.passed) self.assertIn(message, result.detail) - def test_guest_warp_check_runs_the_idle_schedule_and_applies_the_bound(self): + def test_guest_warp_check_runs_the_idle_schedules_and_applies_the_bound(self): context = doctor_context(self.root) for path in (context.openvmm, context.kernel, context.initrd): path.write_bytes(b"") - boot = ( - BOOT_LINE.replace("cpus=4", "cpus=8") + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" - ) - text = boot + warp_output(pairs=28) + warp_output(pairs=28, offset=60) + + def boot(cpus: int) -> str: + line = BOOT_LINE.replace("cpus=4", f"cpus={cpus}") + return line + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + + smp = boot(8) + warp_output(pairs=28) * 4 + warp_output(pairs=28, offset=60) + single = boot(1) + warp_output(pairs=0, offset=0) with ( patch.object(doctor.os, "cpu_count", return_value=8), patch.object( - doctor, "run_guest_script", return_value={"text": text} + doctor, + "run_guest_script", + side_effect=[{"text": smp}, {"text": single}], ) as run, ): result = doctor.check_guest_warp(context) self.assertTrue(result.passed, result.detail) - self.assertIn("rounds=2 idle_s=1", result.detail) + self.assertIn("vcpus=8 rounds=5 idle_gaps_s=0.1,1,5,1", result.detail) + self.assertIn("vcpus=1 rounds=1 idle_gaps_s=none", result.detail) self.assertEqual(context.facts["guest_warp_ns"], "60") - command, script, marker = run.call_args.args - self.assertEqual(command[command.index("--processors") + 1], "8") - self.assertEqual(script, time_abi.warp_probe_script() + "nvx-exit 0\n") - self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + (smp_call, single_call) = run.call_args_list + for call, processors, gaps in ( + (smp_call, "8", time_abi.QUALIFICATION_WARP_GAPS), + (single_call, "1", ()), + ): + command, script, marker = call.args + self.assertEqual(command[command.index("--processors") + 1], processors) + self.assertEqual(script, time_abi.warp_probe_script(gaps) + "nvx-exit 0\n") + self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + # CI's schedule runs the probe twice with a 1 s gap. + context.schedule = doctor.CI_SCHEDULE + ci = boot(8) + warp_output(pairs=28) * 2 + with ( + patch.object(doctor.os, "cpu_count", return_value=8), + patch.object( + doctor, + "run_guest_script", + side_effect=[{"text": ci}, {"text": single}], + ), + ): + result = doctor.check_guest_warp(context) + self.assertTrue(result.passed, result.detail) + self.assertIn("vcpus=8 rounds=2 idle_gaps_s=1 ", result.detail) + context.schedule = doctor.QUALIFICATION_SCHEDULE cases = ( - (boot + warp_output(pairs=28), "ran 1 rounds instead of 2"), ( - boot + warp_output(pairs=28) + warp_output(pairs=28, offset=1500), - "in round 2: max_abs_offset_ns=1500", + [boot(8) + warp_output(pairs=28) * 2], + "vcpus=8", + "ran 2 rounds instead of 5", + ), + ( + [ + boot(8) + + warp_output(pairs=28) * 4 + + warp_output(pairs=28, offset=1500) + ], + "vcpus=8", + "in round 5: max_abs_offset_ns=1500", + ), + ([warp_output(pairs=28) * 5], "vcpus=8", "NVX-TIME-ABI boot marker"), + ( + [smp, boot(1) + warp_output(pairs=0, backward=2000, verdict="FAIL")], + "vcpus=1", + "max_backward_ns=2000", ), - (warp_output(pairs=28) * 2, "NVX-TIME-ABI boot marker"), ) - for output, message in cases: + for outputs, guest, message in cases: with ( self.subTest(message=message), patch.object(doctor.os, "cpu_count", return_value=8), - patch.object(doctor, "run_guest_script", return_value={"text": output}), + patch.object( + doctor, + "run_guest_script", + side_effect=[{"text": output} for output in outputs], + ), ): result = doctor.check_guest_warp(context) self.assertFalse(result.passed) - self.assertIn("vcpus=8", result.detail) + self.assertTrue(result.detail.startswith(f"{guest}: "), result.detail) self.assertIn(message, result.detail) + # A 1-vCPU host runs only the 1-vCPU guest. + with ( + patch.object(doctor.os, "cpu_count", return_value=1), + patch.object( + doctor, "run_guest_script", return_value={"text": single} + ) as run, + ): + result = doctor.check_guest_warp(context) + self.assertTrue(result.passed, result.detail) + self.assertEqual(run.call_count, 1) + self.assertTrue(result.detail.startswith("vcpus=1 rounds=1 idle_gaps_s=none")) missing = doctor.check_guest_warp(doctor_context(self.root / "missing")) self.assertIn("OpenVMM was not found", missing.detail) @@ -1165,9 +1371,11 @@ def broken(context: doctor.DoctorContext) -> doctor.CheckResult: def test_run_selects_the_cpu_fingerprint_or_runs_without_openvmm(self): seen: list[Path | None] = [] + schedules: list[doctor.Schedule] = [] def cpu(context: doctor.DoctorContext) -> doctor.CheckResult: seen.append(context.fingerprint) + schedules.append(context.schedule) context.facts["surface_digest"] = "sha256:62" return doctor.CheckResult("H2", True, "ok") @@ -1181,7 +1389,7 @@ def cpu(context: doctor.DoctorContext) -> doctor.CheckResult: for extra in ( [], ["--cpu-fingerprint", str(self.root / "fingerprint.json")], - ["--no-openvmm"], + ["--no-openvmm", "--ci-schedule"], ): arguments = doctor_parser().parse_args([*common, *extra]) self.assertEqual(doctor.run(arguments), 0) @@ -1193,6 +1401,14 @@ def cpu(context: doctor.DoctorContext) -> doctor.CheckResult: None, ], ) + self.assertEqual( + schedules, + [doctor.QUALIFICATION_SCHEDULE] * 2 + [doctor.CI_SCHEDULE], + ) + self.assertEqual(doctor.QUALIFICATION_SCHEDULE.rate_samples, 13) + self.assertEqual(doctor.QUALIFICATION_SCHEDULE.rate_interval_ms, 10_000) + self.assertEqual(doctor.CI_SCHEDULE.rate_samples, 3) + self.assertEqual(doctor.CI_SCHEDULE.rate_interval_ms, 1_000) self.assertIn("CPU surface digest: `sha256:62`", summary.read_text("utf-8")) arguments = doctor_parser().parse_args(["--backend", "kvm", "--no-openvmm"]) with self.assertRaisesRegex(doctor.ScriptError, "cannot run H3 and H6"): From 42b23d9bc7aa4b8bf763ed4e97850dabbd23bdf3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 16:15:59 -0700 Subject: [PATCH 051/188] doc: bound the deferred time ABI work and keep the agent step numbers Restore the snapshot agent's step numbers that the guest code cites: step 11 defers the restore checks to step 13, step 12 acknowledges, and step 13 is the daemon's asynchronous work. Steps 6 to 10 and 12 are the readiness path. Bound the fail-fast window under a CPU-bound workload: step 13 and the asynchronous boot checks start at SCHED_IDLE and continue at normal priority if they have not finished 100 ms after the acknowledgement or shell-ready, instead of SCHED_IDLE's 0.5 s or more. Require the single helper for untiered restores with nothing to activate, and define elapsed_us at restore: steps 6 to 12 and the step 11 checks, without the grace period wait or the deferred C7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 62 ++++++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 23 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 0af8b8747..32616495e 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -973,8 +973,10 @@ mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the [uncertainty bound](#uncertainty-bounds) and steps the clock to host UTC (`C12`), so the workload starts on host time; this step is not counted as a discontinuity. Init then reports shell-ready (or starts the workload, in -modes without a shell) and runs every other boot check asynchronously, at -`SCHED_IDLE`. The checks start the daemon when they pass. Every check stays +modes without a shell) and runs every other boot check asynchronously: at +`SCHED_IDLE` for the first 100 ms after shell-ready, and at normal priority +after that (see [Snapshot agent](#snapshot-agent)). The checks start the +daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the workload is running. A guest that powers off before its checks finish skips them, which is why CI takes its evidence from `nvx-time status`, which waits for them. @@ -1043,7 +1045,9 @@ NVX-TIME-ABI: v=1 phase=runtime status= generation= It exits with status 0. If a check is still pending after 30 s, the first line reports `status=pending`, without `elapsed_us`, and the exit status is -1. `elapsed_us` covers the checks and, at boot, the daemon start. A failed +1. At boot, `elapsed_us` covers the checks and the daemon start. At restore, +it covers steps 6 to 12 of the [snapshot agent](#snapshot-agent) and the +step 11 checks, but not the grace period wait or the deferred `C7`. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193. @@ -1119,8 +1123,9 @@ Before the capture request: If the write returns without a restore (no destination or a rejected capture), the agent removes the barriers and then restores the values saved -in steps 2 and 3. Otherwise, in the restored process, only these steps run -before the acknowledgement: +in steps 2 and 3. Otherwise, the restored process runs these steps. Steps 6 +to 10 and 12 are the readiness path: nothing else runs before the +acknowledgement. 6. Read the status from `0xea`. Bit 1 is always set after a restore. 7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, @@ -1141,11 +1146,13 @@ before the acknowledgement: 10. Run the existing entropy and identity repair for the tier. The RTC-based wall-clock refresh is removed; `instance-checkpoint` restores also get step 8. -11. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. -12. After the acknowledgement, or after step 10 when none is required, the - daemon finishes the restore asynchronously, and fail-fast: - - it runs the restore checks (`C1`, `C2`, and `C5` on newly onlined - CPUs; `C3` for CPU 0; `C6`; `C10`); +11. Defer the restore checks to step 13: `C1`, `C2`, and `C5` on newly + onlined CPUs; `C3` for CPU 0; `C6`; and `C10`. They do not wait, and a + failure still emits its violation event and powers off with status 193. +12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. +13. After the acknowledgement, or after step 10 when none is required, signal + the daemon, which finishes the restore asynchronously: + - it runs the step 11 checks; - it waits for the [grace period release](#rcu-grace-period-release); - it restores the values saved in steps 2 and 3 (`G_REPAIR_SUPPRESSION`); @@ -1156,16 +1163,25 @@ before the acknowledgement: It prints nothing unless a check fails. No restore check runs before the acknowledgement. The post-acknowledgement -checks are `C1`, `C2`, `C3`, `C5`, `C6`, `C7`, and `C10`. Steps 6, 7, and 8 -run in one helper process, which may also run every other step before the -acknowledgement and signal the daemon, so that a restore without shell work -(for example, untiered with no processor or memory target) needs no second -helper. Step 12 and the asynchronous boot checks run at `SCHED_IDLE`, so on -few vCPUs they never take a CPU from the restored or starting workload; the -watcher and the discipline run at normal priority. Under a CPU-bound -workload they still progress, more slowly, and stall suppression stays set -until the release. Repair failures emit a violation -event and power off with status 195. +checks are `C1`, `C2`, `C3`, `C5`, `C6`, `C7`, and `C10`. + +Steps 6, 7, and 8 run in one helper process. For an untiered restore with no +processor or memory to activate, that helper runs every step through 12 and +signals the daemon, so no other process starts on the readiness path. Each +extra process costs 2 to 4 ms after a restore (fork and exec under demand +faulting, measured on KVM). + +Step 13 and the asynchronous boot checks start at `SCHED_IDLE`, so on few +vCPUs they never take a CPU from the restored or starting workload. Work +that has not finished 100 ms after the acknowledgement (or after step 10, +when none is required) or after shell-ready continues at normal priority. +This bounds how long a CPU-bound workload runs before a failing check powers +the guest off: about 100 ms plus the checks' own run time, where +`SCHED_IDLE` alone would allow about 0.5 s or more. On an idle KVM guest, a +failing restore check powers off within 1 to 3 ms. The watcher and the +discipline always run at normal priority, and stall suppression stays set +until the release. Repair failures emit a violation event and power off with +status 195. ### RCU grace period release @@ -1233,7 +1249,7 @@ every change, and `nvx-time status` reads it. The sandbox agent bind-mounts | `check_phase` | `boot`, `capture`, or `restore`: the last conformance check | | `check_status` | `pending` while it runs, then `ok`; a failed check powers the guest off | | `check_cpus` | Online CPUs the check covered | -| `check_elapsed_us` | Its duration, with the daemon start at boot | +| `check_elapsed_us` | Its duration, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | | `tsc_hz`, `lapic_hz` | `F` and `L` | | `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | | `last_discontinuity` | `none`, `restore`, or `step` | @@ -1461,8 +1477,8 @@ and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that selection to the arrival of the `0x605` acknowledgement). The `restore.guest_repair_gate` milestone still spans from the VP release to the release of the acknowledgement boundary. The guest's restore checks run -after the acknowledgement, outside these phases; `nvx-time status` reports -their duration as `elapsed_us`. +after the acknowledgement, outside these phases. `nvx-time status` reports +`elapsed_us`, which covers the guest's readiness path and those checks. ## Test matrix From e829d8c9db51f1523bca0b4d4ae19b4e5e28bfd7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 16:36:13 -0700 Subject: [PATCH 052/188] doc: make the effective CPUID the single source of the guest's CPUID List the terminating subleaf of the topology leaves in the effective CPUID: an invalid level with its own number in ECX[7:0] and the x2APIC ID in EDX, which Linux reads to end its enumeration. KVM synthesized it, WHP zeroed it, and MSHV showed the hypervisor's value. State that core passes the effective CPUID as TimeAbiConfig::cpuid and every backend presents it, while banks and pinned MSRs come from the profile. Entries that no architectural enumeration reaches are reserved: backends may return zeros, Intel's out-of-range result, or the hypervisor's value, and nothing records or checks them. Strict zeros could not be uniform, because KVM answers out-of-range leaves and topology subleaves itself, and they would cost MSHV 3 to 4.5 us per entry at every boot and restore. Bring the MSHV and WHP CPUID cells of the backend obligations up to date. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 32616495e..a8425ff3d 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus CPUID intercept results for leaves 1, 6, 7.0, `0xA`, and `0x80000007` | Processor feature banks derived from the profile (`hv_banks`), and the profile's pinned leaves as `CpuidResultList2` results (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the per-VP topology leaves (`1`, `4`, `0xB`, and `0x1F` on Intel): each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID, with the per-VP fields left to the hypervisor | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -303,9 +303,11 @@ an SMP-restored VM for its lifetime. ## CPU profiles A CPU profile is the complete guest-visible CPU surface of one vendor and CPU -generation, shared by every backend. The guest sees only profile values, never -host passthrough, except for a fixed, code-defined set of VMM-owned fields and -the fields the time ABI owns. Profiles are data: derived mechanically by +generation, shared by every backend. In every entry that an architectural +enumeration reaches, the guest sees only profile values, never host +passthrough, except for a fixed, code-defined set of VMM-owned fields and +the fields the time ABI owns; entries beyond are reserved (see **Effective +CPUID**). Profiles are data: derived mechanically by intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker `cpu-template-helper` workflow) per register, first within each backend and then across backends, reviewed, checked in at @@ -365,9 +367,12 @@ the profile, the VM topology, and the time ABI's identity leaves. **Effective CPUID.** The effective CPUID (`openvmm-effective-cpuid/v1`) lists exactly the governed leaves: every leaf and subleaf of the profile's -`cpuid` table; the topology leaves `0xB` and `0x1F`, at the subleaves the -VM's topology defines, when they are within the maximum basic leaf; and the -identity leaves `0x40000000..=0x40000005` with the explicit zero leaves. Each +`cpuid` table; the topology leaves `0xB` and `0x1F`, when they are within +the maximum basic leaf, at the subleaves the VM's topology defines and the +subleaf that terminates them (an invalid level: `EAX` and `EBX` zero, +`ECX[7:0]` its own number, `ECX[15:8]` zero, and the x2APIC ID in `EDX`), +which Linux reads to end its enumeration; and the identity leaves +`0x40000000..=0x40000005` with the explicit zero leaves. Each entry holds the four registers and their masks; the per-VP fields hold VP 0's APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, records it, and recomputes it on restore, so the record never depends on how @@ -379,6 +384,26 @@ WHP from VP 0's register view), reading an entry without a subleaf at subleaf 0, and OpenVMM compares the report with the effective CPUID under its masks. +The effective CPUID is the single source of the guest's CPUID. Core passes it +to the backend as `TimeAbiConfig::cpuid`, with the per-VP fields unmasked, +and the backend presents every entry with each VP's own APIC identity (see +[Backend obligations](#backend-obligations)). Banks and pinned MSRs come +from the profile itself, which the backend looks up by +`TimeAbiConfig::cpu_profile`. The governed entries are every entry an +architectural enumeration reaches. Entries beyond them are reserved: + +- a leaf above the maximum basic or extended leaf; +- a subleaf past its leaf's reported maximum or terminator; +- a `0xD` subleaf of a component that is not enabled. + +A backend may answer a reserved entry with zeros, Intel's out-of-range +result, or the hypervisor's own value. Reserved entries are neither recorded +nor checked, and Linux reads none of them. Strict zeros would not be uniform +anyway: KVM answers above the maximum basic leaf with the highest basic +leaf's result, and it synthesizes a terminator for every topology subleaf +past the last level. Registering zeros for them would also add 3 to 4.5 µs +per entry to every MSHV cold boot and restore. + Informational fields: - The brand string (`0x80000002..=0x80000004`) is generic per generation: From 3fa136ffdf5929072f0d82e5d4d98522a6c0cc0a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 16:57:02 -0700 Subject: [PATCH 053/188] doc: report a wrong per-VP APIC identity as a time ABI violation Under the time ABI, MSHV presents x2APIC ID 0 in CPUID 0xB on every VP: it registers the effective CPUID's 0xB entries with EDX left to the hypervisor, which does not implement 0xB for these partitions. Linux then logs "APIC ID mismatch" for every application processor, and initial apicid reads 0 everywhere. Add the watcher pattern G_APIC_ID_MISMATCH, which the boot check C4 reports before the daemon starts and the daemon reports afterwards, for example for processors a restore brings online. Correct the MSHV CPUID cell: each VP's x2APIC ID must come from OpenVMM, by a mechanism the mshv agent settles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index a8425ff3d..ddb188fda 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID, with the per-VP fields left to the hypervisor | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID. Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -1109,6 +1109,7 @@ record matches when its message text contains every substring of a row: | `G_HARD_LOCKUP` | `Watchdog detected hard LOCKUP` | | `G_HUNG_TASK` | `INFO: task ` and ` blocked for more than ` | | `G_UNCHECKED_MSR` | `unchecked MSR access error` | +| `G_APIC_ID_MISMATCH` | `APIC ID mismatch` (Linux found a CPUID APIC identity that differs from the CPU's APIC: a backend did not present a per-VP field of the [effective CPUID](#cpu-profiles)) | | `G_KMSG_OVERRUN` | A read of `/dev/kmsg` fails with `EPIPE` (records were lost) | Before the boot check passes, a match is reported by `C4`. Afterwards it is @@ -1346,7 +1347,7 @@ signals, and 255): | Status | Class | Guest codes | | ---: | --- | --- | | 193 | Conformance | `G_CONFORMANCE_C1` to `G_CONFORMANCE_C12`, `G_KERNEL_WX` | -| 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_KMSG_OVERRUN` | +| 194 | Runtime violation | `G_TSC_UNSTABLE`, `G_CLOCKSOURCE_UNSTABLE`, `G_CLOCKSOURCE_SWITCH`, `G_CLOCKSOURCE_SKEW`, `G_TSC_WARP`, `G_TSC_ADJUST`, `G_RCU_STALL`, `G_RCU_STARVED`, `G_SOFT_LOCKUP`, `G_HARD_LOCKUP`, `G_HUNG_TASK`, `G_UNCHECKED_MSR`, `G_APIC_ID_MISMATCH`, `G_KMSG_OVERRUN` | | 195 | Restore repair | `G_REPAIR_PACKET`, `G_REPAIR_GENERATION`, `G_REPAIR_SAMPLE`, `G_REPAIR_CLOCK`, `G_REPAIR_SUPPRESSION` | `G_SAMPLE_UNCERTAIN` is the only non-fatal guest code: the discipline records From 77249d0cc1ec1be76115c011517c7bdaa069110a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 17:10:16 -0700 Subject: [PATCH 054/188] doc: report every recorded time ABI check from nvx-time status The CI harness needs the boot, capture, and restore results after a restore, and a status run that cannot race a restore's deferred work. The state file now keeps one record per phase (boot_*, capture_*, and restore_*, with the generation of each check) instead of the last check only, and nvx-time status prints one line per recorded phase, then the runtime line. The capture record travels in the snapshot. Step 8 of the snapshot agent records the restore as pending in the state-file update it already makes, and a missing boot record counts as pending, so status always waits for the checks of the boot or restore it follows. Define the exit status (0 only when every phase line is ok; 1 on a timeout or, in report-only mode, a failure), elapsed_us at capture, and what the conformance runs require from the output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 76 +++++++++++++++++++++++++++++------------- 1 file changed, 53 insertions(+), 23 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index ddb188fda..684565c23 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1058,30 +1058,55 @@ NVX-TIME-ABI-EXHAUSTIVE: v=1 status= cpus= failures= **Status command.** Checks print nothing on success; they record their result in the [state file](#wall-clock-discipline). `/sbin/nvx-time status` prints it on demand: CI scenarios and the matrix driver run it over the -console when they need evidence, and production paths never do. It waits -until no check is pending (the asynchronous boot checks, or a restore's -deferred work), for at most 30 s. It then prints the marker of the last -completed check and the runtime line: +console when they need evidence, outside measured paths, and production +paths never do. It waits until no recorded check is pending, for at most +30 s. A missing state file or boot record counts as pending, because every +boot runs the boot check, and step 8 of the [snapshot agent](#snapshot-agent) +records the restore check as pending before the workload resumes, so a +`status` run after a restore always waits for that restore's deferred work. +It then prints one line per recorded phase, in the order boot, capture, and +restore, then the runtime line: ```text -NVX-TIME-ABI: v=1 phase= status=ok cpus= tsc_hz= lapic_hz= generation= elapsed_us= +NVX-TIME-ABI: v=1 phase= status= cpus= tsc_hz= lapic_hz= generation= elapsed_us= NVX-TIME-ABI: v=1 phase=runtime status= generation= discontinuities= offset_ns= uncertainty_ns= rejected_samples= last_sample_error= ``` -It exits with status 0. If a check is still pending after 30 s, the first -line reports `status=pending`, without `elapsed_us`, and the exit status is -1. At boot, `elapsed_us` covers the checks and the daemon start. At restore, -it covers steps 6 to 12 of the [snapshot agent](#snapshot-agent) and the -step 11 checks, but not the grace period wait or the deferred `C7`. A failed -check prints no marker: it emits the violation event with code -`G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193. +Each phase line reports its own check: + +- `cpus` is the number of online CPUs the check covered. +- `generation` is `g` when the check ran: 0 for boot, the capture's `g` for + capture, and at least 1 for a restore. +- The capture line comes from the checks before the capture request (step + 1), which the snapshot carries, so a restored guest still reports it. +- `status` reports the latest capture, which in a restored guest is the + capture that produced its snapshot, and the restore of this process: a + restored process never captures, and every restore is a new process. +- The runtime line describes the guest at the time `status` runs. + +No other line uses the `NVX-TIME-ABI:` prefix, so a harness can parse every +line by its `phase`. + +`status` exits with status 0 when every phase line reports `ok`. If a check +is still pending after 30 s, its line reports `status=pending`, without +`elapsed_us`, and the exit status is 1. + +At boot, `elapsed_us` covers the checks and the daemon start; at capture, +the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it +covers steps 6 to 12 and the step 11 checks, but not the grace period wait +or the deferred `C7`. + +A failed check prints no marker: it emits the violation event with code +`G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, +so `status` never reports a failure. **Report-only mode (test only).** The kernel token `nvx_time_abi=report-only` makes the checks and the watcher report instead of powering off, so the guest can be evaluated under an OpenVMM that does not implement the time ABI. -Violations print `NVX-TIME-REPORT-VIOLATION` lines with the event's fields, -and `nvx-time status` prints its marker with the `NVX-TIME-REPORT` prefix and -` failures=` appended. These prefixes are reserved for this mode; +Violations print `NVX-TIME-REPORT-VIOLATION` lines with the event's fields. +`nvx-time status` prints its phase lines with the `NVX-TIME-REPORT` prefix, +`status=fail` for a failed check, and ` failures=` appended, and it exits +with status 1 if any check failed. These prefixes are reserved for this mode; production images never set the token, and harnesses never accept a report-only boot as conformant. @@ -1140,7 +1165,8 @@ builds the soft-lockup and hung-task detectors. Before the capture request: 1. Run the capture checks (`C6`, `C7`, `C10`). They do not wait and print - nothing. + nothing; they record `capture_status=ok` in the state file, which the + snapshot carries. 2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. 3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and `/proc/sys/kernel/hung_task_timeout_secs`. @@ -1167,7 +1193,9 @@ acknowledgement. `time = theta`, then `ADJ_FREQUENCY | ADJ_STATUS` with `freq = -rate_deviation` (clamped to ±500 ppm) and `status = STA_PLL | STA_NANO` (`G_REPAIR_CLOCK`). Record the restore - discontinuity and the new `g`. + discontinuity and the new `g`, with `restore_status=pending` and + `restore_generation=g` in the same state-file update, so that + `nvx-time status` waits for this restore. 9. Run the existing processor and memory activation. 10. Run the existing entropy and identity repair for the tier. The RTC-based wall-clock refresh is removed; `instance-checkpoint` restores also get @@ -1272,10 +1300,10 @@ every change, and `nvx-time status` reads it. The sandbox agent bind-mounts | --- | --- | | `version` | 1 | | `generation` | `g` | -| `check_phase` | `boot`, `capture`, or `restore`: the last conformance check | -| `check_status` | `pending` while it runs, then `ok`; a failed check powers the guest off | -| `check_cpus` | Online CPUs the check covered | -| `check_elapsed_us` | Its duration, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | +| `boot_status`, `capture_status`, `restore_status` | `pending` while the phase's check runs, then `ok`; absent until the phase first runs (a failed check powers the guest off). The capture keys hold the latest capture, and the restore keys this process's restore | +| `boot_cpus`, `capture_cpus`, `restore_cpus` | Online CPUs the check covered | +| `boot_elapsed_us`, `capture_elapsed_us`, `restore_elapsed_us` | Its duration, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | +| `capture_generation`, `restore_generation` | `g` when the check ran (the boot check's is 0) | | `tsc_hz`, `lapic_hz` | `F` and `L` | | `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | | `last_discontinuity` | `none`, `restore`, or `step` | @@ -1522,8 +1550,10 @@ parser; four-byte portb reads; and the generation counter. all 18 registered hosts, plus the exhaustive CI check and the warp probe on every backend. Production paths print no marker, so CI scenarios and the matrix driver run `/sbin/nvx-time status` over the console after shell-ready -and after every restore, and require its `phase=boot` or `phase=restore` -marker with `status=ok` and the expected `generation`. The fleet runs them on +and after every restore. They require exit status 0 and, after a restore, a +`phase=restore` line with `status=ok`, the restored `generation`, and `cpus` +equal to the online CPUs; after a cold boot, a `phase=boot` line with +`generation=0`. The fleet runs them on the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on From a3e250cef157e8fda0e32699f6c395894adc733f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:01:42 -0700 Subject: [PATCH 055/188] doc: read the restore packet with inl and wait for boot checks at capture OpenVMM's instruction emulator, which serves string port I/O on MSHV and WHP, raises #GP for a CPL 3 rep insl although the TSS I/O permission bitmap grants the port, so step 7 reads the packet with four-byte inl reads only. A capture right after shell-ready may find the asynchronous boot checks still running, and C10 requires the daemon they start. Step 1 now waits for them, at most 30 s like nvx-time status, and fails the capture with G_CONFORMANCE_C10 on a timeout. Report-only mode keeps its failures in the state file: a phase's status may be fail, and per-phase failure counts are reserved keys. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 684565c23..e1a1bdbf2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1106,9 +1106,11 @@ can be evaluated under an OpenVMM that does not implement the time ABI. Violations print `NVX-TIME-REPORT-VIOLATION` lines with the event's fields. `nvx-time status` prints its phase lines with the `NVX-TIME-REPORT` prefix, `status=fail` for a failed check, and ` failures=` appended, and it exits -with status 1 if any check failed. These prefixes are reserved for this mode; -production images never set the token, and harnesses never accept a -report-only boot as conformant. +with status 1 if any check failed. The state file carries them: in this mode +only, a phase's `_status` key may be `fail`, and `boot_failures`, +`capture_failures`, and `restore_failures` count each check's failures. +These prefixes and keys are reserved for this mode; production images never +set the token, and harnesses never accept a report-only boot as conformant. ### Violation watcher @@ -1164,9 +1166,13 @@ builds the soft-lockup and hung-task detectors. Before the capture request: -1. Run the capture checks (`C6`, `C7`, `C10`). They do not wait and print - nothing; they record `capture_status=ok` in the state file, which the - snapshot carries. +1. Wait until no boot check is pending, for at most 30 s, as + `nvx-time status` does: a capture right after shell-ready may find the + asynchronous boot checks still running, and the daemon, which `C10` + requires, starts only when they pass. A timeout fails the capture with + `G_CONFORMANCE_C10` (193). Then run the capture checks (`C6`, `C7`, + `C10`), which do not wait and print nothing; they record + `capture_status=ok` in the state file, which the snapshot carries. 2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. 3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and `/proc/sys/kernel/hung_task_timeout_secs`. @@ -1181,9 +1187,10 @@ acknowledgement. 6. Read the status from `0xea`. Bit 1 is always set after a restore. 7. Read `CLOCK_REALTIME` as `t0`, write `0xa5` to `0xea`, read it as `t1`, - and read the packet with four-byte reads: `inl`, or `rep insl`, which - KVM serves in one exit and MSHV and WHP through OpenVMM's instruction - emulator. Validate the magic, version, + and read the packet with four-byte `inl` reads. (`rep insl` would take a + single exit on KVM, but OpenVMM's instruction emulator, which serves + string port I/O on MSHV and WHP, raises #GP for it at CPL 3 although the + TSS I/O permission bitmap grants the port.) Validate the magic, version, reserved bits, counts, `g`, and the generation ID (`G_REPAIR_PACKET`, `G_REPAIR_GENERATION`). 8. Set the wall clock. Compute `theta` and `epsilon` from the bracket. If @@ -1304,6 +1311,7 @@ every change, and `nvx-time status` reads it. The sandbox agent bind-mounts | `boot_cpus`, `capture_cpus`, `restore_cpus` | Online CPUs the check covered | | `boot_elapsed_us`, `capture_elapsed_us`, `restore_elapsed_us` | Its duration, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | | `capture_generation`, `restore_generation` | `g` when the check ran (the boot check's is 0) | +| `boot_failures`, `capture_failures`, `restore_failures` | [Report-only mode](#conformance-checks-and-the-nvx-time-abi-marker) only: the check's failure count | | `tsc_hz`, `lapic_hz` | `F` and `L` | | `discontinuities` | Wall-clock discontinuities since cold boot: restores plus discipline steps | | `last_discontinuity` | `none`, `restore`, or `step` | From ccfb489881e45f95efa922e059b8e1b5add50729 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:05:38 -0700 Subject: [PATCH 056/188] doc: judge H4 interval agreement against an undisciplined clock chrony steers CLOCK_MONOTONIC's rate every few seconds, which failed the short H4 schedule on 3 of 9 Azure MSHV runner runs (agreement up to 12.2 ppm). H4 now judges interval agreement and conclusiveness against a clock that time synchronization never steers, CLOCK_MONOTONIC_RAW on Linux and QueryPerformanceCounter on Windows (35 runs agree within 0.050 ppm), and keeps the whole-window rate against F_d on the disciplined CLOCK_MONOTONIC. A clock's resolution is measured as its smallest observed step, 100 ns for the Hyper-V reference TSC page, instead of clock_getres's 1 ns. The bounds and schedules are unchanged. State the limit: where the host clocksource is the TSC, no host clock can catch a TSC step, and the guest warp probe is the detector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 74 ++++++++++++++++++++++++++++++------------ 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index e1a1bdbf2..e50b42d32 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1412,24 +1412,37 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | `H1` | The backend device or API is present and usable | | `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | -| `H4` | TSC rate stability against the host's monotonic clock (see [Rate stability](#rate-stability-h4)). On a Linux host the detail also reports the host clocksource, as evidence only | +| `H4` | TSC rate stability against the host's clocks: interval agreement against an undisciplined clock (`CLOCK_MONOTONIC_RAW` or `QueryPerformanceCounter`) and the whole-window rate against `F_d` on the disciplined one (see [Rate stability](#rate-stability-h4)). On a Linux host the detail also reports the host clocksource, as evidence only | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | | `H6` | Guest warp probe on the qualification schedule (see [Warp schedules](#warp-schedules-h6-and-ci)) | | `H7` | Host UTC is synchronized: no `STA_UNSYNC` on Linux; a synchronized `w32tm` source on Windows | ### Rate stability (`H4`) -A host probe compares the TSC with the host's monotonic clock: -`CLOCK_MONOTONIC` on Linux, which is never stepped and runs at the rate time -synchronization disciplines, and `QueryPerformanceCounter` on Windows. A -sample reads the TSC between two reads of that clock, keeping the tightest -of up to 64 such brackets; its uncertainty is half the bracket plus half the -clock's resolution. The probe sleeps between samples, so the host's CPUs can -idle: 3 samples 1 s apart in `validate-runner`, and 13 samples 10 s apart -(120 s) in `doctor`. For each interval between consecutive samples it -computes the rate `r_i`, in TSC cycles per second of the clock, and its -uncertainty `u_i`, the two samples' uncertainties divided by the interval; -`r` is the rate over the whole window. The check passes if: +A host probe compares the TSC with two host clocks: + +- **Interval agreement and conclusiveness** use a clock that time + synchronization never steers: `CLOCK_MONOTONIC_RAW` on Linux and + `QueryPerformanceCounter` on Windows. +- **The whole-window rate**, checked against `F_d`, uses the disciplined + `CLOCK_MONOTONIC` on Linux, which is never stepped and runs at the true + rate, and `QueryPerformanceCounter` on Windows. + +A sample reads the TSC between two reads of a clock and keeps the tightest of +up to 64 such brackets. Its uncertainty is half the bracket plus half the +clock's resolution. The resolution is measured, not taken from +`clock_getres`: it is the smallest step the clock is observed to take, for +example 100 ns for the Hyper-V reference TSC page clock +(`hyperv_clocksource_tsc_page`), which `clock_getres` reports as 1 ns, and +one 10 MHz tick for `QueryPerformanceCounter`. + +The probe sleeps between samples, so the host's CPUs can idle: 3 samples 1 s +apart in `validate-runner`, and 13 samples 10 s apart (120 s) in `doctor`. +For each interval between consecutive samples, it computes against the +undisciplined clock the rate `r_i`, in TSC cycles per second, and its +uncertainty `u_i`, the two samples' uncertainties divided by the interval. +`r` is the rate over the whole window against the disciplined clock. The +check passes if: - every `u_i` is at most 0.25 ppm of `r`; otherwise the measurement is inconclusive and fails; @@ -1438,16 +1451,35 @@ uncertainty `u_i`, the two samples' uncertainties divided by the interval; run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. The detail reports `r`, its deviation from `F_d` in ppm, the agreement, the -largest `u_i`, the clock, and, on Linux, the host clocksource, which never -gates. An invariant TSC agrees with the host clock far -inside 1 ppm: 0.001 to 0.089 ppm over 1 s windows on the eleven CI runners, -and 0.000 ppm with a residual of at most 0.07 µs over 118 s on the WHP hosts. +largest `u_i`, both clocks and their measured resolutions, and, on Linux, the +host clocksource, which never gates. + +Interval agreement uses the undisciplined clock because chrony steers +`CLOCK_MONOTONIC`'s rate every few seconds, on Azure through the Hyper-V PTP +clock: + +- Against `CLOCK_MONOTONIC`, the short schedule failed 3 of 9 runs on the + Azure MSHV runners. The agreement reached 12.2 ppm, and 30 1-s intervals + spread by up to 2.7 ppm while chrony's frequency moved by up to 2.7 ppm. +- Against `CLOCK_MONOTONIC_RAW` and `QueryPerformanceCounter`, 35 + short-schedule runs on the Azure KVM, MSHV, and WHP runners agree within + 0.000 to 0.050 ppm, with the largest `u_i` between 0.04 and 0.106 ppm. The + long schedule agrees within 0.002 ppm on prometheus30, where + `CLOCK_MONOTONIC` gave 0.402 ppm. +- The WHP hosts show 0.000 ppm with a residual of at most 0.07 µs over 118 s. + A TSC that stops, slows, or is rescaled across idle misses the bound by -orders of magnitude. The 100 ppm bound keeps the guest's wall-clock -discipline, which steers at most 500 ppm, clear of the TSC's error plus a -restore's rate deviation of up to 250 ppm. On Windows the clock derives from -the same TSC, so the deviation compares `F_d` with the host's own -calibration. +orders of magnitude, where the host's clocksource is independent of the TSC: +the Hyper-V reference page, the HPET, or `QueryPerformanceCounter` in a VM. +Where the host clocksource is `tsc` itself (the Azure KVM runners and +prometheus32), every kernel clock derives from the TSC, so no host clock can +catch a TSC step; the guest warp probe (`H6` and the CI schedule) is the +detector there. + +The 100 ppm bound keeps the guest's wall-clock discipline, which steers at +most 500 ppm, clear of the TSC's error plus a restore's rate deviation of up +to 250 ppm. On Windows the clock derives from the same TSC, so the deviation +compares `F_d` with the host's own calibration. ### Warp schedules (`H6` and CI) From 8e714a86a35a96a3e5079976adb81c1a86f16cd4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:09:30 -0700 Subject: [PATCH 057/188] doc: align H4's clock rules with the qualification code Use ci's exact rules from cd0a995: a clock's resolution is the larger of its reported resolution and, when two consecutive reads repeat a value, the smallest step observed between reads; the disciplined clock is the rate reference because its long-run rate is the true one; the probe's output names both clocks, and the H4 detail reports the stability clock. Cite the 0.07 ppm spread against CLOCK_MONOTONIC_RAW next to the 12.2 ppm against CLOCK_MONOTONIC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index e50b42d32..8a2bc56ae 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1424,17 +1424,19 @@ A host probe compares the TSC with two host clocks: - **Interval agreement and conclusiveness** use a clock that time synchronization never steers: `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows. -- **The whole-window rate**, checked against `F_d`, uses the disciplined - `CLOCK_MONOTONIC` on Linux, which is never stepped and runs at the true - rate, and `QueryPerformanceCounter` on Windows. +- **The whole-window rate `r`** and its deviation from `F_d` use the + disciplined `CLOCK_MONOTONIC` on Linux, which is never stepped and whose + long-run rate is the true one, and `QueryPerformanceCounter` on Windows. A sample reads the TSC between two reads of a clock and keeps the tightest of up to 64 such brackets. Its uncertainty is half the bracket plus half the -clock's resolution. The resolution is measured, not taken from -`clock_getres`: it is the smallest step the clock is observed to take, for -example 100 ns for the Hyper-V reference TSC page clock -(`hyperv_clocksource_tsc_page`), which `clock_getres` reports as 1 ns, and -one 10 MHz tick for `QueryPerformanceCounter`. +clock's resolution. A clock's resolution is the larger of its reported +resolution and, when two consecutive reads return the same value, the +smallest step observed between consecutive reads. Hyper-V's reference TSC +page clock (`hyperv_clocksource_tsc_page`), the clocksource of MSHV roots and +the Azure MSHV runners, advances the Linux clocks in 100 ns steps, although +`clock_getres` reports 1 ns. `QueryPerformanceCounter`'s resolution is one +10 MHz tick. The probe sleeps between samples, so the host's CPUs can idle: 3 samples 1 s apart in `validate-runner`, and 13 samples 10 s apart (120 s) in `doctor`. @@ -1450,17 +1452,20 @@ check passes if: - where `F_d` is known (in `doctor`, and in the `validate-runner` jobs that run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. -The detail reports `r`, its deviation from `F_d` in ppm, the agreement, the -largest `u_i`, both clocks and their measured resolutions, and, on Linux, the -host clocksource, which never gates. +The probe's output names both clocks, each with its role (rate or +stability), resolution, and observed step. The detail reports `r`, its +deviation from `F_d` in ppm, the agreement, the largest `u_i`, the stability +clock, and, on Linux, the host clocksource, which never gates. -Interval agreement uses the undisciplined clock because chrony steers -`CLOCK_MONOTONIC`'s rate every few seconds, on Azure through the Hyper-V PTP -clock: +Interval agreement uses the undisciplined clock because chrony's frequency +updates move `CLOCK_MONOTONIC`'s rate between seconds, on Azure through the +Hyper-V PTP clock: - Against `CLOCK_MONOTONIC`, the short schedule failed 3 of 9 runs on the - Azure MSHV runners. The agreement reached 12.2 ppm, and 30 1-s intervals - spread by up to 2.7 ppm while chrony's frequency moved by up to 2.7 ppm. + Azure MSHV runners: their interval rates differed by up to 12.2 ppm. Over + 30 1-s intervals they spread by up to 2.7 ppm, against at most 0.07 ppm + versus `CLOCK_MONOTONIC_RAW`, while chrony's frequency moved by up to + 2.7 ppm. - Against `CLOCK_MONOTONIC_RAW` and `QueryPerformanceCounter`, 35 short-schedule runs on the Azure KVM, MSHV, and WHP runners agree within 0.000 to 0.050 ppm, with the largest `u_i` between 0.04 and 0.106 ppm. The From 84eeea41c1a8cfe97bb03f812c22187220f2ff9d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:13:33 -0700 Subject: [PATCH 058/188] doc: guarantee that CPUID entries outside the profile carry no host data Revise the unlisted-entry decision of 955df3e from "reserved" to a guarantee. Every backend already meets it: KVM_SET_CPUID2 makes an unlisted entry zero or a value KVM derives from the effective CPUID (Intel's out-of-range result, topology terminators), WHP reads zero natively there (its hardware sweep of subleaves 0 to 63 and four leaves past each maximum on all three hosts), and MSHV registers a zero result for each such entry its host enumerates (4f3b55e4b, a few microseconds each in its partition build). So "never host passthrough" holds for every entry, and a guest reads the same values on any host of a generation, before and after a restore. The boot check and the snapshot record still cover the governed entries only; the backends' hardware tests cover the rest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 8a2bc56ae..104adbcc8 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID. Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID, and a zero result for each entry the host's CPUID enumerates that the effective CPUID does not list. Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); WHP reads zero natively for every other entry. CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -303,11 +303,11 @@ an SMP-restored VM for its lifetime. ## CPU profiles A CPU profile is the complete guest-visible CPU surface of one vendor and CPU -generation, shared by every backend. In every entry that an architectural -enumeration reaches, the guest sees only profile values, never host -passthrough, except for a fixed, code-defined set of VMM-owned fields and -the fields the time ABI owns; entries beyond are reserved (see **Effective -CPUID**). Profiles are data: derived mechanically by +generation, shared by every backend. The guest sees only profile values, +never host passthrough, except for a fixed, code-defined set of VMM-owned +fields and the fields the time ABI owns; an entry the effective CPUID does +not list reads zero or a value derived from it (see **Effective CPUID**). +Profiles are data: derived mechanically by intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker `cpu-template-helper` workflow) per register, first within each backend and then across backends, reviewed, checked in at @@ -390,19 +390,30 @@ and the backend presents every entry with each VP's own APIC identity (see [Backend obligations](#backend-obligations)). Banks and pinned MSRs come from the profile itself, which the backend looks up by `TimeAbiConfig::cpu_profile`. The governed entries are every entry an -architectural enumeration reaches. Entries beyond them are reserved: +architectural enumeration reaches. + +The other entries, which no architectural enumeration reaches, are: - a leaf above the maximum basic or extended leaf; - a subleaf past its leaf's reported maximum or terminator; - a `0xD` subleaf of a component that is not enabled. -A backend may answer a reserved entry with zeros, Intel's out-of-range -result, or the hypervisor's own value. Reserved entries are neither recorded -nor checked, and Linux reads none of them. Strict zeros would not be uniform -anyway: KVM answers above the maximum basic leaf with the highest basic -leaf's result, and it synthesizes a terminator for every topology subleaf -past the last level. Registering zeros for them would also add 3 to 4.5 µs -per entry to every MSHV cold boot and restore. +They read no host data either. Each reads zero, or, on KVM, a value KVM +derives from the effective CPUID by the architecture's rules: above the +maximum basic leaf, Intel's out-of-range result (the highest basic leaf's); +past the last topology level, an invalid level with the x2APIC ID. + +- WHP already reads zero there. +- MSHV would answer from the hypervisor's own view, so it registers a zero + result for each such entry that the host's CPUID enumerates, at 3 to + 4.5 µs each in its partition build. + +The MSHV and WHP hardware tests sweep subleaves 0 to 63 of every indexed leaf +and the four leaves past each maximum; KVM's values follow from +`KVM_SET_CPUID2`'s rules. The boot check and the snapshot +record cover only the governed entries. A guest therefore reads the same +value for every entry on any host of the profile's generation with the same +backend, before and after a restore. Informational fields: From 8eb36f6c269b566a99975fdec0a15d382c7bb299 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:24:55 -0700 Subject: [PATCH 059/188] doc: reserve unlisted CPUID entries and verify they hide host features Apply the orchestrator's decision, option (A) plus verification, in place of the zero guarantee of 3c2aed7. CPUID entries reachable only by probing past an advertised maximum are reserved: their values are implementation-defined, and guests must not rely on them. KVM answers them from the effective CPUID; MSHV and WHP pass them through from the hypervisor's guest view, so profile verification fails a host whose hypervisor presents a non-zero entry outside the profile's tables, with the new code E_CPU_UNLISTED (verification step 6, --cpu-fingerprint, and the support check at every cold boot and restore). This keeps host features out at no run-time cost, where zero results would add tens of registrations, about 1 ms per WHP restore. The support check's surface derives from the root's CPUID, which can show values a guest does not see, so step 6 requires it to report the guest view for these entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 55 ++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 104adbcc8..3370b045b 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID, and a zero result for each entry the host's CPUID enumerates that the effective CPUID does not list. Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); WHP reads zero natively for every other entry. CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -303,11 +303,11 @@ an SMP-restored VM for its lifetime. ## CPU profiles A CPU profile is the complete guest-visible CPU surface of one vendor and CPU -generation, shared by every backend. The guest sees only profile values, -never host passthrough, except for a fixed, code-defined set of VMM-owned -fields and the fields the time ABI owns; an entry the effective CPUID does -not list reads zero or a value derived from it (see **Effective CPUID**). -Profiles are data: derived mechanically by +generation, shared by every backend. In every governed entry the guest sees +only profile values, never host passthrough, except for a fixed, code-defined +set of VMM-owned fields and the fields the time ABI owns; the other entries +are reserved, and verification keeps host features out of them (see +**Effective CPUID**). Profiles are data: derived mechanically by intersecting host fingerprints (`openvmm --cpu-fingerprint`, the Firecracker `cpu-template-helper` workflow) per register, first within each backend and then across backends, reviewed, checked in at @@ -392,28 +392,26 @@ from the profile itself, which the backend looks up by `TimeAbiConfig::cpu_profile`. The governed entries are every entry an architectural enumeration reaches. -The other entries, which no architectural enumeration reaches, are: +The other entries, which only probing past an advertised maximum reaches, are +reserved: their values are implementation-defined, and guests must not rely +on them. They are: - a leaf above the maximum basic or extended leaf; - a subleaf past its leaf's reported maximum or terminator; - a `0xD` subleaf of a component that is not enabled. -They read no host data either. Each reads zero, or, on KVM, a value KVM -derives from the effective CPUID by the architecture's rules: above the -maximum basic leaf, Intel's out-of-range result (the highest basic leaf's); -past the last topology level, an invalid level with the x2APIC ID. - -- WHP already reads zero there. -- MSHV would answer from the hypervisor's own view, so it registers a zero - result for each such entry that the host's CPUID enumerates, at 3 to - 4.5 µs each in its partition build. - -The MSHV and WHP hardware tests sweep subleaves 0 to 63 of every indexed leaf -and the four leaves past each maximum; KVM's values follow from -`KVM_SET_CPUID2`'s rules. The boot check and the snapshot -record cover only the governed entries. A guest therefore reads the same -value for every entry on any host of the profile's generation with the same -backend, before and after a restore. +KVM answers them from the effective CPUID: zero, Intel's out-of-range result +(the highest basic leaf's) above the maximum basic leaf, or, past the last +topology level, an invalid level with the x2APIC ID. MSHV and WHP pass them +through from the hypervisor's own guest view. Verification therefore keeps +host features out of them, at no run-time cost: profile verification fails a +host whose hypervisor presents a non-zero entry outside the profile's tables +(`E_CPU_UNLISTED`, verification step 6). Registering a zero result for each +such entry instead would add tens of registrations to every partition setup, +about 1 ms per restore on WHP. On every fleet host both hypervisors present +zeros there: the fleet fingerprints, and the MSHV and WHP hardware sweeps of +subleaves 0 to 63 of every indexed leaf and four leaves past each maximum. +The boot check and the snapshot record cover only the governed entries. Informational fields: @@ -477,6 +475,14 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: exact-equality CPU contract. 5. The backend presents the effective CPUID: VP 0's CPUID for every governed leaf equals it under its masks (`E_CPU_SURFACE`). +6. On MSHV and WHP, which pass reserved entries through, the hypervisor + presents no non-zero CPUID entry outside the profile's tables, apart from + the identity range and the topology leaves (`E_CPU_UNLISTED`). + - `--cpu-fingerprint` checks the probe partition's guest view. + - The support check at every cold boot and restore checks the backend's + supported surface. That surface derives from the root's CPUID, which + can show values a guest does not see, so it must report the guest's + view for these entries, or the check would fail sound hosts. The catalog has three profiles, derived from the fingerprints of three bare-metal hosts (one per backend) and fifteen Azure hosts: @@ -1365,6 +1371,7 @@ rollback-safe: the guest continues, and OpenVMM logs the code. | `E_CPU_GENERATION` | Host CPU vendor, family, model, or stepping not in the profile | Cold boot, restore | | `E_PROFILE_UNSUPPORTED` | Backend lacks a feature, limit, XSAVE layout, MSR value, or feature-bank bit of the profile, or the host is not qualified | Cold boot, restore | | `E_CPU_SURFACE` | Recomputed effective CPUID differs from the recorded one, or VP 0's CPUID differs from the effective CPUID under its masks | Cold boot, restore | +| `E_CPU_UNLISTED` | On MSHV or WHP, the hypervisor presents a non-zero CPUID entry outside the profile's tables, a reserved entry that would expose a host feature | Cold boot, restore, `--cpu-fingerprint` | | `E_IDENTITY_ROUTING` | Backend cannot deliver the identity CPUID or MSRs | Cold boot, restore | | `E_TSC_SYNC_UNSUPPORTED` | Backend lacks the synchronized TSC set | Cold boot, restore | | `E_TSC_SCALING_ACTIVE` | The guest TSC would be scaled | Cold boot, restore | @@ -1421,7 +1428,7 @@ fail if any check fails. The time checks replace the `nonstop_tsc` check. | ID | Check | | --- | --- | | `H1` | The backend device or API is present and usable | -| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | +| `H2` | CPU fingerprint: vendor, family, model, stepping, microcode, host kernel or OS build, the generation name, and the profile that `auto` selects; an unmapped generation fails (`E_PROFILE_HOST_UNKNOWN`). `openvmm --hypervisor --cpu-fingerprint ` writes the fingerprint (`openvmm-cpu-fingerprint/v1`), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line (`E_PROFILE_HOST_UNKNOWN`, `E_PROFILE_UNSUPPORTED`, `E_CPU_UNLISTED`). The host's invariant TSC (`constant_tsc` and `nonstop_tsc` on Linux, the CPUID bit on Windows) is reported as evidence only | | `H3` | OpenVMM preflight in verification mode: profile support, identity routing, synchronized TSC set, no scaling, and both rates, without booting a guest | | `H4` | TSC rate stability against the host's clocks: interval agreement against an undisciplined clock (`CLOCK_MONOTONIC_RAW` or `QueryPerformanceCounter`) and the whole-window rate against `F_d` on the disciplined one (see [Rate stability](#rate-stability-h4)). On a Linux host the detail also reports the host clocksource, as evidence only | | `H5` | Host cross-CPU TSC skew: a pinned-thread probe over all host CPU pairs, `max_abs_offset_ns <= 1000` | From 9c6f82cc6b8ff61a72c284a86db2b027c8783e25 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 19:17:09 -0700 Subject: [PATCH 060/188] doc: check unlisted CPUID entries on VP 0's view at every start The boot-time E_CPU_UNLISTED check cannot use the backend's supported surface: it derives from the root's CPUID, which shows values that a guest does not see at exactly these entries (Intel PT's 0x14.1, RDT's 0xF.1 and 0x10.1 to 0x10.3), so it would fail sound hosts. Verification step 6 now runs as part of step 5's presented-CPUID check: MSHV and WHP also report VP 0's view at the entries the host's CPUID enumerates outside the profile's tables (cpu_profile::unlisted_cpuid_candidates), and each must read zero. KVM answers reserved entries from the effective CPUID itself and needs no check. The effective-CPUID paragraph names the extra reads, the reserved-entry paragraph limits the governed-only statement to the E_CPU_SURFACE comparison and the snapshot record, and restore step 8 lists E_CPU_UNLISTED among its preflight codes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 3370b045b..73f89f8f1 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -378,11 +378,12 @@ APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, records it, and recomputes it on restore, so the record never depends on how a backend enumerates or caches CPUID, and one profile and topology give the same record on every backend. At preflight, before any VP runs, each backend -reports VP 0's CPUID for exactly the governed leaves (KVM from the +reports VP 0's CPUID for the governed leaves (KVM from the `KVM_SET_CPUID2` table it programmed, MSHV with `get_cpuid_values` on VP 0, WHP from VP 0's register view), reading an entry without a subleaf at subleaf 0, and OpenVMM compares the report with the effective CPUID under its -masks. +masks. MSHV and WHP also report VP 0's view at the reserved entries the +host's CPUID enumerates (verification step 6). The effective CPUID is the single source of the guest's CPUID. Core passes it to the backend as `TimeAbiConfig::cpuid`, with the per-VP fields unmasked, @@ -411,7 +412,8 @@ such entry instead would add tens of registrations to every partition setup, about 1 ms per restore on WHP. On every fleet host both hypervisors present zeros there: the fleet fingerprints, and the MSHV and WHP hardware sweeps of subleaves 0 to 63 of every indexed leaf and four leaves past each maximum. -The boot check and the snapshot record cover only the governed entries. +The `E_CPU_SURFACE` comparison and the snapshot record cover only the +governed entries. Informational fields: @@ -479,10 +481,16 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: presents no non-zero CPUID entry outside the profile's tables, apart from the identity range and the topology leaves (`E_CPU_UNLISTED`). - `--cpu-fingerprint` checks the probe partition's guest view. - - The support check at every cold boot and restore checks the backend's - supported surface. That surface derives from the root's CPUID, which - can show values a guest does not see, so it must report the guest's - view for these entries, or the check would fail sound hosts. + - At every cold boot and restore, step 5's check covers it on VP 0's + view. The candidates are the entries that the host's CPUID enumerates + outside the profile's tables (`cpu_profile::unlisted_cpuid_candidates`), + each read at subleaf 0 if subleaf-independent. The backend reads them + with the governed leaves, a few reads more. The supported surface of + step 3 cannot serve: it derives from the root's CPUID, which shows + values that a guest does not see there, such as Intel PT's `0x14.1` and + RDT's `0xF.1` and `0x10.1` to `0x10.3`, so it would fail sound hosts. + - KVM answers reserved entries from the effective CPUID itself, so it + needs no check. The catalog has three profiles, derived from the fingerprints of three bare-metal hosts (one per backend) and fifteen Azure hosts: @@ -698,11 +706,11 @@ In the worker, with every VP stopped: 8. Create the partition with the profile CPUID, the identity leaves, and the time platform declaring `F = F_s` and `L = L_s`. Preflight the backend before any VP runs and before any VP state is restored: profile support - and effective CPUID (`E_PROFILE_UNSUPPORTED`, `E_CPU_SURFACE`), identity - routing (`E_IDENTITY_ROUTING`), the synchronized-set primitive - (`E_TSC_SYNC_UNSUPPORTED`), and no scaling (`E_TSC_SCALING_ACTIVE`). - MSHV's preflight writes `TimeFreeze` to probe the primitive and reads - VP 0's CPUID at its reset state. + and effective CPUID (`E_PROFILE_UNSUPPORTED`, `E_CPU_SURFACE`, + `E_CPU_UNLISTED`), identity routing (`E_IDENTITY_ROUTING`), the + synchronized-set primitive (`E_TSC_SYNC_UNSUPPORTED`), and no scaling + (`E_TSC_SCALING_ACTIVE`). MSHV's preflight writes `TimeFreeze` to probe + the primitive and reads VP 0's CPUID at its reset state. 9. Read `F_d` and `L_d` and apply the [rate policy](#tsc-rate-policy-and-lapic-rate-rule) (`E_TSC_RATE_UNAVAILABLE`, `E_TSC_RATE_IMPLAUSIBLE`, From d8cbf767be16733eef5c682ed00383cd766bf260 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 19:52:03 -0700 Subject: [PATCH 061/188] doc: register MSHV's per-VP topology results at VP creation mshv's fix (openvmm 12aed6e8b) closes the backend table's last TBD(mshv). MSHV does not implement 0xB for time ABI partitions and reads EDX as 0 on every VP, so OpenVMM registers the 0xB and 0x1F entries as per-VP intercept results with each VP's x2APIC ID in EDX. It does so as each VP is created, because the hypervisor refuses a per-VP result for a VP that does not exist yet (InvalidVpIndex). Leaf 1's initial APIC ID is the hypervisor's own and correct. With the 0xB terminator (openvmm acf50ab18), every vCPU of a 4-vCPU guest reads its own x2APIC ID in 0xB.0 to 0xB.2 on prometheus30 (MSHV) and prometheus32 (KVM), with no APIC ID mismatch lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 73f89f8f1..4c80fe39b 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions, so `0xB` `EDX` reads 0 on every VP, and Linux logs `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). Each VP's x2APIC ID must come from OpenVMM: TBD(mshv) | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | From f2d1aeaa9a0f4560cc6eddfb06bd3bdc019e237e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 19:55:00 -0700 Subject: [PATCH 062/188] doc: list WHP's 0x8000001E exit where a profile lists it virt_whp (dbe5e26f8) exits on every leaf with a per-VP APIC field that the effective CPUID lists: 1, 0xB, 0x1F, and the extended APIC ID leaf 0x8000001E. No v1 profile lists 0x1F or 0x8000001E, so every v1 profile still has 8 exits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 4c80fe39b..82afff075 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and `0x1F` where listed), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | From 34c24451da073e59c5cd61100642f775cd7c1d64 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 20:17:01 -0700 Subject: [PATCH 063/188] doc: cite only Intel PT's 0x14.1 as the root views' unlisted entry The profiles agent measured the MSHV and WHP root views on prometheus28 and prometheus30 with cpu_profile::cpuid::enumerate: RDT's 0xF.1 and 0x10.1 to 0x10.3 read zero there, against its earlier inference from /proc/cpuinfo. The one non-zero entry outside intel.skylake-sp.v1's tables is Intel PT's 0x14.1, which every probe partition reads as zero. Verification step 6 now cites only that entry. The conclusion is unchanged: the boot check cannot run on the root-derived supported surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 82afff075..648f29bf5 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -487,8 +487,9 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: each read at subleaf 0 if subleaf-independent. The backend reads them with the governed leaves, a few reads more. The supported surface of step 3 cannot serve: it derives from the root's CPUID, which shows - values that a guest does not see there, such as Intel PT's `0x14.1` and - RDT's `0xF.1` and `0x10.1` to `0x10.3`, so it would fail sound hosts. + values that a guest does not see there, so it would fail sound hosts. + On the Skylake-SP hosts, the MSHV and WHP roots read Intel PT's + `0x14.1` as non-zero, while every probe partition reads it as zero. - KVM answers reserved entries from the effective CPUID itself, so it needs no check. From 720933d0f5a6e5f625f8cd200390cf67862c848b Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 20:19:53 -0700 Subject: [PATCH 064/188] doc: state the evidence for reserved CPUID entries precisely MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The profiles agent points out that the fleet fingerprints never reach Intel PT's 0x14.1: a probe partition's 0x14.0 reads zero, so its own enumeration stops there. Only the host-derived candidates of the boot check reach it. The evidence is therefore: - WHP: its hardware sweep of VP 0 under the time ABI reads zero at every reserved entry, 0x14.1 included. - MSHV: its sweep reads zero with its zero results registered. MSHV registers a zero result for each reserved entry that the host's CPUID enumerates (15 or 16 per partition, about 65 µs), and its native view there, 0x14.1 included, is untested. The reserved-entry paragraph, verification step 6, and the MSHV backend cell now say so. Whether MSHV keeps its zero results depends on that native view, TBD(mshv). If MSHV dropped them and served a non-zero value there, every MSHV boot would fail E_CPU_UNLISTED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 648f29bf5..1d372f787 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries that the host's CPUID enumerates get zero results (TBD(mshv): kept only if the native view is not zero there), the others pass through from the hypervisor's guest view, and verification requires every one to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -403,17 +403,22 @@ on them. They are: KVM answers them from the effective CPUID: zero, Intel's out-of-range result (the highest basic leaf's) above the maximum basic leaf, or, past the last -topology level, an invalid level with the x2APIC ID. MSHV and WHP pass them -through from the hypervisor's own guest view. Verification therefore keeps -host features out of them, at no run-time cost: profile verification fails a -host whose hypervisor presents a non-zero entry outside the profile's tables -(`E_CPU_UNLISTED`, verification step 6). Registering a zero result for each -such entry instead would add tens of registrations to every partition setup, -about 1 ms per restore on WHP. On every fleet host both hypervisors present -zeros there: the fleet fingerprints, and the MSHV and WHP hardware sweeps of -subleaves 0 to 63 of every indexed leaf and four leaves past each maximum. -The `E_CPU_SURFACE` comparison and the snapshot record cover only the -governed entries. +topology level, an invalid level with the x2APIC ID. WHP passes them through +from its own guest view. MSHV registers a zero result for each one that the +host's CPUID enumerates (15 or 16 per partition, about 65 µs) and passes the +rest through; whether it keeps those zero results depends on its native view +of them, TBD(mshv). Verification keeps host features out of every entry +passed through, at no run-time cost: profile verification fails a host whose +hypervisor presents a non-zero entry outside the profile's tables +(`E_CPU_UNLISTED`, verification step 6). A zero result for each such entry +would instead add tens of registrations to every WHP partition setup, about +1 ms per restore. Under the time ABI, WHP's hardware sweep of VP 0 +(subleaves 0 to 63 of every indexed leaf and four leaves past each maximum) +reads zero at every reserved entry, including Intel PT's `0x14.1`, which the +Skylake-SP roots show as non-zero; MSHV's sweep reads zero with its zero +results registered. The fleet fingerprints cover only the entries that a +guest's own enumeration reaches. The `E_CPU_SURFACE` comparison and the +snapshot record cover only the governed entries. Informational fields: @@ -480,7 +485,8 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: 6. On MSHV and WHP, which pass reserved entries through, the hypervisor presents no non-zero CPUID entry outside the profile's tables, apart from the identity range and the topology leaves (`E_CPU_UNLISTED`). - - `--cpu-fingerprint` checks the probe partition's guest view. + - `--cpu-fingerprint` checks the probe partition's guest view, at the + entries its own enumeration reaches. - At every cold boot and restore, step 5's check covers it on VP 0's view. The candidates are the entries that the host's CPUID enumerates outside the profile's tables (`cpu_profile::unlisted_cpuid_candidates`), @@ -489,7 +495,8 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: step 3 cannot serve: it derives from the root's CPUID, which shows values that a guest does not see there, so it would fail sound hosts. On the Skylake-SP hosts, the MSHV and WHP roots read Intel PT's - `0x14.1` as non-zero, while every probe partition reads it as zero. + `0x14.1` as non-zero. A guest's own enumeration stops at `0x14.0`, + which reads zero, so only the host-derived candidates reach `0x14.1`. - KVM answers reserved entries from the effective CPUID itself, so it needs no check. From 2b6a7a3905d42be69c96fa0bc51db91eae0ea8e7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 21:27:43 -0700 Subject: [PATCH 065/188] doc: record the CPU profile by constants and compare it at restore The orchestrator requires no canonical encoding, decoding, or digest on the cold-boot or restore path, and a time ABI and CPU profile start-up overhead under 0.5 ms against the pre-profile head (e7ec0ca6c), measured on prometheus32 with the kvm agent's attribution harness. OpenVMM c63394446 implements it. - Snapshot manifest: SnapshotCpuProfile field 2 is the pinned profile's precomputed digest and field 3 its canonical encoding, both copied at capture. Field 4 is the effective CPUID as packed 44-byte little-endian entries (the leaf, a subleaf flag and the subleaf, four registers, four masks; 1 to 1,024 entries). Field 5, the digest of its former JSON form, is retired. - Restore step 2 validates only the record's shape (E_MANIFEST_TIME). Step 4 compares the recorded digest and document with the pinned record byte for byte (E_PROFILE_DIGEST). Verification step 4 says that nothing is hashed or decoded, and E_PROFILE_DIGEST's description follows. - Effective CPUID: snapshots hold the binary record, and the canonical JSON form serves tools only. - Performance: a start-up budget section with the 0.5 ms target and the measured costs, and the worker's new "time ABI CPU checks passed" line under attribution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 71 +++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 1d372f787..abecbe128 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -365,7 +365,7 @@ sees it. Host qualification measures them instead. Profiles also pin policy zero profiles' derivation policy). The effective guest CPUID is a pure function of the profile, the VM topology, and the time ABI's identity leaves. -**Effective CPUID.** The effective CPUID (`openvmm-effective-cpuid/v1`) +**Effective CPUID.** The effective CPUID lists exactly the governed leaves: every leaf and subleaf of the profile's `cpuid` table; the topology leaves `0xB` and `0x1F`, when they are within the maximum basic leaf, at the subleaves the VM's topology defines and the @@ -375,9 +375,11 @@ which Linux reads to end its enumeration; and the identity leaves `0x40000000..=0x40000005` with the explicit zero leaves. Each entry holds the four registers and their masks; the per-VP fields hold VP 0's APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, -records it, and recomputes it on restore, so the record never depends on how -a backend enumerates or caches CPUID, and one profile and topology give the -same record on every backend. At preflight, before any VP runs, each backend +records it as the manifest's packed binary record, and recomputes it on +restore, so the record never depends on how a backend enumerates or caches +CPUID, and one profile and topology give the same record on every backend. +Its canonical JSON form (`openvmm-effective-cpuid/v1`) serves tools, never +the start paths. At preflight, before any VP runs, each backend reports VP 0's CPUID for the governed leaves (KVM from the `KVM_SET_CPUID2` table it programmed, MSHV with `get_cpuid_values` on VP 0, WHP from VP 0's register view), reading an entry without a subleaf at @@ -476,10 +478,11 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: without a probe partition, which would add 2.6 to 5.5 ms per cold boot and restore on WHP; the full fingerprint stays in `--cpu-fingerprint`. 4. On restore only: this OpenVMM pins a profile with the same ID and digest, - and the recorded document hashes to it (`E_PROFILE_UNKNOWN`, - `E_PROFILE_DIGEST`), and the recomputed effective CPUID equals the - recorded one (`E_CPU_SURFACE`). The effective-CPUID record replaces the - exact-equality CPU contract. + and the recorded document is its canonical encoding + (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the recomputed effective + CPUID equals the recorded one (`E_CPU_SURFACE`). Both are comparisons + with precomputed or recomputed values; nothing is hashed or decoded. The + effective-CPUID record replaces the exact-equality CPU contract. 5. The backend presents the effective CPUID: VP 0's CPUID for every governed leaf equals it under its masks (`E_CPU_SURFACE`). 6. On MSHV and WHP, which pass reserved entries through, the hypervisor @@ -692,14 +695,15 @@ Before worker construction: recapture). 2. Validate the time contract and the CPU profile record: `time_abi_version` is 1, `tsc_tolerance_ppm` is 250, `F_s` is plausible, `L_s` is a backend - constant, identities are 16 bytes, `capture_generation < 2^32 - 1`, and - the embedded profile and effective-CPUID digests verify - (`E_MANIFEST_TIME`, `E_PROFILE_DIGEST`). + constant, identities are 16 bytes, `capture_generation < 2^32 - 1`, the + profile digest is 32 bytes, and the effective CPUID record is whole, + well-formed entries (`E_MANIFEST_TIME`). Validation computes no digest. 3. Require the destination backend to equal `source_hypervisor` (`E_BACKEND_MISMATCH`). -4. Require a pinned profile with the recorded ID and digest - (`E_PROFILE_UNKNOWN`, `E_PROFILE_DIGEST`), and the host's CPU generation - to be in it (`E_CPU_GENERATION`). +4. Require a pinned profile with the recorded ID and digest, whose canonical + encoding is the embedded document byte for byte (`E_PROFILE_UNKNOWN`, + `E_PROFILE_DIGEST`), both compared with the build's precomputed record, + and the host's CPU generation to be in it (`E_CPU_GENERATION`). 5. Preflight the downtime: sample the host clocks, select the source, and check the bounds (`E_DOWNTIME_*`, `E_HOST_IDENTITY`). The authoritative `D` is measured again in step 12. @@ -809,7 +813,10 @@ state units are quiesced, capture: within 100 µs of it, keeping the tightest of at most 64 samples (`E_TSC_ANCHOR`), plus the host identities (`E_HOST_IDENTITY`); and 3. records the declared rates, the CPU profile, the effective CPUID, and the - process's generation counter. + process's generation counter. The CPU profile record is the pinned + profile's precomputed digest and canonical encoding, copied, and the + effective CPUID is a packed binary record: capture encodes no document and + computes no digest. Each of these failures is a rollback-safe capture failure. @@ -845,12 +852,16 @@ reused. It adds two required fields: | Number | Field | Type | Content | | --- | --- | --- | --- | | 1 | `id` | `string` | Profile ID | -| 2 | `sha256` | `bytes` | Profile digest, 32 bytes | -| 3 | `profile` | `bytes` | Canonical profile encoding, at most 1 MiB | -| 4 | `effective_cpuid` | `bytes` | The [effective CPUID](#cpu-profiles) in its canonical encoding, `openvmm-effective-cpuid/v1` (compact canonical JSON): every governed leaf with its values and masks | -| 5 | `effective_cpuid_sha256` | `bytes` | Digest of `effective_cpuid`, 32 bytes | +| 2 | `sha256` | `bytes` | Profile digest, 32 bytes: the pinned profile's precomputed digest | +| 3 | `profile` | `bytes` | The pinned profile's canonical encoding, at most 1 MiB | +| 4 | `effective_cpuid` | `bytes` | The [effective CPUID](#cpu-profiles), 1 to 1,024 entries in its order (by leaf, then subleaf). Each entry is 44 bytes: eleven little-endian `u32` values, namely the leaf, then 1 and the subleaf (or 0 and 0 for an entry that applies to every subleaf), then the four registers `EAX` to `EDX`, then their four masks | | 6 | `capture_cpu_signature` | `u32` | CPUID.1:EAX of the capture host, for diagnostics | +Field 5, a digest of field 4's earlier canonical JSON form, is retired and +never reused. Restore checks the record by comparison: the digest and the +document against the build's pinned record, and the effective CPUID against +its recomputation, encoded the same way. + Other rules: - The effective command line contains no `tsc_early_khz=` or @@ -1381,7 +1392,7 @@ rollback-safe: the guest continues, and OpenVMM logs the code. | `E_MANIFEST_TIME` | Time contract missing or malformed, `time_abi_version` not 1, or tolerance not 250 | Restore | | `E_BACKEND_MISMATCH` | Snapshot taken on another backend | Restore | | `E_PROFILE_UNKNOWN` | Profile ID not pinned in this OpenVMM, or a restore's explicit `--cpu-profile` names another profile than the snapshot's | Cold boot, restore | -| `E_PROFILE_DIGEST` | Recorded, embedded, and pinned profile digests disagree, or the effective-CPUID digest is wrong | Restore | +| `E_PROFILE_DIGEST` | The recorded profile digest or embedded document is not the pinned profile's of the same ID | Restore | | `E_PROFILE_HOST_UNKNOWN` | `--cpu-profile auto` maps the host to no profile, or to profiles of more than one generation | Cold boot | | `E_PROFILE_TIME_BITS` | The profile is invalid or violates the CPU time bits | Cold boot, restore | | `E_CPU_GENERATION` | Host CPU vendor, family, model, or stepping not in the profile | Cold boot, restore | @@ -1600,6 +1611,22 @@ Expected effects: Expected wins are tracked separately and do not relax the gate. +**Start-up budget.** The time ABI and CPU profile work of a cold boot or +restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), +measured on prometheus32 (KVM) with the kvm agent's attribution harness; the +flip requires it. Neither path encodes, decodes, or hashes a profile document +or an effective CPUID: + +- the pinned profiles are compile-time constants with precomputed canonical + encodings and digests (`cpu_profile::pinned_record`), so selecting one + copies constants (about 30 µs); +- capture copies the pinned record and writes the effective CPUID as a packed + binary record; +- restore compares the recorded digest and document with the pinned record, + and the recomputed effective CPUID with the binary record, about 5 µs; +- `verify_support` costs 2 to 10 µs of CPU time, plus the backend's surface: + about 11 µs on MSHV with its cached host CPUID, and 0.1 to 0.3 ms on WHP. + **Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle profile records three exclusive time ABI restore phases: `restore.time_abi_clock` (restore steps 12 to 16, whose phases OpenVMM also @@ -1611,7 +1638,9 @@ selection to the arrival of the `0x605` acknowledgement). The `restore.guest_repair_gate` milestone still spans from the VP release to the release of the acknowledgement boundary. The guest's restore checks run after the acknowledgement, outside these phases. `nvx-time status` reports -`elapsed_us`, which covers the guest's readiness path and those checks. +`elapsed_us`, which covers the guest's readiness path and those checks. On +every cold boot and restore, the worker logs `time ABI CPU checks passed` +with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. ## Test matrix From 4c1c27a453643aec850cea1f64c6c121e9bd9b21 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 21:34:46 -0700 Subject: [PATCH 066/188] doc: align H4's resolution, agreement, and detail with the probe The ci agent checked "Rate stability (H4)" line by line against cd0a995 (host_time_probe.rs and doctor.py). Three places described the code inexactly. The text now matches it, and no code changes: - Resolution: the probe reads each clock 100,000 times back to back. Only a clock coarser than one read (two equal consecutive reads) takes the larger of its reported resolution and its smallest nonzero step, as for Hyper-V's reference TSC page (100 ns) and QPC (one 10 MHz tick). A clock that advances on every read keeps its reported resolution. - Agreement: u_i is a relative uncertainty compared with 0.25 ppm directly. The interval rates' spread is relative to their own whole-window rate against the same undisciplined clock, not to r. - Detail: it reports both clocks and the undisciplined clock's measured resolution, while the probe's output records both clocks' reported resolutions and observed steps. It also takes ci's precision on what H4 can catch: the Hyper-V reference page and QPC in a VM derive from the TSC, but the hypervisor keeps them continuous when it corrects TSC offsets. The HPET has its own oscillator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index abecbe128..dec12b9bc 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1475,32 +1475,37 @@ A host probe compares the TSC with two host clocks: A sample reads the TSC between two reads of a clock and keeps the tightest of up to 64 such brackets. Its uncertainty is half the bracket plus half the -clock's resolution. A clock's resolution is the larger of its reported -resolution and, when two consecutive reads return the same value, the -smallest step observed between consecutive reads. Hyper-V's reference TSC -page clock (`hyperv_clocksource_tsc_page`), the clocksource of MSHV roots and -the Azure MSHV runners, advances the Linux clocks in 100 ns steps, although -`clock_getres` reports 1 ns. `QueryPerformanceCounter`'s resolution is one -10 MHz tick. +clock's resolution. The probe reads each clock 100,000 times back to back. +If two consecutive reads ever return the same value, the clock is coarser +than one read, and its resolution is the larger of the reported resolution +and the smallest nonzero step between consecutive reads: 100 ns for Hyper-V's +reference TSC page clock (`hyperv_clocksource_tsc_page`), the clocksource of +MSHV roots and the Azure MSHV runners, which `clock_getres` reports as 1 ns, +and one 10 MHz tick (100 ns) for `QueryPerformanceCounter`. A clock that +advances on every read keeps its reported resolution, because its smallest +step is then only the time one read takes (for example +`CLOCK_MONOTONIC_RAW` on a host whose clocksource is `tsc`). The probe sleeps between samples, so the host's CPUs can idle: 3 samples 1 s apart in `validate-runner`, and 13 samples 10 s apart (120 s) in `doctor`. For each interval between consecutive samples, it computes against the undisciplined clock the rate `r_i`, in TSC cycles per second, and its -uncertainty `u_i`, the two samples' uncertainties divided by the interval. -`r` is the rate over the whole window against the disciplined clock. The -check passes if: +relative uncertainty `u_i`, the two samples' uncertainties divided by the +interval. `r` is the rate over the whole window against the disciplined +clock. The check passes if: -- every `u_i` is at most 0.25 ppm of `r`; otherwise the measurement is +- every `u_i` is at most 0.25 ppm; otherwise the measurement is inconclusive and fails; -- the interval rates agree within 1 ppm of `r` (largest minus smallest); and +- the interval rates agree within 1 ppm (largest minus smallest, relative to + their whole-window rate against the same clock); and - where `F_d` is known (in `doctor`, and in the `validate-runner` jobs that run `H3`), `|r - F_d|` is at most 100 ppm of `F_d`. The probe's output names both clocks, each with its role (rate or -stability), resolution, and observed step. The detail reports `r`, its -deviation from `F_d` in ppm, the agreement, the largest `u_i`, the stability -clock, and, on Linux, the host clocksource, which never gates. +stability), reported resolution, and observed step. The detail reports `r`, +its deviation from `F_d` in ppm, the agreement, the largest `u_i`, both +clocks and the undisciplined clock's measured resolution, and, on Linux, the +host clocksource, which never gates. Interval agreement uses the undisciplined clock because chrony's frequency updates move `CLOCK_MONOTONIC`'s rate between seconds, on Azure through the @@ -1519,8 +1524,10 @@ Hyper-V PTP clock: - The WHP hosts show 0.000 ppm with a residual of at most 0.07 µs over 118 s. A TSC that stops, slows, or is rescaled across idle misses the bound by -orders of magnitude, where the host's clocksource is independent of the TSC: -the Hyper-V reference page, the HPET, or `QueryPerformanceCounter` in a VM. +orders of magnitude where the host's clocksource does not follow the TSC's +steps: the Hyper-V reference page and `QueryPerformanceCounter` in a VM, +which the hypervisor computes from the TSC but keeps continuous when it +corrects TSC offsets, or the HPET, which has its own oscillator. Where the host clocksource is `tsc` itself (the Azure KVM runners and prometheus32), every kernel clock derives from the TSC, so no host clock can catch a TSC step; the guest warp probe (`H6` and the CI schedule) is the From 20f49983a4741906fed9846a2ab5814647650753 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 21:52:57 -0700 Subject: [PATCH 067/188] doc: start the restore's step 13 100 ms after the acknowledgement The guest agent measured on WHP (prometheus28, 512 MiB, 30 interleaved restores per cell, measurement-only builds) that step 13's checks, run right after the acknowledgement even at SCHED_IDLE, slow the restored guest. They contend with the readiness path while the VMM is still demand-faulting guest memory: wiring v5 was +4.2 to +6.1 ms at 2 vCPUs over the release guest, against a gate of 6.9. Delaying step 13 by 100 ms took it to +0.8 and +1.2 ms at 1 and 2 vCPUs. Delaying only the RCU release left the cost, so the checks themselves interfere. KVM showed no effect. Step 13 now starts 100 ms after the acknowledgement (or after step 10) at normal priority. The fail-fast bound stays about 100 ms plus the checks' run time, and an idle guest's detection moves from about 2 ms to about 100 ms. restore elapsed_us excludes the delay, so its budget is unchanged. The asynchronous boot checks keep SCHED_IDLE with the raise after 100 ms, for lack of boot data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index dec12b9bc..b81f54100 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1137,8 +1137,8 @@ is still pending after 30 s, its line reports `status=pending`, without At boot, `elapsed_us` covers the checks and the daemon start; at capture, the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it -covers steps 6 to 12 and the step 11 checks, but not the grace period wait -or the deferred `C7`. +covers steps 6 to 12 and the step 11 checks, but not step 13's 100 ms +delay, the grace period wait, or the deferred `C7`. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, @@ -1256,7 +1256,8 @@ acknowledgement. failure still emits its violation event and powers off with status 193. 12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. 13. After the acknowledgement, or after step 10 when none is required, signal - the daemon, which finishes the restore asynchronously: + the daemon, which finishes the restore asynchronously, starting 100 ms + later at normal priority: - it runs the step 11 checks; - it waits for the [grace period release](#rcu-grace-period-release); - it restores the values saved in steps 2 and 3 @@ -1276,17 +1277,24 @@ signals the daemon, so no other process starts on the readiness path. Each extra process costs 2 to 4 ms after a restore (fork and exec under demand faulting, measured on KVM). -Step 13 and the asynchronous boot checks start at `SCHED_IDLE`, so on few -vCPUs they never take a CPU from the restored or starting workload. Work -that has not finished 100 ms after the acknowledgement (or after step 10, -when none is required) or after shell-ready continues at normal priority. -This bounds how long a CPU-bound workload runs before a failing check powers -the guest off: about 100 ms plus the checks' own run time, where -`SCHED_IDLE` alone would allow about 0.5 s or more. On an idle KVM guest, a -failing restore check powers off within 1 to 3 ms. The watcher and the -discipline always run at normal priority, and stall suppression stays set -until the release. Repair failures emit a violation event and power off with -status 195. +Step 13 starts 100 ms after the acknowledgement (or after step 10, when none +is required) and runs at normal priority. Starting it earlier, even at +`SCHED_IDLE`, slows the restored guest: its checks contend with the +readiness path while the VMM is still demand-faulting guest memory. On WHP +(prometheus28, 512 MiB, measurement-only builds) that added 4 to 12 ms to +2-vCPU restores, whereas the 100 ms delay left them 0.8 and 1.2 ms over the +release guest at 1 and 2 vCPUs. A failing restore check therefore powers the +guest off about 100 ms plus the checks' own run time after the +acknowledgement, whether or not the guest is idle. + +The asynchronous boot checks start at `SCHED_IDLE`, so on few vCPUs they +never take a CPU from the starting workload. Work that has not finished +100 ms after shell-ready continues at normal priority. This bounds how long +a CPU-bound workload runs before a failing check powers the guest off: about +100 ms plus the checks' own run time, where `SCHED_IDLE` alone would allow +about 0.5 s or more. The watcher and the discipline always run at normal +priority, and stall suppression stays set until the release. Repair failures +emit a violation event and power off with status 195. ### RCU grace period release @@ -1613,7 +1621,7 @@ Expected effects: | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their `elapsed_us`, budgeted at 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` run asynchronously after the acknowledgement | +| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. From 57a1ba7131915b8a088e8d1e1d96f84fd5ebebc5 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 21:54:40 -0700 Subject: [PATCH 068/188] doc: name the one unpinned ARCH_CAPABILITIES bit that differs by backend The spec said a shared profile gives the same guest behavior on every backend. The profiles agent found one guest-visible difference outside the v1 mask: PSCHANGE_MC_NO (IA32_ARCH_CAPABILITIES bit 6) on Skylake-SP, which KVM always sets and MSHV shows as the hardware's 0. RSBA (bit 2) is 1 everywhere. The unpinned bits stay the backend's, because MSHV and WHP can neither set nor read them back, so verification could not check a pin. In Linux the bit only decides X86_BUG_ITLB_MULTIHIT, which matters only to KVM inside the guest. An NVX guest has no VMX, so only the sysfs itlb_multihit line differs, and no mitigation changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index b81f54100..3f7b2a0b0 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -327,7 +327,13 @@ the SPEC_CTRL family and MD_CLEAR from every backend on both generations Emerald Rapids profiles pin that value. A shared profile gives the same guest behavior on every backend; it does not make snapshots portable, because cross-backend restore is rejected -(`E_BACKEND_MISMATCH`). +(`E_BACKEND_MISMATCH`). The `IA32_ARCH_CAPABILITIES` bits outside the pinned +mask are the backend's, because MSHV and WHP can neither set nor read them +back. Measured, they differ only in `PSCHANGE_MC_NO` (bit 6) on Skylake-SP, +where KVM sets it and MSHV shows the hardware's 0. Linux uses that bit only +for `X86_BUG_ITLB_MULTIHIT`, which matters only to KVM inside the guest, so +it changes no mitigation of an NVX guest, which has no VMX. Only the sysfs +`itlb_multihit` line differs. **Format.** Schema `openvmm-cpu-profile/v1`. The canonical encoding is compact canonical JSON (object keys sorted by their UTF-8 bytes, no From d4dace632d5a7c4074cca7e10b61e23096ec1872 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:00:11 -0700 Subject: [PATCH 069/188] doc: budget the checks' CPU time, and report their wall time unbudgeted The guest agent's Q22: the asynchronous checks run at SCHED_IDLE, so their wall time mostly measures waiting behind the workload. On KVM (prometheus32, wiring v5, 3 boots each) the checker's CPU time (all threads) was 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs, against a 1-vCPU wall time of 23 to 26 ms. - The status line and the state file gain cpu_us (boot_cpu_us, capture_cpu_us, restore_cpu_us): the CPU time of the same work that elapsed_us covers, summed over every thread and process (CLOCK_PROCESS_CPUTIME_ID). It is the background cost to the workload, and the budget applies to it. - elapsed_us keeps its meaning as wall time. It is the fail-fast latency, which the 100 ms rules bound, and is no longer budgeted. - The KVM and MSHV budget becomes 2.5 ms plus 0.75 ms per additional vCPU, from the KVM data; the old 0.3 ms per vCPU was below the measurement at 8 vCPUs. WHP's is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 3f7b2a0b0..4eb1db190 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1118,7 +1118,7 @@ It then prints one line per recorded phase, in the order boot, capture, and restore, then the runtime line: ```text -NVX-TIME-ABI: v=1 phase= status= cpus= tsc_hz= lapic_hz= generation= elapsed_us= +NVX-TIME-ABI: v=1 phase= status= cpus= tsc_hz= lapic_hz= generation= elapsed_us= cpu_us= NVX-TIME-ABI: v=1 phase=runtime status= generation= discontinuities= offset_ns= uncertainty_ns= rejected_samples= last_sample_error= ``` @@ -1139,12 +1139,18 @@ line by its `phase`. `status` exits with status 0 when every phase line reports `ok`. If a check is still pending after 30 s, its line reports `status=pending`, without -`elapsed_us`, and the exit status is 1. +`elapsed_us` or `cpu_us`, and the exit status is 1. -At boot, `elapsed_us` covers the checks and the daemon start; at capture, +At boot, the checks' work is the checks and the daemon start; at capture, the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it -covers steps 6 to 12 and the step 11 checks, but not step 13's 100 ms -delay, the grace period wait, or the deferred `C7`. +is steps 6 to 12 and the step 11 checks, but not step 13's 100 ms delay, the +grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall +time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast +latency and is not budgeted. `cpu_us` is the CPU time it consumes, summed +over every thread and process that runs it (`CLOCK_PROCESS_CPUTIME_ID`); +it is the background cost to the workload, and the +[performance budget](#performance-expectations-and-acceptance-gate) applies +to it. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, @@ -1367,7 +1373,8 @@ every change, and `nvx-time status` reads it. The sandbox agent bind-mounts | `generation` | `g` | | `boot_status`, `capture_status`, `restore_status` | `pending` while the phase's check runs, then `ok`; absent until the phase first runs (a failed check powers the guest off). The capture keys hold the latest capture, and the restore keys this process's restore | | `boot_cpus`, `capture_cpus`, `restore_cpus` | Online CPUs the check covered | -| `boot_elapsed_us`, `capture_elapsed_us`, `restore_elapsed_us` | Its duration, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | +| `boot_elapsed_us`, `capture_elapsed_us`, `restore_elapsed_us` | Its wall time, as `elapsed_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | +| `boot_cpu_us`, `capture_cpu_us`, `restore_cpu_us` | Its CPU time, as `cpu_us` in the [status command](#conformance-checks-and-the-nvx-time-abi-marker) | | `capture_generation`, `restore_generation` | `g` when the check ran (the boot check's is 0) | | `boot_failures`, `capture_failures`, `restore_failures` | [Report-only mode](#conformance-checks-and-the-nvx-time-abi-marker) only: the check's failure count | | `tsc_hz`, `lapic_hz` | `F` and `L` | @@ -1625,7 +1632,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their `elapsed_us`, budgeted at 2.5 ms at one vCPU plus 0.3 ms per additional vCPU on KVM and MSHV, and 5 ms plus 0.6 ms per additional vCPU on WHP | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks: 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | @@ -1659,7 +1666,8 @@ selection to the arrival of the `0x605` acknowledgement). The `restore.guest_repair_gate` milestone still spans from the VP release to the release of the acknowledgement boundary. The guest's restore checks run after the acknowledgement, outside these phases. `nvx-time status` reports -`elapsed_us`, which covers the guest's readiness path and those checks. On +`elapsed_us` and `cpu_us`, which cover the guest's readiness path and those +checks. On every cold boot and restore, the worker logs `time ABI CPU checks passed` with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. From f5fe624b84fa4487aca9bbf8da3248d1b4c4f7ef Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:02:31 -0700 Subject: [PATCH 070/188] doc: keep the restore's delayed step 13 at SCHED_IDLE The orchestrator approved Q21, the guest agent's 100 ms delay of the deferred post-restore work, with one difference from 48c1afe4. The work (the restore checks, the RCU release, and C7) starts 100 ms after the 0x605 acknowledgement but stays at SCHED_IDLE, and it stays fail-fast. Step 13 now starts at SCHED_IDLE 100 ms after the acknowledgement, and work that has not finished 100 ms after it starts continues at normal priority, as the boot checks do 100 ms after shell-ready. A failing restore check powers an idle guest off about 100 ms after the acknowledgement, and a CPU-bound one at most about 200 ms plus the checks' run time after it. nvx-time status still waits for the deferred work, within its 30 s bound. The measured cost of starting earlier is stated as 6 to 12 ms at 2 vCPUs on WHP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 4eb1db190..81d63c2c2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1269,7 +1269,7 @@ acknowledgement. 12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. 13. After the acknowledgement, or after step 10 when none is required, signal the daemon, which finishes the restore asynchronously, starting 100 ms - later at normal priority: + later at `SCHED_IDLE`: - it runs the step 11 checks; - it waits for the [grace period release](#rcu-grace-period-release); - it restores the values saved in steps 2 and 3 @@ -1290,23 +1290,25 @@ extra process costs 2 to 4 ms after a restore (fork and exec under demand faulting, measured on KVM). Step 13 starts 100 ms after the acknowledgement (or after step 10, when none -is required) and runs at normal priority. Starting it earlier, even at -`SCHED_IDLE`, slows the restored guest: its checks contend with the -readiness path while the VMM is still demand-faulting guest memory. On WHP -(prometheus28, 512 MiB, measurement-only builds) that added 4 to 12 ms to -2-vCPU restores, whereas the 100 ms delay left them 0.8 and 1.2 ms over the -release guest at 1 and 2 vCPUs. A failing restore check therefore powers the -guest off about 100 ms plus the checks' own run time after the -acknowledgement, whether or not the guest is idle. - -The asynchronous boot checks start at `SCHED_IDLE`, so on few vCPUs they -never take a CPU from the starting workload. Work that has not finished -100 ms after shell-ready continues at normal priority. This bounds how long -a CPU-bound workload runs before a failing check powers the guest off: about -100 ms plus the checks' own run time, where `SCHED_IDLE` alone would allow -about 0.5 s or more. The watcher and the discipline always run at normal -priority, and stall suppression stays set until the release. Repair failures -emit a violation event and power off with status 195. +is required). Starting it earlier slows the restored guest: its checks +contend with the readiness path while the VMM is still demand-faulting guest +memory, even at `SCHED_IDLE`. On WHP (prometheus28, 512 MiB, measurement-only +builds) that added 6 to 12 ms to 2-vCPU restores, whereas the 100 ms delay +left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. + +Step 13 and the asynchronous boot checks run at `SCHED_IDLE`, so on few +vCPUs they never take a CPU from the restored or starting workload. Work +that has not finished 100 ms after it starts (step 13) or after shell-ready +(the boot checks) continues at normal priority. Every check stays +fail-fast: a failing restore check powers the guest off about 100 ms after +the acknowledgement on an idle guest, and at most about 200 ms plus the +checks' own run time on a CPU-bound one. A failing boot check powers it off +within about 100 ms plus the checks' run time after shell-ready, where +`SCHED_IDLE` alone would allow about 0.5 s or more. `nvx-time status` waits +for this deferred work, within its 30 s bound. The watcher and the +discipline always run at normal priority, and stall suppression stays set +until the release. Repair failures emit a violation event and power off +with status 195. ### RCU grace period release @@ -1634,7 +1636,7 @@ Expected effects: | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks: 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start 100 ms after the acknowledgement, after the readiness path | +| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. From 2b03e69005011afa7e3ce3da3dbab33c9b7b243c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:14:26 -0700 Subject: [PATCH 071/188] doc: state Skylake-SP's MMIO Stale Data status under the shared profile The kvm agent found in the attribution's guest logs (prometheus32) that under intel.skylake-sp.v1 a KVM guest reports MMIO Stale Data as "Vulnerable: Clear CPU buffers attempted, no microcode". Before the profiles it reported "Mitigation: Clear CPU buffers". Linux needs FB_CLEAR, or MD_CLEAR and FLUSH_L1D without MDS_NO, to confirm that VERW clears the fill buffers. The shared Skylake-SP profile drops FLUSH_L1D, because WHP cannot present it, and pins FB_CLEAR clear. The guest still clears the CPU buffers with VERW (MD_CLEAR), as its MDS and TAA status "Mitigation: Clear CPU buffers" shows, and the host has the microcode, so only the reported status differs. The spec now states this next to the list of what the Skylake-SP profile drops. The Ice Lake-SP and Emerald Rapids profiles, which set MDS_NO, and the development-only scope of Skylake-SP are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 81d63c2c2..c19652001 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -523,7 +523,14 @@ offers the same features. On Skylake-SP, used only by the bare-metal development hosts, the shared profile drops what one backend alone offers: PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use -the `7.0` EDX equivalents, which every backend presents). +the `7.0` EDX equivalents, which every backend presents). Without FLUSH_L1D, +and with `FB_CLEAR` clear, Linux cannot confirm the microcode that makes +VERW clear the fill buffers. It therefore reports MMIO Stale Data as +"Vulnerable: Clear CPU buffers attempted, no microcode" where a KVM-native +guest on the same host reports "Mitigation: Clear CPU buffers". The guest +still clears the CPU buffers with VERW (`MD_CLEAR`), as its MDS and TAA +status, "Mitigation: Clear CPU buffers", shows, and the host has the +microcode, so only the reported status differs. No profile pins the legacy P6 L2-cache MSRs; they raise #GP on every backend (see [MSRs](#msrs)). From b8b553809e9e9a9dcf5ad55bec4c0a3520ef20aa Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:16:23 -0700 Subject: [PATCH 072/188] doc: give the unlisted-CPUID candidate counts per host class The profiles agent measured unlisted_cpuid_candidates over each host class's root CPUID: 7 on the Skylake-SP hosts (0xF.1, 0x10.1 to 0x10.3, 0x12.1, 0x12.2, and 0x14.1) and 6 on the Azure 8370C and 8573C runners, whose roots report no Intel PT. Verification step 6 now gives the counts in place of "a few reads more". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c19652001..6ee8053d6 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -500,7 +500,10 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: view. The candidates are the entries that the host's CPUID enumerates outside the profile's tables (`cpu_profile::unlisted_cpuid_candidates`), each read at subleaf 0 if subleaf-independent. The backend reads them - with the governed leaves, a few reads more. The supported surface of + with the governed leaves: 7 more reads on the Skylake-SP hosts + (`0xF.1`, `0x10.1` to `0x10.3`, `0x12.1`, `0x12.2`, and `0x14.1`) and 6 + on the Azure 8370C and 8573C runners, whose roots report no Intel PT. + The supported surface of step 3 cannot serve: it derives from the root's CPUID, which shows values that a guest does not see there, so it would fail sound hosts. On the Skylake-SP hosts, the MSHV and WHP roots read Intel PT's From d160d65813ab145defde10254b03a5ebd39c132a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:16:55 -0700 Subject: [PATCH 073/188] doc: rewrap verification step 6 Whitespace only, after the candidate counts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6ee8053d6..61a0d17db 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -503,9 +503,9 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: with the governed leaves: 7 more reads on the Skylake-SP hosts (`0xF.1`, `0x10.1` to `0x10.3`, `0x12.1`, `0x12.2`, and `0x14.1`) and 6 on the Azure 8370C and 8573C runners, whose roots report no Intel PT. - The supported surface of - step 3 cannot serve: it derives from the root's CPUID, which shows - values that a guest does not see there, so it would fail sound hosts. + The supported surface of step 3 cannot serve: it derives from the + root's CPUID, which shows values that a guest does not see there, so + it would fail sound hosts. On the Skylake-SP hosts, the MSHV and WHP roots read Intel PT's `0x14.1` as non-zero. A guest's own enumeration stops at `0x14.0`, which reads zero, so only the host-derived candidates reach `0x14.1`. From 084b028c9263d93c95929e2519b12a3945211aca Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:18:39 -0700 Subject: [PATCH 074/188] doc: start the asynchronous boot checks 100 ms after shell-ready The guest agent measured on MSHV (prometheus30, guest-only A/B, 15 to 30 cold boots per cell) that the asynchronous boot checks, run right after shell-ready at SCHED_IDLE, cost about 3 ms of 2-vCPU cold boot. Starting them 100 ms later cut v5's 2-vCPU delta from +10 to +7 ms. It is the same second-vCPU contention as the WHP restore case that Q21 fixed for step 13, and KVM showed no clear effect. For consistency with the approved Q21, the boot checks now also start 100 ms after shell-ready at SCHED_IDLE, and move to normal priority 100 ms after they start. The fail-fast bounds are the restore's: about 100 ms after shell-ready on an idle guest, and at most about 200 ms plus the checks' run time on a busy one. Boot elapsed_us and cpu_us exclude the delay, and the capture's wait for pending boot checks is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 45 ++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 61a0d17db..f57066e08 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1058,10 +1058,10 @@ mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the [uncertainty bound](#uncertainty-bounds) and steps the clock to host UTC (`C12`), so the workload starts on host time; this step is not counted as a discontinuity. Init then reports shell-ready (or starts the workload, in -modes without a shell) and runs every other boot check asynchronously: at -`SCHED_IDLE` for the first 100 ms after shell-ready, and at normal priority -after that (see [Snapshot agent](#snapshot-agent)). The checks start the -daemon when they pass. Every check stays +modes without a shell) and runs every other boot check asynchronously, +starting 100 ms after shell-ready at `SCHED_IDLE`, and at normal priority +from 100 ms after they start (see [Snapshot agent](#snapshot-agent)). The +checks start the daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the workload is running. A guest that powers off before its checks finish skips them, which is why CI takes its evidence from `nvx-time status`, which waits for them. @@ -1151,7 +1151,8 @@ line by its `phase`. is still pending after 30 s, its line reports `status=pending`, without `elapsed_us` or `cpu_us`, and the exit status is 1. -At boot, the checks' work is the checks and the daemon start; at capture, +At boot, the checks' work is the checks and the daemon start, but not the +100 ms delay after shell-ready; at capture, the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it is steps 6 to 12 and the step 11 checks, but not step 13's 100 ms delay, the grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall @@ -1300,25 +1301,27 @@ extra process costs 2 to 4 ms after a restore (fork and exec under demand faulting, measured on KVM). Step 13 starts 100 ms after the acknowledgement (or after step 10, when none -is required). Starting it earlier slows the restored guest: its checks -contend with the readiness path while the VMM is still demand-faulting guest -memory, even at `SCHED_IDLE`. On WHP (prometheus28, 512 MiB, measurement-only -builds) that added 6 to 12 ms to 2-vCPU restores, whereas the 100 ms delay -left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. +is required), and the asynchronous boot checks start 100 ms after +shell-ready. Starting them earlier slows the guest: their checks contend +with the readiness path on the other vCPUs, even at `SCHED_IDLE`. On WHP +(prometheus28, 512 MiB, measurement-only builds), checks right after the +acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas the 100 ms +delay left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. On +MSHV (prometheus30), boot checks right after shell-ready added about 3 ms +to 2-vCPU cold boots. KVM showed no clear effect either way. Step 13 and the asynchronous boot checks run at `SCHED_IDLE`, so on few vCPUs they never take a CPU from the restored or starting workload. Work -that has not finished 100 ms after it starts (step 13) or after shell-ready -(the boot checks) continues at normal priority. Every check stays -fail-fast: a failing restore check powers the guest off about 100 ms after -the acknowledgement on an idle guest, and at most about 200 ms plus the -checks' own run time on a CPU-bound one. A failing boot check powers it off -within about 100 ms plus the checks' run time after shell-ready, where -`SCHED_IDLE` alone would allow about 0.5 s or more. `nvx-time status` waits -for this deferred work, within its 30 s bound. The watcher and the -discipline always run at normal priority, and stall suppression stays set -until the release. Repair failures emit a violation event and power off -with status 195. +that has not finished 100 ms after it starts continues at normal priority. +Every check stays fail-fast: a failing check powers the guest off about +100 ms after the acknowledgement or shell-ready on an idle guest, and at +most about 200 ms plus the checks' own run time after it on a CPU-bound +one, where `SCHED_IDLE` alone would allow about 0.5 s or more. The `elapsed_us` +and `cpu_us` of both exclude the 100 ms delay. `nvx-time status` waits for +this deferred work, within its 30 s bound. The watcher and the discipline +always run at normal priority, and stall suppression stays set until the +release. Repair failures emit a violation event and power off with status +195. ### RCU grace period release From febc584c4ecfd3d000454bff341ac93c88110755 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:23:22 -0700 Subject: [PATCH 075/188] doc: name why FB_CLEAR is clear on Skylake-SP The profiles agent traced the MMIO Stale Data status to its sources. FLUSH_L1D is absent because WHP does not offer it, and FB_CLEAR is absent because KVM does not present it on the Skylake-SP hosts, while MSHV and WHP do. Before the profiles, KVM reported "Mitigation: Clear CPU buffers" through FLUSH_L1D and MSHV through FB_CLEAR. With the shared profile every backend reports "Vulnerable: Clear CPU buffers attempted, no microcode", while the guest runs VERW at the same points. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index f57066e08..060fe0839 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -527,13 +527,15 @@ development hosts, the shared profile drops what one backend alone offers: PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use the `7.0` EDX equivalents, which every backend presents). Without FLUSH_L1D, -and with `FB_CLEAR` clear, Linux cannot confirm the microcode that makes -VERW clear the fill buffers. It therefore reports MMIO Stale Data as -"Vulnerable: Clear CPU buffers attempted, no microcode" where a KVM-native -guest on the same host reports "Mitigation: Clear CPU buffers". The guest -still clears the CPU buffers with VERW (`MD_CLEAR`), as its MDS and TAA -status, "Mitigation: Clear CPU buffers", shows, and the host has the -microcode, so only the reported status differs. +and with `FB_CLEAR` clear because KVM does not present it on these hosts, +Linux cannot confirm the microcode that makes VERW clear the fill buffers. +It therefore reports MMIO Stale Data as "Vulnerable: Clear CPU buffers +attempted, no microcode" on every backend. Before the profiles, KVM +reported "Mitigation: Clear CPU buffers" through FLUSH_L1D, and MSHV +through `FB_CLEAR`. The guest still clears the CPU buffers with VERW at the +same points (`MD_CLEAR`), as its MDS and TAA status, "Mitigation: Clear CPU +buffers", shows, and the host has the microcode, so only the reported status +differs. No profile pins the legacy P6 L2-cache MSRs; they raise #GP on every backend (see [MSRs](#msrs)). From 6cb9cbdd08f01739a38e62b7a45f59d15ae2a0c8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:35:08 -0700 Subject: [PATCH 076/188] doc: keep MSHV's reserved-entry zeros and batch its CPUID read-back Close both TBD(mshv) cells: MSHV keeps a zero result for each reserved entry its host's CPUID enumerates, because a subleaf that no result names reads the hypervisor's own value, which OpenVMM does not control. Its far probes (9204548b5) read zero where the hypervisor answers by itself: 2,510 probes on prometheus30 and 2,888 on azure-azlinux-5. MSHV reads every reported entry back in rep HvCallGetVpCpuidValues calls of up to 128 entries (985ad1534), falling back to one entry per call with a warning. Only the identity range's explicit zero leaves are reported as registered; preflight instead requires zero at six sentinel leaves. Start-up budget: the per-start profile work costs about 26 us on prometheus32 (profiles' audit). Name the codecs and digests that are offline-only. MSHV and WHP enumerate the host CPUID once per process without the hypervisor range (56213a814); MSHV's report takes 73 us bare metal and 51 us nested in one call, and WHP's about 0.78 ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 86 +++++++++++++++++++++++++++++------------- 1 file changed, 60 insertions(+), 26 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 060fe0839..bf9114c54 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -263,8 +263,8 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves, read back with `get_cpuid_values` at preflight; programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries that the host's CPUID enumerates get zero results (TBD(mshv): kept only if the native view is not zero there), the others pass through from the hypervisor's guest view, and verification requires every one to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves; preflight reads the identity leaves back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`); programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries that the host's CPUID enumerates get zero results, since a subleaf that no result names reads the hypervisor's own value; the others pass through from the hypervisor's guest view, and verification requires every one to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -385,13 +385,21 @@ records it as the manifest's packed binary record, and recomputes it on restore, so the record never depends on how a backend enumerates or caches CPUID, and one profile and topology give the same record on every backend. Its canonical JSON form (`openvmm-effective-cpuid/v1`) serves tools, never -the start paths. At preflight, before any VP runs, each backend -reports VP 0's CPUID for the governed leaves (KVM from the -`KVM_SET_CPUID2` table it programmed, MSHV with `get_cpuid_values` on VP 0, -WHP from VP 0's register view), reading an entry without a subleaf at -subleaf 0, and OpenVMM compares the report with the effective CPUID under its -masks. MSHV and WHP also report VP 0's view at the reserved entries the -host's CPUID enumerates (verification step 6). +the start paths. At preflight, before any VP runs, each backend reports +VP 0's CPUID for the governed leaves (KVM from the `KVM_SET_CPUID2` table it +programmed, MSHV by reading VP 0, WHP from VP 0's register view), reading an +entry without a subleaf at subleaf 0, and OpenVMM compares the report with +the effective CPUID under its masks. MSHV and WHP also report VP 0's view at +the reserved entries the host's CPUID enumerates (verification step 6). MSHV +reads every reported entry from the hypervisor, including those it +registered with full masks; only the identity range's explicit zero leaves +are reported as registered, and preflight instead requires VP 0 to read zero +at six sentinel leaves of the range (`0x40000006`, `0x40000081`, +`0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`; +`E_IDENTITY_ROUTING`). It batches the reads into rep +`HvCallGetVpCpuidValues` calls of up to 128 entries, with the VP's XFEM and +XSS and the registered results applied, and reads one entry per call, with a +warning, if a batched call fails. The effective CPUID is the single source of the guest's CPUID. Core passes it to the backend as `TimeAbiConfig::cpuid`, with the per-VP fields unmasked, @@ -414,19 +422,23 @@ KVM answers them from the effective CPUID: zero, Intel's out-of-range result topology level, an invalid level with the x2APIC ID. WHP passes them through from its own guest view. MSHV registers a zero result for each one that the host's CPUID enumerates (15 or 16 per partition, about 65 µs) and passes the -rest through; whether it keeps those zero results depends on its native view -of them, TBD(mshv). Verification keeps host features out of every entry -passed through, at no run-time cost: profile verification fails a host whose -hypervisor presents a non-zero entry outside the profile's tables -(`E_CPU_UNLISTED`, verification step 6). A zero result for each such entry -would instead add tens of registrations to every WHP partition setup, about -1 ms per restore. Under the time ABI, WHP's hardware sweep of VP 0 -(subleaves 0 to 63 of every indexed leaf and four leaves past each maximum) -reads zero at every reserved entry, including Intel PT's `0x14.1`, which the -Skylake-SP roots show as non-zero; MSHV's sweep reads zero with its zero -results registered. The fleet fingerprints cover only the entries that a -guest's own enumeration reaches. The `E_CPU_SURFACE` comparison and the -snapshot record cover only the governed entries. +rest through. It keeps those results: without them, a subleaf that no result +names reads the hypervisor's own value, which OpenVMM does not control. +Verification keeps host features out of every entry passed through, at no +run-time cost: profile verification fails a host whose hypervisor presents a +non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification +step 6). A zero result for each such entry would instead add tens of +registrations to every WHP partition setup, about 1 ms per restore. Under the +time ABI, WHP's hardware sweep of VP 0 (subleaves 0 to 63 of every indexed +leaf and four leaves past each maximum) reads zero at every reserved entry, +including Intel PT's `0x14.1`, which the Skylake-SP roots show as non-zero. +MSHV's sweep reads zero with its zero results registered, and so do its far +probes, where the hypervisor answers by itself: subleaves 64 to 255 of every +indexed leaf, the leaves past the sweep up to `0xff` and `0x800000ff`, the +hypervisor range up to `0x400001ff`, and nine distant leaves (2,510 probes +on prometheus30 and 2,888 on azure-azlinux-5). The fleet fingerprints cover +only the entries that a guest's own enumeration reaches. The `E_CPU_SURFACE` +comparison and the snapshot record cover only the governed entries. Informational fields: @@ -500,7 +512,7 @@ reports every violation at once, naming the leaf, subleaf, register, and bit: view. The candidates are the entries that the host's CPUID enumerates outside the profile's tables (`cpu_profile::unlisted_cpuid_candidates`), each read at subleaf 0 if subleaf-independent. The backend reads them - with the governed leaves: 7 more reads on the Skylake-SP hosts + with the governed leaves: 7 more entries on the Skylake-SP hosts (`0xF.1`, `0x10.1` to `0x10.3`, `0x12.1`, `0x12.2`, and `0x14.1`) and 6 on the Azure 8370C and 8573C runners, whose roots report no Intel PT. The supported surface of step 3 cannot serve: it derives from the @@ -1664,13 +1676,35 @@ or an effective CPUID: - the pinned profiles are compile-time constants with precomputed canonical encodings and digests (`cpu_profile::pinned_record`), so selecting one - copies constants (about 30 µs); + copies constants; - capture copies the pinned record and writes the effective CPUID as a packed binary record; - restore compares the recorded digest and document with the pinned record, and the recomputed effective CPUID with the binary record, about 5 µs; -- `verify_support` costs 2 to 10 µs of CPU time, plus the backend's surface: - about 11 µs on MSHV with its cached host CPUID, and 0.1 to 0.3 ms on WHP. +- `verify_support` costs 2 to 10 µs of CPU time, plus the backend's surface. + +On prometheus32, selection, the pinned-record and generation checks, the +effective CPUID merge, `verify_support`, and the unlisted-entry check cost +about 26 µs together (cold medians over 30 fresh processes). The codecs and +digests (`CpuProfile::{encode, decode, digest, digest_string, +to_pretty_json, from_pretty_json}` and `EffectiveCpuid::{encode, decode, +decode_verified, digest}`), `cpu_profile::{verify_profile_record, +restore_profile, pinned_profiles}`, and the fingerprint and derive paths are +offline-only: tools and tests use them, and no start path may. + +The backends add their own CPUID work: + +- MSHV and WHP enumerate the host CPUID once per process, for the supported + surface and the unlisted-entry candidates, without the hypervisor range, + which neither uses (`cpu_profile::cpuid::enumerate_basic_and_extended`). + Every CPUID instruction exits in a Hyper-V root, so a full enumeration + costs 70 to 91 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's + nested roots; skipping the range saves about a fifth. KVM takes its + surface from `KVM_GET_SUPPORTED_CPUID` and does not enumerate. +- MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on + prometheus30 and 51 µs on azure-azlinux-5, against 0.51 and 0.82 ms with + one call per entry. WHP's report takes 50 native reads, about 0.78 ms + (median) on prometheus28. **Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle profile records three exclusive time ABI restore phases: From c61b8482b801f62b2867f6b73261b96b161b401c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:38:41 -0700 Subject: [PATCH 077/188] doc: drop the CPU profile functions that profiles removed profiles' 8f4eab707 removes cpu_profile::restore_profile and verify_profile_record, whose checks now live in the restore preflight, so the start-up budget's offline-only list names only what remains: the codecs and digests, pinned_profiles, and the fingerprint and derive paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index bf9114c54..7a4ae128d 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1688,9 +1688,9 @@ effective CPUID merge, `verify_support`, and the unlisted-entry check cost about 26 µs together (cold medians over 30 fresh processes). The codecs and digests (`CpuProfile::{encode, decode, digest, digest_string, to_pretty_json, from_pretty_json}` and `EffectiveCpuid::{encode, decode, -decode_verified, digest}`), `cpu_profile::{verify_profile_record, -restore_profile, pinned_profiles}`, and the fingerprint and derive paths are -offline-only: tools and tests use them, and no start path may. +decode_verified, digest}`), `cpu_profile::pinned_profiles`, and the +fingerprint and derive paths are offline-only: tools, tests, and +`--cpu-fingerprint` use them, and no start path may. The backends add their own CPUID work: From bb0470f0b3162dc133c53a56626d744cc56f7ae5 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:43:21 -0700 Subject: [PATCH 078/188] doc: give MSHV's current zero-result count and enumeration cost Since the extended topology terminators became listed entries, MSHV's zero-fill registers 15 results on prometheus30 and 13 on azure-azlinux-5, not 15 or 16: about 50 to 60 us at 3.2 and 4.5 us per registration. MSHV's in-worker host CPUID enumeration takes 98 us on prometheus30, so the bare-metal range becomes 70 to 100 us. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 7a4ae128d..9bc14dc27 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -421,17 +421,18 @@ KVM answers them from the effective CPUID: zero, Intel's out-of-range result (the highest basic leaf's) above the maximum basic leaf, or, past the last topology level, an invalid level with the x2APIC ID. WHP passes them through from its own guest view. MSHV registers a zero result for each one that the -host's CPUID enumerates (15 or 16 per partition, about 65 µs) and passes the -rest through. It keeps those results: without them, a subleaf that no result -names reads the hypervisor's own value, which OpenVMM does not control. -Verification keeps host features out of every entry passed through, at no -run-time cost: profile verification fails a host whose hypervisor presents a -non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification -step 6). A zero result for each such entry would instead add tens of -registrations to every WHP partition setup, about 1 ms per restore. Under the -time ABI, WHP's hardware sweep of VP 0 (subleaves 0 to 63 of every indexed -leaf and four leaves past each maximum) reads zero at every reserved entry, -including Intel PT's `0x14.1`, which the Skylake-SP roots show as non-zero. +host's CPUID enumerates (15 on prometheus30 and 13 on azure-azlinux-5, about +50 to 60 µs) and passes the rest through. It keeps those results: without +them, a subleaf that no result names reads the hypervisor's own value, which +OpenVMM does not control. Verification keeps host features out of every entry +passed through, at no run-time cost: profile verification fails a host whose +hypervisor presents a non-zero entry outside the profile's tables +(`E_CPU_UNLISTED`, verification step 6). A zero result for each such entry +would instead add tens of registrations to every WHP partition setup, about +1 ms per restore. Under the time ABI, WHP's hardware sweep of VP 0 +(subleaves 0 to 63 of every indexed leaf and four leaves past each maximum) +reads zero at every reserved entry, including Intel PT's `0x14.1`, which the +Skylake-SP roots show as non-zero. MSHV's sweep reads zero with its zero results registered, and so do its far probes, where the hypervisor answers by itself: subleaves 64 to 255 of every indexed leaf, the leaves past the sweep up to `0xff` and `0x800000ff`, the @@ -1698,7 +1699,7 @@ The backends add their own CPUID work: surface and the unlisted-entry candidates, without the hypervisor range, which neither uses (`cpu_profile::cpuid::enumerate_basic_and_extended`). Every CPUID instruction exits in a Hyper-V root, so a full enumeration - costs 70 to 91 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's + costs 70 to 100 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's nested roots; skipping the range saves about a fifth. KVM takes its surface from `KVM_GET_SUPPORTED_CPUID` and does not enumerate. - MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on From a1bbd776c634d5d2eb5dde9ffa80f2cc0ccdfd7f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:02:12 -0700 Subject: [PATCH 079/188] doc: pass reserved CPUID entries through on MSHV, as on WHP mshv measured VP 0's native view with no zero results registered (05e56c2bc): zero at every host-enumerated reserved entry (15 on prometheus30, 13 on azure-azlinux-5) and at 2,755 and 3,261 sweep probes, near and far, including Intel PT's 0x14.1, on hypervisor builds 10.0.26100.30000 and 10.0.26100.9444. By the rule that held the TBD(mshv) cells, MSHV drops its zero results, so ed02c2a4's "keep" is reversed: MSHV and WHP both pass reserved entries through, and E_CPU_UNLISTED still fails a host whose hypervisor ever presents a non-zero one. The zero results would cost about 65 us per MSHV start. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 46 +++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 9bc14dc27..0c4dacba2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -264,7 +264,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves; preflight reads the identity leaves back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`); programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries that the host's CPUID enumerates get zero results, since a subleaf that no result names reads the hypervisor's own value; the others pass through from the hypervisor's guest view, and verification requires every one to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -419,27 +419,31 @@ on them. They are: KVM answers them from the effective CPUID: zero, Intel's out-of-range result (the highest basic leaf's) above the maximum basic leaf, or, past the last -topology level, an invalid level with the x2APIC ID. WHP passes them through -from its own guest view. MSHV registers a zero result for each one that the -host's CPUID enumerates (15 on prometheus30 and 13 on azure-azlinux-5, about -50 to 60 µs) and passes the rest through. It keeps those results: without -them, a subleaf that no result names reads the hypervisor's own value, which -OpenVMM does not control. Verification keeps host features out of every entry -passed through, at no run-time cost: profile verification fails a host whose -hypervisor presents a non-zero entry outside the profile's tables +topology level, an invalid level with the x2APIC ID. MSHV and WHP pass them +through from their own guest views. Verification keeps host features out of +every entry passed through, at no run-time cost: profile verification fails a +host whose hypervisor presents a non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification step 6). A zero result for each such entry -would instead add tens of registrations to every WHP partition setup, about -1 ms per restore. Under the time ABI, WHP's hardware sweep of VP 0 -(subleaves 0 to 63 of every indexed leaf and four leaves past each maximum) -reads zero at every reserved entry, including Intel PT's `0x14.1`, which the -Skylake-SP roots show as non-zero. -MSHV's sweep reads zero with its zero results registered, and so do its far -probes, where the hypervisor answers by itself: subleaves 64 to 255 of every -indexed leaf, the leaves past the sweep up to `0xff` and `0x800000ff`, the -hypervisor range up to `0x400001ff`, and nine distant leaves (2,510 probes -on prometheus30 and 2,888 on azure-azlinux-5). The fleet fingerprints cover -only the entries that a guest's own enumeration reaches. The `E_CPU_SURFACE` -comparison and the snapshot record cover only the governed entries. +would instead add registrations to every partition setup: tens on WHP, about +1 ms per restore, and 13 to 15 on MSHV, about 65 µs. Under the time ABI, +the MSHV and WHP guest views of VP 0 read zero at every reserved entry their +sweeps reach, including Intel PT's `0x14.1`, which the Skylake-SP roots show +as non-zero: + +- WHP's hardware sweep covers subleaves 0 to 63 of every indexed leaf and four + leaves past each maximum, on all three host types. +- MSHV's sweep, run with no result registered for any reserved entry, covers + the host-enumerated reserved entries (15 on prometheus30 and 13 on + azure-azlinux-5) plus 2,755 and 3,261 probes: the same near range as + WHP's, and far probes where the hypervisor answers by itself (subleaves 64 + to 255 of every indexed leaf, the leaves past the near range up to `0xff` + and `0x800000ff`, the hypervisor range up to `0x400001ff`, and nine distant + leaves). The hypervisor builds were 10.0.26100.30000 and, nested, + 10.0.26100.9444. + +The fleet fingerprints cover only the entries that a guest's own enumeration +reaches. The `E_CPU_SURFACE` comparison and the snapshot record cover only +the governed entries. Informational fields: From ebf954bfd9e6f1d35fd54e9b07b2977620cee34c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:26:19 -0700 Subject: [PATCH 080/188] doc: run restore steps 14 and 15 concurrently Core's 3ed78a903 joins the LAPIC advance with the VM time and RTC advance. They touch disjoint state because no time ABI LAPIC reads VM time: KVM's in-kernel LAPIC, the MSHV hypervisor's LAPIC (now named in the LAPIC rate row), and WHP's offloaded APIC. Step 16 waits for both. On MSHV this cut restore.time_abi_clock from 304 to 216 us at 1 vCPU and from 531 to 420 us at 8 vCPUs. The attribution paragraph now says that lapic_us and vm_time_us overlap, each measured from the end of the TSC set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 0c4dacba2..c0a1cbf3d 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -268,7 +268,7 @@ settled cell on the registered hosts. | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | -| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | +| LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | The hypervisor's LAPIC at 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | | TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile through a CPUID intercept result: Azure's nested MSHV cannot offer the bit to its guests, although the host OS sees an invariant TSC, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: on Azure, WHP cannot offer it through the feature banks (bank 1 lacks `TscInvariant`), although the host OS sees an invariant TSC. There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | @@ -802,7 +802,10 @@ In the worker, with every VP stopped: timer (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`) before any VP runs, so restore takes no separate pass over the VPs to check them. 15. Advance VM time by `D` and the RTC's UTC by `D` (milliseconds). The PIT - catches up from its saved VM-time cursor; it is idle. + catches up from its saved VM-time cursor; it is idle. Steps 14 and 15 + touch disjoint state, because no time ABI LAPIC reads VM time (see the + LAPIC rate row of [Backend obligations](#backend-obligations)), so + OpenVMM runs them concurrently and waits for both before step 16. 16. Seal the time fields of the restore packet: `D`, its source, the rate deviation, `g`, and the test-hook flag. 17. Start the state units, with host input gated when the restore requires an @@ -1714,8 +1717,9 @@ The backends add their own CPUID work: **Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle profile records three exclusive time ABI restore phases: `restore.time_abi_clock` (restore steps 12 to 16, whose phases OpenVMM also -logs as `time ABI restore clock` with `tsc_set_us`, `lapic_us`, -`vm_time_us`, and `total_us`), `restore.guest_resume` +logs as `time ABI restore clock`: `tsc_set_us`, then `lapic_us` and +`vm_time_us`, which overlap and are each measured from the end of the TSC +set, and `total_us`), `restore.guest_resume` (from the VP release to the guest's first selection of the restore packet), and, for a restore with `ACK_REQUIRED`, `restore.guest_repair` (from that selection to the arrival of the `0x605` acknowledgement). The From 11227f0043d690f5691d5a7cd03d8f87c06ef172 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:31:43 -0700 Subject: [PATCH 081/188] doc: give WHP's measured cost of reserved-entry zero results whp measured CpuidResultList2 on prometheus28: about 5 us per entry at partition setup (+14 us for 7 entries, +267 us for 50, +1.0 ms for 200), and a fleet host has only 6 or 7 candidates, so zero results would cost about 35 us on WHP, not "tens, about 1 ms per restore". The paragraph now says that neither MSHV nor WHP programs them because both read zero natively, gives both costs, and drops "at no run-time cost", since the candidate reads are part of every start. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c0a1cbf3d..6dd73f585 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -420,15 +420,16 @@ on them. They are: KVM answers them from the effective CPUID: zero, Intel's out-of-range result (the highest basic leaf's) above the maximum basic leaf, or, past the last topology level, an invalid level with the x2APIC ID. MSHV and WHP pass them -through from their own guest views. Verification keeps host features out of -every entry passed through, at no run-time cost: profile verification fails a -host whose hypervisor presents a non-zero entry outside the profile's tables -(`E_CPU_UNLISTED`, verification step 6). A zero result for each such entry -would instead add registrations to every partition setup: tens on WHP, about -1 ms per restore, and 13 to 15 on MSHV, about 65 µs. Under the time ABI, -the MSHV and WHP guest views of VP 0 read zero at every reserved entry their -sweeps reach, including Intel PT's `0x14.1`, which the Skylake-SP roots show -as non-zero: +through from their own guest views. Both read zero at every reserved entry +their sweeps reach (below), so neither programs zero results there, which +would cost about 5 µs per entry at partition setup: about 35 µs on WHP for +the 6 or 7 candidates of a fleet host, and about 65 µs on MSHV for its former +13 to 15. Verification keeps host features out of every entry passed +through: profile verification fails a host whose hypervisor presents a +non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification +step 6), checked on VP 0's view at every start. Under the time ABI, the MSHV +and WHP sweeps of VP 0 read zero there, including Intel PT's `0x14.1`, which +the Skylake-SP roots show as non-zero: - WHP's hardware sweep covers subleaves 0 to 63 of every indexed leaf and four leaves past each maximum, on all three host types. From dc44ef6c6a2ed3605e89883f9179aaac9cd78e64 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:34:54 -0700 Subject: [PATCH 082/188] doc: require delivery of a pending LAPIC vector to a halted VP A one-shot timer that expires during the downtime queues its vector in the restored LAPIC state (step 14), and a snapshot can hold pending vectors too. KVM's KVM_SET_LAPIC raises KVM_REQ_EVENT and wakes the vCPU, but MSHV and WHP do not wake a halted VP for an IRR bit written with its LAPIC state. Their backends therefore assert the highest pending edge-triggered vector to the VP's own APIC after the write and again before the VP next runs (mshv's b846c4182 and 1affa4447, whp's fe3fdb914). State this as a backend obligation, with a new row in the table and a sentence in the LAPIC rule. The evidence is from busy 8-vCPU guests: 29 of 60 time ABI restores on MSHV and 5 of 12 on WHP left a CPU halted with its vector pending, and 0 of 300 and 0 of 12 do with the fix. As whp's corrected R11 shows, step 14's place after the synchronized set does not prevent this; the re-assertion does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6dd73f585..3aabc2684 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -269,6 +269,7 @@ settled cell on the registered hosts. | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | | LAPIC rate `L` | In-kernel LAPIC at 1 GHz; `KVM_CAP_X86_APIC_BUS_CYCLES_NS` never set | The hypervisor's LAPIC at 200 MHz | Offloaded APIC at its fixed 200 MHz, verified at preflight (setting `InterruptClockFrequency` is not supported); the emulated APIC is not used | +| Pending LAPIC vector after a LAPIC state write | Nothing more: `KVM_SET_LAPIC` raises `KVM_REQ_EVENT`, which wakes a halted vCPU for a deliverable vector | Writing the LAPIC state does not wake a halted VP, so the backend asserts the highest pending edge-triggered vector (16 and up) to the VP's own APIC through the hypervisor's interrupt path after the write, and again when the VP next runs; level-triggered vectors stay with the IOAPIC | Writing the offloaded LAPIC state does not reliably wake a halted VP, so the backend asserts the highest pending edge-triggered vector with `WHvRequestInterrupt` to the VP's own APIC ID after the write, and again before the VP next runs | | TSC-deadline and `TSC_ADJUST` hidden; both MSRs raise #GP | CPUID bits cleared; the MSR filter also denies `IA32_TSC_ADJUST` (`0x3b`) and `IA32_TSC_DEADLINE` (`0x6e0`), which KVM would otherwise serve (reading 0 and ignoring writes to `0x6e0`), and OpenVMM raises #GP | Feature-bank bits `tsc_deadline_tmr_support`, `tsc_adjust_support`, and `a_count_m_count_support` cleared, and CPUID bits cleared; the hypervisor then raises #GP for reads and writes of both MSRs on every CPU | Feature-bank bits `TscDeadlineTmr`, `TscAdjust`, and `ACountMCount` cleared, and CPUID bits cleared; both MSRs exit to OpenVMM, which raises #GP on every CPU | | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile through a CPUID intercept result: Azure's nested MSHV cannot offer the bit to its guests, although the host OS sees an invariant TSC, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: on Azure, WHP cannot offer it through the feature banks (bank 1 lacks `TscInvariant`), although the host OS sees an invariant TSC. There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | @@ -605,9 +606,16 @@ ticks = floor(D * L / 1_000_000_000 / divide) where `divide` is the divide-configuration value (1 to 128). If `ticks` is at least the current count, the count becomes 0 and the timer interrupt is queued in the restored LAPIC state unless the LVT is masked; otherwise the current -count decreases by `ticks`. A periodic or TSC-deadline LAPIC timer is never -valid: capture and restore reject one that is armed (`E_LAPIC_PERIODIC`, -`E_LAPIC_TSC_DEADLINE`). +count decreases by `ticks`. A vector queued this way, or one pending in the +restored state, must reach its VP when the VP is released, even if the VP is +halted. MSHV and WHP do not wake a halted VP for an IRR bit written with its +LAPIC state, so their backends assert the vector again (see +[Backend obligations](#backend-obligations)). Without that, on busy 8-vCPU +guests, 29 of 60 time ABI restores on MSHV (prometheus30) and 5 of 12 on WHP +(prometheus28) left a CPU halted with its timer vector pending until another +interrupt woke it, about 2 s later or never; with it, 0 of 300 and 0 of 12. +A periodic or TSC-deadline LAPIC timer is never valid: capture and restore +reject one that is armed (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`). ## Cross-vCPU skew bound From 63d0a81dcd5322d35b15db47af5c1799ac9a5abf Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:36:35 -0700 Subject: [PATCH 083/188] doc: say that CI reports cpu_us against its budget without gating ci's 369c45e reports each phase's cpu_us range and a _cpu_over_budget count from the status line, and gates on neither cpu_us nor elapsed_us. The orchestrator made the A/B comparison the performance gate. The status section and the expected-effects row now say that the cpu_us budget is reported against, not gated, so "applies" and "holds" are not read as an enforced limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 3aabc2684..86b6cb0a7 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1190,9 +1190,10 @@ grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast latency and is not budgeted. `cpu_us` is the CPU time it consumes, summed over every thread and process that runs it (`CLOCK_PROCESS_CPUTIME_ID`); -it is the background cost to the workload, and the -[performance budget](#performance-expectations-and-acceptance-gate) applies -to it. +it is the background cost to the workload, which CI reports against the +budget in +[Performance expectations](#performance-expectations-and-acceptance-gate) +without gating on it. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, @@ -1678,7 +1679,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks: 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | From ec6a839989cb66215802520396763f260c52bd57 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:41:47 -0700 Subject: [PATCH 084/188] doc: give MSHV's measured savings from pass-through and the range skip mshv measured both changes per boot and restore. Dropping the zero results (05e56c2bc) took registration from 244 to 195 us on prometheus30 and from 364 to 301 us nested, so they cost 49 to 63 us, not "about 65 us". Skipping the hypervisor range (277ed844b) took the host enumeration from 96 to 49 us and from 305 to 220 us, more than the "about a fifth" estimated from the query count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 86b6cb0a7..c515a4f66 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -424,7 +424,7 @@ topology level, an invalid level with the x2APIC ID. MSHV and WHP pass them through from their own guest views. Both read zero at every reserved entry their sweeps reach (below), so neither programs zero results there, which would cost about 5 µs per entry at partition setup: about 35 µs on WHP for -the 6 or 7 candidates of a fleet host, and about 65 µs on MSHV for its former +the 6 or 7 candidates of a fleet host, and 49 to 63 µs on MSHV for its former 13 to 15. Verification keeps host features out of every entry passed through: profile verification fails a host whose hypervisor presents a non-zero entry outside the profile's tables (`E_CPU_UNLISTED`, verification @@ -1717,8 +1717,10 @@ The backends add their own CPUID work: which neither uses (`cpu_profile::cpuid::enumerate_basic_and_extended`). Every CPUID instruction exits in a Hyper-V root, so a full enumeration costs 70 to 100 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's - nested roots; skipping the range saves about a fifth. KVM takes its - surface from `KVM_GET_SUPPORTED_CPUID` and does not enumerate. + nested roots. Skipping the range took MSHV's from 96 to 49 µs on + prometheus30 and from 305 to 220 µs nested, more than its share of the + queries suggests. KVM takes its surface from `KVM_GET_SUPPORTED_CPUID` and + does not enumerate. - MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on prometheus30 and 51 µs on azure-azlinux-5, against 0.51 and 0.82 ms with one call per entry. WHP's report takes 50 native reads, about 0.78 ms From 3c6366dcb19ee53e4483049bd3929e822e13f4bd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:45:07 -0700 Subject: [PATCH 085/188] doc: say that WHP presents PSCHANGE_MC_NO on Skylake-SP too whp's guest reads at 06:31Z show WHP presenting ARCH_CAPABILITIES 0x44 (RSBA and PSCHANGE_MC_NO) on prometheus28, the same Skylake-SP hardware that MSHV presents as 0. So the unpinned difference is KVM and WHP presenting 1 against MSHV presenting 0, not KVM against "the hardware's 0" (profiles' refinement). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c515a4f66..589221886 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -331,10 +331,10 @@ make snapshots portable, because cross-backend restore is rejected (`E_BACKEND_MISMATCH`). The `IA32_ARCH_CAPABILITIES` bits outside the pinned mask are the backend's, because MSHV and WHP can neither set nor read them back. Measured, they differ only in `PSCHANGE_MC_NO` (bit 6) on Skylake-SP, -where KVM sets it and MSHV shows the hardware's 0. Linux uses that bit only -for `X86_BUG_ITLB_MULTIHIT`, which matters only to KVM inside the guest, so -it changes no mitigation of an NVX guest, which has no VMX. Only the sysfs -`itlb_multihit` line differs. +where KVM and WHP present 1 and MSHV presents 0 on the same hardware. Linux +uses that bit only for `X86_BUG_ITLB_MULTIHIT`, which matters only to KVM +inside the guest, so it changes no mitigation of an NVX guest, which has no +VMX. Only the sysfs `itlb_multihit` line differs. **Format.** Schema `openvmm-cpu-profile/v1`. The canonical encoding is compact canonical JSON (object keys sorted by their UTF-8 bytes, no From e7f174570ceb93db55169263d029b8fdf10b163d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:47:54 -0700 Subject: [PATCH 086/188] doc: name the matrix driver as the cpu_us budget's enforcer The orchestrator decided at 22:14 PDT, relayed by ci: CI reports cpu_us and elapsed_us without gating, the performance gate stays the A/B comparison outside CI, and the fleet's matrix driver enforces the spec's cpu_us budget during fleet validation (p6), which kvm is adding. The status section, the expected-effects row, and the conformance matrix now say so. dcd4e2ac had stated only CI's part. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 589221886..b5f959bc2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1190,10 +1190,10 @@ grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast latency and is not budgeted. `cpu_us` is the CPU time it consumes, summed over every thread and process that runs it (`CLOCK_PROCESS_CPUTIME_ID`); -it is the background cost to the workload, which CI reports against the -budget in +it is the background cost to the workload, which the fleet matrix driver +checks against the budget in [Performance expectations](#performance-expectations-and-acceptance-gate) -without gating on it. +during validation; CI reports it without gating on it. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, @@ -1679,7 +1679,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | @@ -1762,7 +1762,10 @@ matrix driver run `/sbin/nvx-time status` over the console after shell-ready and after every restore. They require exit status 0 and, after a restore, a `phase=restore` line with `status=ok`, the restored `generation`, and `cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with -`generation=0`. The fleet runs them on +`generation=0`. During fleet validation (`p6`), the matrix driver also +requires each phase's `cpu_us` to be within the budget in +[Performance expectations](#performance-expectations-and-acceptance-gate); +CI reports it without gating on it. The fleet runs them on the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on From 55e28d8f547563950f212f1aa0f7d4f202b506e8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:52:28 -0700 Subject: [PATCH 087/188] doc: settle guest's Q23 to Q26 on the boot anchor and cpu_us Q23: the asynchronous boot checks start 100 ms after init's time ABI boot step (nvx-time boot), the one point every mode reaches. A mode that execs an agent reports readiness later, on its own protocol. In shell mode the boot step is 15 to 44 ms before shell-ready, so the checks start about 55 to 85 ms after shell-ready. Q24: restore cpu_us counts only step 13's part, the daemon's preparation and the step 11 checks. Steps 6 to 12 are the readiness path, which the restore latency gate already measures, so cpu_us is the background cost to the workload. elapsed_us keeps its scope. Q25: the NVX kernel turns on CONFIG_IRQ_TIME_ACCOUNTING, so interrupts are not charged to the checks. cpu_us still scales with the host processor's clock; prometheus32's 800 MHz idle explains KVM's 4.0 ms one-vCPU boot. Q26: cpu_us counts only the nvx-time processes, not the shell's children in steps 9 and 10, whose wall time elapsed_us covers. The budget row now gives wiring v6's medians and keeps the values TBD(guest) for every backend, to be set from the next guest build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index b5f959bc2..de7753b8a 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1090,8 +1090,12 @@ mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the (`C12`), so the workload starts on host time; this step is not counted as a discontinuity. Init then reports shell-ready (or starts the workload, in modes without a shell) and runs every other boot check asynchronously, -starting 100 ms after shell-ready at `SCHED_IDLE`, and at normal priority -from 100 ms after they start (see [Snapshot agent](#snapshot-agent)). The +starting 100 ms after its time ABI boot step (`nvx-time boot`) at +`SCHED_IDLE`, and at normal priority from 100 ms after they start (see +[Snapshot agent](#snapshot-agent)). The boot step is the anchor because +every mode reaches it, whereas a mode that execs an agent reports readiness +later, on the agent's own protocol. In shell mode the boot step comes 15 to +44 ms before shell-ready, so the checks start about 55 to 85 ms after it. The checks start the daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the workload is running. A guest that powers off before its checks finish skips them, which @@ -1183,15 +1187,21 @@ is still pending after 30 s, its line reports `status=pending`, without `elapsed_us` or `cpu_us`, and the exit status is 1. At boot, the checks' work is the checks and the daemon start, but not the -100 ms delay after shell-ready; at capture, +100 ms delay after the boot step; at capture, the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it is steps 6 to 12 and the step 11 checks, but not step 13's 100 ms delay, the grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast -latency and is not budgeted. `cpu_us` is the CPU time it consumes, summed -over every thread and process that runs it (`CLOCK_PROCESS_CPUTIME_ID`); -it is the background cost to the workload, which the fleet matrix driver -checks against the budget in +latency and is not budgeted. `cpu_us` is the CPU time that the `nvx-time` +processes spend on it (`CLOCK_PROCESS_CPUTIME_ID`, summed over their +threads); other processes, such as the shell's children in steps 9 and 10, +are not counted. At restore, `cpu_us` counts only step 13's part: the +daemon's preparation and the step 11 checks. Steps 6 to 12 are the readiness +path, whose cost the restore latency gate already measures, so `cpu_us` is +the background cost to the workload. The NVX kernel does not charge +interrupt time to it (`CONFIG_IRQ_TIME_ACCOUNTING`), but it still scales with +the host processor's clock: a host idling at a low frequency runs the checks +more slowly. The fleet matrix driver checks `cpu_us` against the budget in [Performance expectations](#performance-expectations-and-acceptance-gate) during validation; CI reports it without gating on it. @@ -1333,8 +1343,8 @@ extra process costs 2 to 4 ms after a restore (fork and exec under demand faulting, measured on KVM). Step 13 starts 100 ms after the acknowledgement (or after step 10, when none -is required), and the asynchronous boot checks start 100 ms after -shell-ready. Starting them earlier slows the guest: their checks contend +is required), and the asynchronous boot checks start 100 ms after the boot +step. Starting them earlier slows the guest: their checks contend with the readiness path on the other vCPUs, even at `SCHED_IDLE`. On WHP (prometheus28, 512 MiB, measurement-only builds), checks right after the acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas the 100 ms @@ -1346,7 +1356,7 @@ Step 13 and the asynchronous boot checks run at `SCHED_IDLE`, so on few vCPUs they never take a CPU from the restored or starting workload. Work that has not finished 100 ms after it starts continues at normal priority. Every check stays fail-fast: a failing check powers the guest off about -100 ms after the acknowledgement or shell-ready on an idle guest, and at +100 ms after the acknowledgement or the boot step on an idle guest, and at most about 200 ms plus the checks' own run time after it on a CPU-bound one, where `SCHED_IDLE` alone would allow about 0.5 s or more. The `elapsed_us` and `cpu_us` of both exclude the 100 ms delay. `nvx-time status` waits for @@ -1679,7 +1689,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV. That is measured on KVM (prometheus32, wiring v5): 1.1, 2.3 to 3.2, 2.4 to 3.6, and 4.6 to 7.2 ms at 1, 2, 4, and 8 vCPUs. On WHP it is 5 ms plus 1.5 ms per additional vCPU, TBD(guest) until WHP and MSHV CPU times are measured | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is TBD(guest) for every backend, provisionally 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere. The values are set from the next guest build, which counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 100 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | @@ -1862,8 +1872,10 @@ Removed from NVX: and needs ACPI CPPC data the microVM lacks); and `CONFIG_KVM_GUEST` with `CONFIG_PARAVIRT_CLOCK` and `CONFIG_HALTPOLL_CPUIDLE`, which are dormant without a KVM signature. `CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` - stay on and `CONFIG_HYPERV` stays off. The CI debug kernel adds - `CONFIG_DEBUG_KERNEL` with the soft-lockup and hung-task detectors only. + stay on and `CONFIG_HYPERV` stays off. `CONFIG_IRQ_TIME_ACCOUNTING` is + turned on, so `cpu_us` does not charge interrupts to the checks. The CI + debug kernel adds `CONFIG_DEBUG_KERNEL` with the soft-lockup and hung-task + detectors only. - Guest: RTC polling in `nvx-reseed`, and the restore packet v1 to v3 parser in `nvx-port-io`. - Harness: `tsc=reliable` and `no_timer_check` in `BASE_TUNING`; per-backend From a79357dc04d9745b074ee4915ab03fb1ab48494a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 00:33:48 -0700 Subject: [PATCH 088/188] doc: give WHP's measured CPUID read-back cost on every host whp measured the effective CPUID record at 3ed78a903: 57 to 60 native WHvGetVirtualProcessorCpuidOutput reads (the governed leaves plus the unlisted candidates), one call each at 14 to 29 us, which with the one host enumeration cost 0.81 ms on prometheus28, 1.48 ms on the 8370C, and 1.86 ms on the 8573C (medians). The start-up budget said "50 reads, about 0.78 ms" for prometheus28 only. It now gives every host and names the allowed lever, parallel reads before any VP runs, if a gate needs it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index de7753b8a..cea28b678 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1733,8 +1733,11 @@ The backends add their own CPUID work: does not enumerate. - MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on prometheus30 and 51 µs on azure-azlinux-5, against 0.51 and 0.82 ms with - one call per entry. WHP's report takes 50 native reads, about 0.78 ms - (median) on prometheus28. + one call per entry. WHP has no batched read: its report takes 57 to 60 + native reads, one call each, which with its one host enumeration cost + 0.81 ms on prometheus28 and 1.48 to 1.86 ms on Azure (medians). Reading + them on a few threads before any VP runs is the allowed lever if a gate + needs it. **Attribution.** With `OPENVMM_STARTUP_PROFILE` set, OpenVMM's lifecycle profile records three exclusive time ABI restore phases: From 1a855ea2d66c1fcce05362beb711817467e0afc6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 00:43:25 -0700 Subject: [PATCH 089/188] doc: start the deferred restore and boot checks 150 ms later The orchestrator moved Q21's delay from 100 ms to 150 ms after the 0x605 acknowledgement. The perf agent found that the guest-exit window of network_snapshot_restore_wall, 110 to 130 ms after readiness on WHP and 87 to 97 ms on KVM, can overlap checks that start at 100 ms. The asynchronous boot checks used the same constant, so they now start 150 ms after the boot step: about 105 to 135 ms after shell-ready in shell mode. The promotion to normal priority 100 ms after the work starts is unchanged, so the fail-fast bounds become about 150 ms on an idle guest and about 250 ms plus the checks' run time on a CPU-bound one. elapsed_us and cpu_us exclude the 150 ms delay. Guest implements it in v7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index cea28b678..364a8fa07 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1090,13 +1090,13 @@ mounting `/proc`, `/sys`, and `/dev`, it takes a time sample within the (`C12`), so the workload starts on host time; this step is not counted as a discontinuity. Init then reports shell-ready (or starts the workload, in modes without a shell) and runs every other boot check asynchronously, -starting 100 ms after its time ABI boot step (`nvx-time boot`) at +starting 150 ms after its time ABI boot step (`nvx-time boot`) at `SCHED_IDLE`, and at normal priority from 100 ms after they start (see [Snapshot agent](#snapshot-agent)). The boot step is the anchor because every mode reaches it, whereas a mode that execs an agent reports readiness later, on the agent's own protocol. In shell mode the boot step comes 15 to -44 ms before shell-ready, so the checks start about 55 to 85 ms after it. The -checks start the daemon when they pass. Every check stays +44 ms before shell-ready, so the checks start about 105 to 135 ms after it. +The checks start the daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the workload is running. A guest that powers off before its checks finish skips them, which is why CI takes its evidence from `nvx-time status`, which waits for them. @@ -1187,9 +1187,9 @@ is still pending after 30 s, its line reports `status=pending`, without `elapsed_us` or `cpu_us`, and the exit status is 1. At boot, the checks' work is the checks and the daemon start, but not the -100 ms delay after the boot step; at capture, +150 ms delay after the boot step; at capture, the step 1 checks of the [snapshot agent](#snapshot-agent). At restore, it -is steps 6 to 12 and the step 11 checks, but not step 13's 100 ms delay, the +is steps 6 to 12 and the step 11 checks, but not step 13's 150 ms delay, the grace period wait, or the deferred `C7`. `elapsed_us` is that work's wall time, including any wait for a CPU at `SCHED_IDLE`; it bounds the fail-fast latency and is not budgeted. `cpu_us` is the CPU time that the `nvx-time` @@ -1321,7 +1321,7 @@ acknowledgement. failure still emits its violation event and powers off with status 193. 12. If `ACK_REQUIRED` is set, write the acknowledgement (2) to `0x605`. 13. After the acknowledgement, or after step 10 when none is required, signal - the daemon, which finishes the restore asynchronously, starting 100 ms + the daemon, which finishes the restore asynchronously, starting 150 ms later at `SCHED_IDLE`: - it runs the step 11 checks; - it waits for the [grace period release](#rcu-grace-period-release); @@ -1342,24 +1342,28 @@ signals the daemon, so no other process starts on the readiness path. Each extra process costs 2 to 4 ms after a restore (fork and exec under demand faulting, measured on KVM). -Step 13 starts 100 ms after the acknowledgement (or after step 10, when none -is required), and the asynchronous boot checks start 100 ms after the boot +Step 13 starts 150 ms after the acknowledgement (or after step 10, when none +is required), and the asynchronous boot checks start 150 ms after the boot step. Starting them earlier slows the guest: their checks contend with the readiness path on the other vCPUs, even at `SCHED_IDLE`. On WHP (prometheus28, 512 MiB, measurement-only builds), checks right after the -acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas the 100 ms +acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas a 100 ms delay left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. On MSHV (prometheus30), boot checks right after shell-ready added about 3 ms -to 2-vCPU cold boots. KVM showed no clear effect either way. +to 2-vCPU cold boots. KVM showed no clear effect either way. The delay is +150 ms rather than 100 ms because a workload's own activity right after +readiness must not meet the checks either: the guest-exit window of +`network_snapshot_restore_wall` falls 110 to 130 ms after readiness on WHP +and 87 to 97 ms on KVM, which checks starting at 100 ms could overlap. Step 13 and the asynchronous boot checks run at `SCHED_IDLE`, so on few vCPUs they never take a CPU from the restored or starting workload. Work that has not finished 100 ms after it starts continues at normal priority. Every check stays fail-fast: a failing check powers the guest off about -100 ms after the acknowledgement or the boot step on an idle guest, and at -most about 200 ms plus the checks' own run time after it on a CPU-bound +150 ms after the acknowledgement or the boot step on an idle guest, and at +most about 250 ms plus the checks' own run time after it on a CPU-bound one, where `SCHED_IDLE` alone would allow about 0.5 s or more. The `elapsed_us` -and `cpu_us` of both exclude the 100 ms delay. `nvx-time status` waits for +and `cpu_us` of both exclude the 150 ms delay. `nvx-time status` waits for this deferred work, within its 30 s bound. The watcher and the discipline always run at normal priority, and stall suppression stays set until the release. Repair failures emit a violation event and power off with status @@ -1691,7 +1695,7 @@ Expected effects: | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is TBD(guest) for every backend, provisionally 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere. The values are set from the next guest build, which counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 100 ms after the acknowledgement, after the readiness path | +| Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. From 5b92a685b89ef968600d02f3eecb2201a920449e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:11:17 -0700 Subject: [PATCH 090/188] doc: describe the time ABI in the design docs and drop the Proposed marker The time ABI is implemented and integrated in OpenVMM, so the design docs now describe it where they still described the legacy clock path, and the specification becomes the document of record. - time-abi.md: drop the Proposed marker. Note that standard-machine disk snapshots also use manifest version 6 and that manifest fields 9 and 10 are retired. Give the per-backend cost of the CPU checks, and replace the Guide update plan with where the Guide documents the contract. - snapshot-and-restore.md: the manifest's time contract and CPU profile record, version 6 only, the time ABI restore steps, restore packet v4 for processor and memory targets, and a Time and entropy section that summarizes the time ABI and links to it. - machine-and-device-abi.md: the chipset's time ABI clocks, and portb's record reads, status bit 6, selector 0xa7, and the 0xeb window. - cold-boot.md: no clock tokens on the command line (E_CMDLINE_CLOCK_TOKEN). - validation.md: the time ABI's tests in place of the backend TSC repair tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/cold-boot.md | 16 +-- doc/design/machine-and-device-abi.md | 16 ++- doc/design/snapshot-and-restore.md | 198 +++++++++++---------------- doc/design/time-abi.md | 64 ++++----- doc/design/validation.md | 18 ++- 5 files changed, 143 insertions(+), 169 deletions(-) diff --git a/doc/design/cold-boot.md b/doc/design/cold-boot.md index 366896c07..88e916fce 100644 --- a/doc/design/cold-boot.md +++ b/doc/design/cold-boot.md @@ -95,21 +95,17 @@ tokens. A fresh boot then appends host-owned tokens in this order: 2. the optional fixed workload identity (`nvx_workload_uid=` and `nvx_workload_gid=`) and workload lifecycle (`nvx_lifecycle=`); 3. `nvx_snapshot_tier=` for a capture with sandbox blocks; -4. the backend-reported TSC frequency as `tsc_early_khz=` when the boot can - publish a snapshot, and the backend-reported LAPIC timer frequency as - `lapic_timer_hz=` whenever the backend reports one; -5. device-discovery tokens in fixed address order: network, filesystem, boot +4. device-discovery tokens in fixed address order: network, filesystem, boot console, sandbox blocks, and the control console followed by `nvx_control_tty=hvc2`; and -6. network bootstrap tokens, including gateway DNS only when the egress +5. network bootstrap tokens, including gateway DNS only when the egress policy permits it, followed by filesystem bootstrap tokens for an active HostFs export. -The worker inserts the frequency tokens before device discovery and before any -`--` delimiter. When it propagates a frequency, a caller-supplied token for the -same parameter is accepted only when it matches the backend value and is then -rewritten in canonical form; conflicting, duplicate, or malformed values fail -the boot. +The command line carries no clock parameter: the guest reads its TSC and +LAPIC rates from the [time ABI](time-abi.md#rates) MSRs, and a +`tsc_early_khz=` or `lapic_timer_hz=` token, from the caller or a snapshot, +fails with `E_CMDLINE_CLOCK_TOKEN`. Callers may not supply `earlycon=`, `console=`, `virtio_mmio.device=`, `virtnet_ip=`, `virtnet_mask=`, `virtnet_gw=`, `virtnet_dns=`, `virtfs_dir=`, diff --git a/doc/design/machine-and-device-abi.md b/doc/design/machine-and-device-abi.md index 1eebdc363..d267c6a11 100644 --- a/doc/design/machine-and-device-abi.md +++ b/doc/design/machine-and-device-abi.md @@ -22,6 +22,13 @@ level interrupt is neither lost nor treated as an edge while reconstructing the backend. Serial UARTs, debugcon, Hyper-V power management, gameport, PCI, firmware helpers, and standard-PC missing-port shims are absent. +The guest's clocks follow [NVX time ABI v1](time-abi.md) on every backend: a +minimal Hyper-V identity whose MSRs declare the TSC and LAPIC rates, served by +the `time-abi` state unit; the TSC as the only clocksource; the LAPIC timer in +one-shot mode at the backend's fixed rate (1 GHz on KVM, 200 MHz on MSHV and +WHP); and a PIT that the guest disables at boot. Capture rejects a periodic or +TSC-deadline LAPIC timer and a periodically counting PIT channel 0. + ```mermaid %%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% flowchart TB @@ -61,8 +68,9 @@ flowchart TB | Port | Device | Behavior | | ---: | --- | --- | -| `0xe9` | portb data | Raw byte input and output; reads consume one pending byte and zero-fill the remaining access width. | -| `0xea` | portb status | Bit 0 reports pending host input, bit 1 reports a fresh restore packet, bit 2 reports a processor target, bit 3 reports a version-3 memory target, bit 4 reports one or more memory-expansion ranges, and bit 5 reports the fixed generation-ID selector. Writing `0xa5` after restore selects the one-time restore packet. Writing `0xa6` selects the current 16-byte generation ID; it may be selected repeatedly and remains stable for the lifetime of one VM process. | +| `0xe9` | portb data | Raw byte input and output; console reads consume one pending byte and zero-fill the remaining access width. With the restore packet or generation ID selected, a 1-, 2-, or 4-byte read returns that many bytes of the record, zero-filled past its end. | +| `0xea` | portb status | Bit 0 reports pending host input, bit 1 reports a fresh restore packet, bit 2 reports a processor target, bit 3 reports a memory target, bit 4 reports one or more memory-expansion ranges, bit 5 reports the fixed generation-ID selector, and bit 6 reports the time-sample window, which always exists. Writing `0xa5` after restore selects the one-time [restore packet](time-abi.md#restore-packet). Writing `0xa6` selects the current 16-byte generation ID; it may be selected repeatedly and remains stable for the lifetime of one VM process. Writing `0xa7` latches a fresh [time sample](time-abi.md#time-sample) into the window without changing the `0xe9` selection. | +| `0xeb` | portb time window | A 1-, 2-, or 4-byte read returns the next bytes of the latched time sample, zero-filled past its end or when none is latched. The window is not saved state. | | `0x604` | shutdown | The first output byte becomes the process status carried with the VM power-off request. Reads return all ones. | | `0x605` | snapshot request | Reads return all ones. Writes are coalesced and routed asynchronously to the capture controller. Zero requests fresh scratch and a nonzero first byte requests paired scratch. | @@ -72,8 +80,8 @@ backpressure by stopping host reads when its buffer is full. Pending bytes are saved so capture does not silently lose VMM-owned I/O. While host input is gated for a snapshot boundary or post-restore repair, portb stops reading its host endpoint, returns zero for console data reads, and clears the -input-available status bit; guest output, the generation ID, and any restore -packet remain available. +input-available status bit; guest output, the generation ID, any restore +packet, and the time-sample window remain available. Guest-requested process exit drains the portb endpoint and, when present, its host stdout relay before reporting completion. The combined drain has a diff --git a/doc/design/snapshot-and-restore.md b/doc/design/snapshot-and-restore.md index 98a1fa61b..82bce57b4 100644 --- a/doc/design/snapshot-and-restore.md +++ b/doc/design/snapshot-and-restore.md @@ -4,7 +4,7 @@ ## Tier contract -Version-5 sandbox snapshots encode the tier, restore policy, and a consumed +Sandbox snapshots encode the tier, restore policy, and a consumed configuration-section bitmask. The validator accepts only these combinations: | Tier | Restore policy | Scratch | Consumed sections | Intended author and sharing scope | @@ -57,6 +57,7 @@ thaws every guest-owned barrier; failure to thaw terminates the VM. 4. quiesce the remaining device workers and VM time in dependency order, including bounded virtio-blk queue drain; 5. save the exact state-unit inventory, processor state, device-private state, + the [time contract and CPU profile record](time-abi.md#snapshot-manifest), and memory; 6. copy and flush state, memory, and any paired scratch into a unique sibling staging directory; @@ -93,7 +94,7 @@ sequenceDiagram else Capture accepted Controller->>Worker: Quiesce for snapshot Worker->>Units: Stop and save in dependency order - Units-->>Controller: State, inventory, CPU, and clock contract + Units-->>Controller: State, inventory, time contract, and CPU profile Controller->>Storage: Copy state, memory, and optional scratch Controller->>Storage: Publish with no-replace rename Controller->>Worker: Terminate committed source @@ -245,9 +246,9 @@ The manifest is authoritative for: - effective command line and digest; - exact device order, stable IDs, state-unit names, MMIO/PMIO ranges, IRQs, transport, feature masks, and queue limits; -- the canonical CPU compatibility contract (vendor, CPUID results, XSAVE - layout, and processor features) and its digest, TSC and LAPIC timer - frequencies, capture wall clock, and downtime clock policy; +- the time contract (the declared TSC and LAPIC rates, VP 0's TSC paired with + a host time sample, the host identities, and the generation counter) and the + CPU profile record (the pinned profile and the effective CPUID); - required host attachments and their policies; - block roles, access, geometry, layer identities, and scratch policy; - snapshot tier, clone/resume policy, and consumed configuration sections; @@ -256,13 +257,12 @@ The manifest is authoritative for: Snapshot paths and repeated fields are bounded. Restore rejects truncated, oversized, malformed, wrong-type, path-escaping, symlinked, incompatible, -missing, extra, or reordered state before guest execution. New snapshots use -version 5. It does not store or validate embedded checksums for `state.bin` or -`memory.bin`; a same-length change to either payload is therefore outside the -validation contract. Paired scratch is checked because it must match captured -filesystem state. Versions 2 through 4 remain readable; versions 2 and 3 cannot -describe sandbox blocks, and version 4 predates tier metadata. Legacy version-2 -checksum fields are accepted without re-hashing either payload. Snapshot +missing, extra, or reordered state before guest execution. Every snapshot uses +manifest version 6; any other version or format magic is rejected with +`E_SNAPSHOT_VERSION`, which asks for a recapture. The manifest does not store +or validate embedded checksums for `state.bin` or `memory.bin`; a same-length +change to either payload is therefore outside the validation contract. Paired +scratch is checked because it must match captured filesystem state. Snapshot directories rely on host access control, while authenticated export or transport belongs outside the default local artifact format. @@ -297,10 +297,14 @@ Restore proceeds in the opposite direction from capture: file; 5. construct the partition and exact device inventory with the selected active RAM and the snapshot's immutable capacity; -6. compare the destination CPU, TSC, topology, device, and queue contract with - the saved contract; +6. run the [time ABI preflight](time-abi.md#restore-algorithm) (backend, CPU + profile and generation, rates, and downtime), compare the destination + topology, device, and queue contract with the saved contract, and check the + partition's effective CPUID against the snapshot's record; 7. restore VM time, chipset and virtio state, partition state, and vCPU state, - then advance the guest clocks by the host downtime; + then set every VP's TSC at one host instant with read-back, advance the + LAPIC timers, VM time, RTC, and PIT by the downtime, and seal the restore + packet's time fields; 8. finish reconnecting host resources; 9. start the state units, with external input gated first when tier policy, processor activation, or nonempty RAM expansion requires guest repair, and @@ -337,9 +341,10 @@ actions. The restored VM stays paused until the client resumes it; that resume publishes the readiness event, and a publication failure fails the resume and tears the VM down. -The partition must exist before OpenVMM can derive its effective destination -CPU contract. This does not expose a partially restored guest: contract -comparison and all saved-state validation still complete before a vCPU runs. +The partition must exist before OpenVMM can read the CPUID it presents and +check it against the effective CPUID. This does not expose a partially +restored guest: these checks and all saved-state validation still complete +before a vCPU runs. ```mermaid %%{init: {"theme": "base", "themeVariables": {"background": "#ffffff"}}}%% @@ -371,13 +376,13 @@ flushes `resume.claim`; duplicate and concurrent restores fail before worker construction, and a later startup failure does not make the checkpoint reusable. Initial compatibility is same-backend: KVM snapshots restore on compatible KVM hosts, MSHV on compatible MSHV hosts, and WHP on compatible WHP -hosts. Cross-backend conversion and standalone NVX snapshot import are not -supported. +hosts, of the CPU generation the snapshot's profile names. Cross-backend +conversion and standalone NVX snapshot import are not supported. ## Template compatibility and placement A template represents a compatibility class, not an arbitrary fleet image. -The class includes the backend, CPU and TSC contract, guest kernel and +The class includes the backend, CPU profile and generation, guest kernel and agent build, effective command line, device roles and geometry, network identity/policy, and consumed layer identities. Deployment must rebuild or recertify templates on relevant rollouts; the VMM's actual compatibility and @@ -429,9 +434,9 @@ contract remains backend-independent: MSHV creates application processors lazily while binding their VP runners, so discarding suffix binders before that boundary avoids creating processors that -this restore will not run. Restored-TSC alignment on MSHV therefore covers only -the created VPs; the hypervisor rejects register access to a VP that was never -created. KVM and WHP deliberately retain their existing full-capacity +this restore will not run. The synchronized TSC set on MSHV therefore covers +only the created VPs; the hypervisor rejects register access to a VP that was +never created. KVM and WHP deliberately retain their existing full-capacity construction path, including when an explicit online target is sent to the guest. @@ -446,14 +451,13 @@ also fails explicitly instead of indexing nonexistent runtime state. The original immutable snapshot remains reusable for independent restores at other valid targets. -After host-side state restoration, a processor-only activation uses the -version-2 private restore packet to carry `N` to the Alpine agent. Portb status -bit 2 distinguishes it from a legacy version-1 entropy packet. If the same -restore also selects a RAM target, version 3 carries both targets. While -external input remains gated, the agent onlines CPUs from `B` through `N-1`, -verifies that the kernel's online-CPU set is exactly the requested prefix, -and acknowledges through PMIO `0x605`. OpenVMM stops at that post-write -boundary, releases host input, and then resumes the guest. +After host-side state restoration, the +[restore packet](time-abi.md#restore-packet) carries `N` to the Alpine agent as +its online VP count, and portb status bit 2 advertises it; a RAM target travels +in the same packet. While external input remains gated, the agent onlines CPUs +from `B` through `N-1`, verifies that the kernel's online-CPU set is exactly the +requested prefix, and acknowledges through PMIO `0x605`. OpenVMM stops at that +post-write boundary, releases host input, and then resumes the guest. A fixed-capacity `N`-vCPU snapshot and a capacity-`C`, boot-online-`B` template restored to `N` therefore reach the same online prefix but do not contain the @@ -478,8 +482,7 @@ by a snapshot from the active amount selected for one restored process: without the capability rejects an explicit target. The management RPC exposes the same capture capacity and per-launch target, -together with the processor target, gate timeout, fresh-entropy request, and -readiness endpoint. +together with the processor target, gate timeout, and readiness endpoint. The selected expansion is always a prefix of the contract's canonical ranges in logical RAM order. OpenVMM maps the captured ranges privately from @@ -489,101 +492,62 @@ runs. Expansion bytes are neither read from nor written back to the snapshot, so independent restores may choose different valid targets without changing the reusable artifact. -An explicit RAM target selects restore-packet version 3: - -```text -OPENVMM_ENTROPY_V3\0 -u8 optional_online_vp_count -u8 expansion_range_count -repeated { little-endian u64 gpa_start, little-endian u64 length } -64 bytes fresh entropy -``` - -Portb status bit 3 identifies the memory-target packet and bit 4 indicates that -the selected target contains at least one expansion range. Version 3 may carry -a processor target in the same transaction. For a nonempty expansion, -OpenVMM keeps external input gated while the Alpine agent verifies the -128-MiB block size and each range, probes missing blocks through the Linux +An explicit RAM target sets the restore packet's `MEMORY_TARGET` flag, and the +packet lists the selected expansion ranges as `(u64 gpa_start, u64 length)` +pairs ([layout](time-abi.md#restore-packet)). Portb status bit 3 mirrors the +flag and bit 4 indicates that the selected target contains at least one +expansion range. The same packet may carry a processor target. For a nonempty +expansion, OpenVMM keeps external input gated while the Alpine agent verifies +the 128-MiB block size and each range, probes missing blocks through the Linux memory-block probe interface, onlines them and verifies their online state, and then completes entropy and generation-ID repair before acknowledging PMIO `0x605`. Any malformed range or add/online failure terminates the restore. -An explicit base-size target has a zero range count. For an untiered blockless -restore, the agent emits the deterministic zero-add marker without consuming -the packet or entering a restore-gate transaction; a tiered restore still uses -its existing gate. This mechanism is one-shot restore repair, not a general +An explicit base-size target has a zero range count. Unless the tier or a +processor target gates the restore, the packet's `ACK_REQUIRED` flag is clear, +and the agent emits the deterministic zero-add marker without a restore-gate +transaction. This mechanism is one-shot restore repair, not a general post-readiness memory-hotplug interface. ## Time and entropy -Cold microVM boots receive a canonical `lapic_timer_hz` kernel parameter from -the backend's reported LAPIC clock frequency. The NVX kernel uses this known rate -instead of comparing LAPIC interrupts with scheduling-sensitive emulated PIT -interrupts during boot. Without it, delayed PIT delivery can cause Linux to disable -a working LAPIC timer. Native calibration remains available when no frequency is -reported, and the TSC-deadline path is unchanged. A platform snapshot's saved -command-line parameter, when present, must agree with its APIC frequency contract. - -The machine contract records the guest TSC frequency with zero tolerance, but -a cold-booted Linux guest would otherwise calibrate its own, slightly -different rate against emulated timers. A boot that can publish a snapshot -therefore also receives the backend-reported TSC frequency as -`tsc_early_khz`; ordinary boots keep the guest's normal TSC discovery. A -platform snapshot's command line must carry exactly the recorded value. Under -the versioned CPU contract, WHP runs the guest TSC at a fixed 1 GHz when the -host supports that rate and otherwise keeps the host frequency; MSHV exposes no -TSC-deadline mode because it does not reliably deliver those events to -direct-boot guests. - -Capture records a coherent processor and clock boundary. Restore advances TSC, -VM time, RTC, PIT/LAPIC deadlines, and the KVM paravirtual clock by nonnegative -host downtime, then reanchors them before vCPUs start. A destination that -cannot reproduce the saved CPU or clock contract is rejected rather than -silently changing guest behavior. The current restore path rejects negative -downtime and elapsed host downtime greater than 30 days. One-shot and periodic -LAPIC timers advance by the downtime at the LAPIC timer frequency. A timer that -the downtime expires, including a TSC deadline that the advanced counter has -passed or that was already overdue, is delivered by queueing its interrupt in -the restored LAPIC state; an expired TSC deadline is then disarmed. Exact -deadline read-back is excluded from state comparison because a deadline can -expire between restore and read-back. Every backend reads back each advanced -counter before resume and rejects a discarded or incomplete adjustment. -Versioned MSHV CPU contracts do not expose `IA32_TSC_ADJUST` because snapshot -state cannot preserve that register independently of `IA32_TSC`; Linux -therefore does not interpret OpenVMM's host-side TSC correction as per-vCPU -firmware adjustment skew. - -KVM advances each VP's TSC through its TSC offset rather than a counter write, -because KVM can discard a sub-second counter write as a synchronization -attempt; a host without TSC-offset control fails restore explicitly. KVM also -saves and restores the paravirtual-clock MSRs exactly. For restored SMP on -MSHV and WHP, partition time is frozen while the instantiated VP counters are -aligned before execution, avoiding skew introduced by sequential host register -writes. WHP additionally uses a partition-reference-time-based TSC model for -restored SMP timestamp reads, including `RDTSC`/`RDTSCP` and TSC MSR reads. An -intentional guest TSC adjustment returns that VP to its guest-programmed -hardware counter. This backend repair is not general cross-host TSC-frequency -conversion. - +Every microVM uses [NVX time ABI v1](time-abi.md), which defines the guest's +clocks on every backend: + +- The guest sees a minimal Hyper-V identity with the declared TSC and LAPIC + rates in MSRs, a CPU profile pinned for its CPU generation, an invariant TSC + as its clocksource, and the LAPIC timer at the backend's fixed rate. The + command line carries no clock tokens + ([guest-visible contract](time-abi.md#guest-visible-contract)). +- Capture records the time contract and the CPU profile record in the + [manifest](time-abi.md#snapshot-manifest). +- Restore requires the same backend and CPU generation, a native TSC rate + within 250 ppm of the declared rate, and an exactly equal LAPIC rate. It + measures the downtime with same-boot host monotonic time or the host UTC + delta, sets every VP's TSC at one host instant with read-back, and advances + the LAPIC timers, VM time, RTC, and PIT before the first VP runs + ([restore algorithm](time-abi.md#restore-algorithm)). Any failure rejects + the restore with a stable code; there is no fallback. +- After restore, the guest's snapshot agent sets the wall clock from the + restore packet's host UTC sample and re-checks conformance + ([snapshot agent](time-abi.md#snapshot-agent)). Replaying a snapshot also replays the guest's in-memory random-number-generator -state. Tiered restore, processor activation, and an explicit RAM target each -cause OpenVMM to create a fresh one-time packet and expose it through the portb -status/data protocol; callers may also request that packet directly. Every -microVM process receives a fresh 16-byte generation ID before vCPU entry. The -ID is not serialized, and the VMM-owned value is not restored from device -state. When a restore packet exists, the ID is its first 16 entropy bytes; -otherwise it is generated independently. This keeps the gated repair path from -adding port reads. The Alpine agent keeps the prior ID in its captured process -state, rejects a restored ID that did not change, and exports the refreshed -value to the runtime hook before releasing the gate. +state. Every restore therefore exposes a fresh one-time +[restore packet](time-abi.md#restore-packet) through the portb status/data +protocol; with the time fields and any targets, it carries 64 bytes of fresh +entropy. Every microVM process receives a fresh 16-byte generation ID before +vCPU entry. The ID is not serialized, and the VMM-owned value is not restored +from device state. On restore, the ID is the packet's first 16 entropy bytes; +on cold boot it is generated independently. This keeps the gated repair path +from adding port reads. The Alpine agent keeps the prior ID in its captured +process state, rejects a restored ID that did not change, and exports the +refreshed value to the runtime hook before releasing the gate. For clone policy, the Alpine restore path credits the seed, including the generation ID, to the kernel entropy pool, forces a kernel CRNG reseed, -refreshes wall clock and machine identity, and requires the workload-start -runtime hook to reset runtime-owned RNG state before accepting work. RTC -update-in-progress is polled with a bounded read loop rather than a timer -sleep because guest timers are not authoritative until this wall-clock repair -completes. The workload's `/etc/machine-id` is a read-only bind of a +refreshes machine identity, and requires the workload-start runtime hook to +reset runtime-owned RNG state before accepting work. The workload's +`/etc/machine-id` is a read-only bind of a runtime-tmpfs file, so the agent can refresh it while scratch remains frozen; the agent also updates the workload's UTS namespace before acknowledgement. It then acknowledges the VMM gate before thawing scratch and the workload cgroup. diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 364a8fa07..2a1749eea 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -2,12 +2,11 @@ [Design index](../design.md) -**Proposed.** This document specifies NVX time ABI v1, the single -guest-visible time, timer, and clock contract of the microVM profile on KVM, -MSHV, and WHP. It is authoritative for the `time-abi-v1` integration branches -of `microsoft/nvx` and `nanvix/openvmm`. When implemented, it replaces the -time rules in [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) -and the clock tokens in [Cold boot](cold-boot.md#effective-command-line). +This document specifies NVX time ABI v1, the single guest-visible time, +timer, and clock contract of the microVM profile on KVM, MSHV, and WHP, as NVX +and its OpenVMM fork (`nanvix/openvmm`) implement it. It replaces the earlier +time rules of [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) +and the clock tokens of [Cold boot](cold-boot.md#effective-command-line). Notation: @@ -848,7 +847,10 @@ New snapshots use manifest version 6 with format magic `OPENVMM_SNAPSHOT_V6\0`. Restore accepts exactly version 6; versions 2 through 5 and every other value are rejected with `E_SNAPSHOT_VERSION`, whose message tells the operator to recapture the snapshot. The legacy version -branches and their format constants are deleted. +branches and their format constants are deleted. Version 6 is the format's +only version, so standard-machine disk snapshots use it too; they carry no +machine contract and no time records. The manifest's fields 9 and 10, the +version 2 artifact digests, are retired and never reused. **Capture.** After every VP has stopped at the snapshot boundary and the state units are quiesced, capture: @@ -1759,6 +1761,10 @@ after the acknowledgement, outside these phases. `nvx-time status` reports checks. On every cold boot and restore, the worker logs `time ABI CPU checks passed` with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. +Their sum per restore at one vCPU is about 0.19 ms on bare-metal MSHV and +0.14 ms nested, and 0.87 ms on bare-metal WHP and 1.6 to 1.9 ms nested, about +95% of it WHP's per-leaf read-back. OpenVMM's comparison and unlisted-entry +check take 40 to 66 µs of each. ## Test matrix @@ -1867,7 +1873,9 @@ behavior: - MSHV: BSP-copy TSC alignment and exact rate equality. - WHP: the 1 GHz rate request and its silent fallback, and `RestoredTsc` with its RDTSC, RDTSCP, and `IA32_TSC` exits. -- Restore packets v1 to v3 and manifest versions 2 to 5. +- Restore packets v1 to v3, and manifest versions 2 to 5 for every snapshot: + standard-machine disk snapshots also move to version 6, so earlier ones + must be recaptured too. - The effect of `--restore-entropy`: every restore exposes packet v4, which carries fresh entropy. OpenVMM still accepts the option without effect, and the harness stops passing it. @@ -1921,26 +1929,20 @@ Migration impact: ABI](machine-and-device-abi.md), [Cold boot](cold-boot.md), the benchmarks and CI guides, and the OpenVMM Guide (see the appendix). -## Appendix: OpenVMM Guide update plan - -`Guide/src/user_guide/openvmm/snapshots.md` changes as follows when the -implementation lands: - -1. **Overview.** Note that microVM snapshots carry a time contract and a CPU - profile, and that only manifest version 6 is accepted. -2. **Restoring a snapshot.** Replace the paragraphs on MSHV - `IA32_TSC_ADJUST`, KVM TSC offset advancement, MSHV and WHP time freezing - and BSP alignment, and the WHP partition-reference-time TSC with one - paragraph on the synchronized TSC set and its read-back. -3. **Generation ID.** Extend the portb paragraph with selector `0xa7`, window - port `0xeb`, status bit 6, four-byte reads, and packet v4. -4. **Device configuration on restore.** Replace the CPU contract, TSC - frequency, and `lapic_timer_hz` paragraphs with a new **Time and CPU - compatibility** section: the identity, the CPU profile and - `--cpu-profile`, the 250 ppm rate rule, the exact LAPIC rule, the - downtime sources and bounds, the restore steps, and the error codes. -5. **Limitations.** State that restore requires the same backend, CPU - generation, and profile, and that older snapshots must be recaptured. -6. **CLI reference.** Document `--cpu-profile` and the doctor verification - mode. The test hooks stay undocumented in the Guide; this document - describes them. +## Appendix: OpenVMM Guide + +The OpenVMM Guide documents the user-facing contract in +`Guide/src/user_guide/openvmm/snapshots.md`: + +- **Overview:** the time contract and CPU profile in microVM manifests, and + manifest version 6 as the only accepted version. +- **Restoring a snapshot:** restore packet v4, the time sample at `0xeb`, the + generation ID, and the synchronized TSC set with its read-back. +- **Time and CPU compatibility:** the identity, the CPU profile and + `--cpu-profile`, the 250 ppm rate rule, the exact LAPIC rule, the downtime + sources and bounds, and the error codes. +- **Limitations:** the same backend, CPU generation, and profile, and the + recapture of older snapshots. + +The CLI reference documents `--cpu-profile` and `--x-time-abi-verify`. The +test hooks stay undocumented in the Guide; this document describes them. diff --git a/doc/design/validation.md b/doc/design/validation.md index 47ab0970f..f111a3ecf 100644 --- a/doc/design/validation.md +++ b/doc/design/validation.md @@ -23,7 +23,8 @@ are unchanged. The NVX-owned suite boots the same Linux-direct artifacts on the available native backend and covers IRQ0/RTC behavior, raw portb I/O, shutdown status, exact -snapshot sequencing, repeated immutable restore, coherent downtime, fresh +snapshot sequencing, repeated immutable restore, time ABI conformance and +restore downtime, fresh generation IDs, `getrandom()` output, kernel UUIDs, temporary-file identifiers, entropy reseed, active console RX/TX, network policy and HTTP traffic, and live virtio-fs attachment revalidation, including a guest-created symbolic link held @@ -65,7 +66,7 @@ and expanded allocation, and verifies artifact immutability. Unit coverage verifies that only an explicit MSHV processor target selects a runtime prefix, that the complete saved VP inventory is validated before filtering, that reduced-prefix saves are rejected, that dormant VP access fails cleanly, and -that MSHV restored-TSC alignment targets only created VPs. +that the MSHV synchronized TSC set targets only created VPs. Additional unit coverage exercises the control-console slot and attachment inventory, command-line spoofing rejection, the control-session record protocol against language-neutral golden vectors and boundary cases, the @@ -75,11 +76,14 @@ the bounded outcome-report schema, egress-policy enforcement in the endpoint and virtio-net layers, the virtio-fs microVM profile and denied-path policy, state-unit quiesce and rollback, management exclusion at the snapshot boundary, management-RPC guest-exit propagation, output-drain completion and -failures, and backend TSC repair. Hardware-dependent clock tests still require -their native backend. A host-only benchmark measures snapshot publication, -restore preparation, copy-on-write dirtying, and fresh-process restore -preparation without booting a guest. Platform CI and the benchmark histories -in `data/` provide the wider host matrix. +failures, and the time ABI's rate, downtime, LAPIC, restore-packet, and +manifest rules. Hardware-dependent time ABI tests, such as the synchronized TSC +set, identity routing, and the CPU profile read-back, still require their +native backend; the [time ABI test matrix](time-abi.md#test-matrix) lists the +conformance, restore, soak, and performance coverage. A host-only benchmark +measures snapshot publication, restore preparation, copy-on-write dirtying, and +fresh-process restore preparation without booting a guest. Platform CI and the +benchmark histories in `data/` provide the wider host matrix. The separate `test-adversarial` harness adaptively selects tracked deterministic primitives from these same process tests. Copilot has no tools From ef3c74934653e75a752c9412b24c3857e7b6c4b0 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:21:57 -0700 Subject: [PATCH 091/188] doc: list the ITS and multi-vCPU cold-boot costs of the v1 profiles kvm's attribution at the current head shows that restore is back at pre-integration and that the whole one-vCPU cold-boot delta on prometheus32 is Linux's ITS mitigation, which the v1 profiles require by clearing ITS_NO. Record it in the expected effects: about 6 ms where the host's KVM advertises ITS_NO, and no change on the gated Azure KVM, MSHV, and WHP platforms, whose guests were already mitigated. Also record the separate, ungated multi-vCPU cold-boot cost of the profile's CPU view on KVM, which is not yet explained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 2a1749eea..6d3de2f2f 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1697,6 +1697,8 @@ Expected effects: | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is TBD(guest) for every backend, provisionally 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere. The values are set from the next guest build, which counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | +| v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | +| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated and not yet explained: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32, where `msr_init` waits about one 10 ms tick for CPU 1, and about +25 ms at 2 vCPUs on azure-kvm-5 | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | From 0a9652ce2844b004448a49d0e351e9a8c32fc33e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:29:41 -0700 Subject: [PATCH 092/188] doc: give each backend a cpu_us budget per phase whp's wiring v6 final matrix measured restore cpu_us of 22 to 30 ms on bare metal and 29 to 45 ms nested on WHP, against boot medians of 2.2 to 9.6 ms. The restore checks also pay the first-touch faults of WHP's lazily registered copy-on-write RAM, which the workload would otherwise pay, and pre-populating the RAM would move that work onto the gated restore path. One budget for boot, capture, and restore therefore cannot fit WHP: each backend gets a budget per phase, set to the next guest build's maxima plus headroom. The values stay TBD(guest). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6d3de2f2f..230634ca6 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1203,7 +1203,8 @@ path, whose cost the restore latency gate already measures, so `cpu_us` is the background cost to the workload. The NVX kernel does not charge interrupt time to it (`CONFIG_IRQ_TIME_ACCOUNTING`), but it still scales with the host processor's clock: a host idling at a low frequency runs the checks -more slowly. The fleet matrix driver checks `cpu_us` against the budget in +more slowly. The fleet matrix driver checks `cpu_us` against the backend's +budget for the phase in [Performance expectations](#performance-expectations-and-acceptance-gate) during validation; CI reports it without gating on it. @@ -1695,7 +1696,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). The same `cpu_us` budget holds for the boot, capture, and restore checks. The fleet matrix driver enforces it during validation; CI reports each phase against it (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budget is TBD(guest) for every backend, provisionally 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere. The values are set from the next guest build, which counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are TBD(guest): the next guest build's maxima plus headroom. Provisionally they are 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere; WHP restores measured 22 to 30 ms on bare metal and 29 to 45 ms nested. The next guest build counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | | The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated and not yet explained: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32, where `msr_init` waits about one 10 ms tick for CPU 1, and about +25 ms at 2 vCPUs on azure-kvm-5 | @@ -1788,8 +1789,8 @@ and after every restore. They require exit status 0 and, after a restore, a `phase=restore` line with `status=ok`, the restored `generation`, and `cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with `generation=0`. During fleet validation (`p6`), the matrix driver also -requires each phase's `cpu_us` to be within the budget in -[Performance expectations](#performance-expectations-and-acceptance-gate); +requires each phase's `cpu_us` to be within the backend's budget for that +phase in [Performance expectations](#performance-expectations-and-acceptance-gate); CI reports it without gating on it. The fleet runs them on the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 From b4a18ba74d157b02daa1e3d9fcb777b5b5383d9d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:31:29 -0700 Subject: [PATCH 093/188] doc: record the post-resume timer catch-up of the restore Under the time ABI, guest monotonic time includes the downtime, so right after resume the guest kernel runs the timer work that came due during it, first-touching cold pages as it does. guest measured it at one vCPU: about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). Raising the restore helper's priority does not change the gated latency, which includes the catch-up either way. Record it among the expected effects as a design consequence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 230634ca6..595085767 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1701,6 +1701,7 @@ Expected effects: | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | | The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated and not yet explained: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32, where `msr_init` waits about one 10 ms tick for CPU 1, and about +25 ms at 2 vCPUs on azure-kvm-5 | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | +| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. From 3f9cf18f7480aefbc933dbee006feb53fd591e96 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:41:57 -0700 Subject: [PATCH 094/188] doc: keep the boot step as the boot-check anchor (Q23) guest measured Ubuntu's shell-ready at 26 to 84 ms after the time ABI boot step at 1 to 8 vCPUs, later than Alpine's 15 to 44 ms. With the 150 ms delay the boot checks still start about 65 ms or more after shell-ready, at SCHED_IDLE, so they cost the boot path nothing. Keep the boot step as the anchor that every mode reaches, rather than signaling the checker at each of init's hand-offs, and give Ubuntu's numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 595085767..43ae0e66b 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1097,7 +1097,9 @@ starting 150 ms after its time ABI boot step (`nvx-time boot`) at [Snapshot agent](#snapshot-agent)). The boot step is the anchor because every mode reaches it, whereas a mode that execs an agent reports readiness later, on the agent's own protocol. In shell mode the boot step comes 15 to -44 ms before shell-ready, so the checks start about 105 to 135 ms after it. +44 ms before shell-ready on Alpine and 26 to 84 ms before it on Ubuntu at 1 to +8 vCPUs, so the checks start about 65 to 135 ms after it. Because they start +at `SCHED_IDLE`, a start closer to shell-ready costs the boot path nothing. The checks start the daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the workload is running. A guest that powers off before its checks finish skips them, which From bbc4040ca13802f26cf8ec73449866530c4abc78 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:27:57 -0700 Subject: [PATCH 095/188] kernel: drop the LAPIC frequency patch Patch 0004 added the lapic_timer_hz= early parameter so Linux would take the LAPIC timer rate from OpenVMM instead of calibrating the LAPIC against emulated PIT interrupts, whose delayed delivery could make Linux disable a working timer. Under time ABI v1 the guest reads the same rate from HV_X64_MSR_APIC_FREQUENCY (0x40000023) through the Hyper-V platform code, which presets lapic_timer_period, so the kernel needs no patch and no clock command-line token. Remove the patch, its SOURCE-MANIFEST.json entry, and the documentation that described the kernel's use of the token. This must land together with the OpenVMM change that stops injecting lapic_timer_hz=: with the old OpenVMM the token would reach an unpatched kernel that ignores it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- SOURCE-MANIFEST.json | 3 +- doc/benchmarks.md | 10 ++-- .../0004-microvm-lapic-frequency.patch | 57 ------------------- 3 files changed, 6 insertions(+), 64 deletions(-) delete mode 100644 kernel/patches/0004-microvm-lapic-frequency.patch diff --git a/SOURCE-MANIFEST.json b/SOURCE-MANIFEST.json index 366661170..a15f52395 100644 --- a/SOURCE-MANIFEST.json +++ b/SOURCE-MANIFEST.json @@ -20,8 +20,7 @@ "patches": [ "kernel/patches/0001-microvm-xe9-earlycon.patch", "kernel/patches/0002-microvm-hvc-xe9.patch", - "kernel/patches/0003-microvm-virtio-mmio-shared-status.patch", - "kernel/patches/0004-microvm-lapic-frequency.patch" + "kernel/patches/0003-microvm-virtio-mmio-shared-status.patch" ] }, "alpine": { diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 3172ff45a..e5fcab91b 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -259,11 +259,11 @@ Each benchmark boots the guest with a fixed kernel command line. Two bases recur OpenVMM owns `BASE`; each `--cmdline` value below is appended to it. Restore phases do not pass a command line and resume the one captured in the snapshot. OpenVMM also appends device-discovery tokens (`virtio_mmio.device=...` for an attached mount or NIC) and may replace `console=hvc0` with -`console=hvc1` when a virtio console is selected. Snapshot-source boots additionally receive the -backend-derived `tsc_early_khz=...` token. All cold microVM boots also receive -`lapic_timer_hz=...` when the backend reports its LAPIC frequency. The NVX kernel uses -that rate without calibrating or verifying the counting LAPIC against emulated PIT -interrupts, whose delivery can be delayed or coalesced by host scheduling. +`console=hvc1` when a virtio console is selected. It appends no clock token: under the +[time ABI](design/time-abi.md#rates) the NVX kernel reads the TSC and LAPIC timer rates from the +hypervisor identity's frequency MSRs and never calibrates either clock, and OpenVMM rejects a +supplied `tsc_early_khz=` or `lapic_timer_hz=` token (`E_CMDLINE_CLOCK_TOKEN`). No benchmark +command line tunes the clock or clocksource, except the `cold_start_clocksource` variant below. | Benchmark | Phase | Kernel command line | | --- | --- | --- | diff --git a/kernel/patches/0004-microvm-lapic-frequency.patch b/kernel/patches/0004-microvm-lapic-frequency.patch deleted file mode 100644 index df994992c..000000000 --- a/kernel/patches/0004-microvm-lapic-frequency.patch +++ /dev/null @@ -1,57 +0,0 @@ -From: NVX maintainers -Subject: [PATCH] x86/apic: accept the microVM LAPIC clock frequency - -OpenVMM knows the LAPIC clock frequency independently of guest scheduling. -Accept that frequency instead of calibrating and verifying the LAPIC against -emulated PIT interrupts, which can be delayed or coalesced on a busy host. -Otherwise Linux can disable a working LAPIC timer and leave LOC stuck at the -12 interrupts used by its verification loop. - -Keep native calibration when the parameter is absent and preserve the TSC -deadline path. Apply the supplied frequency after native TSC discovery so -CPU-model-specific crystal-clock estimates cannot overwrite it. - ---- a/arch/x86/kernel/apic/apic.c -+++ b/arch/x86/kernel/apic/apic.c -@@ -33,6 +33,7 @@ - #include - #include -+#include - #include - #include - #include - #include -@@ -175,6 +176,20 @@ - - unsigned int lapic_timer_period = 0; - -+static unsigned int lapic_timer_hz __initdata; -+ -+static int __init parse_lapic_timer_hz(char *arg) -+{ -+ unsigned int frequency; -+ -+ if (!arg || kstrtouint(arg, 0, &frequency) || frequency < 1000000) -+ return -EINVAL; -+ -+ lapic_timer_hz = frequency; -+ return 0; -+} -+early_param("lapic_timer_hz", parse_lapic_timer_hz); -+ - static void apic_pm_activate(void); - - /* -@@ -800,6 +815,12 @@ static int __init calibrate_APIC_clock(void) - if (boot_cpu_has(X86_FEATURE_TSC_DEADLINE_TIMER)) - return 0; - -+ if (lapic_timer_hz) { -+ lapic_timer_period = lapic_timer_hz / HZ; -+ pr_info("APIC timer: using supplied frequency %u Hz\n", -+ lapic_timer_hz); -+ } -+ - /* - * Check if lapic timer has already been calibrated by platform - * specific routine, such as tsc calibration code. If so just fill From a0a65df56b27ea5d218034ba3fe6a10c7690dc1c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 01:30:50 -0700 Subject: [PATCH 096/188] kernel: drop the dormant KVM guest options Under time ABI v1 every backend presents a Hyper-V identity and no KVM signature, so kvm_para_available() is false and CONFIG_KVM_GUEST, kvmclock (CONFIG_PARAVIRT_CLOCK), and the haltpoll cpuidle driver and governor never run. Keeping them leaves a kvmclock path that the ABI forbids. Turn off KVM_GUEST, which also drops PARAVIRT_CLOCK, HALTPOLL_CPUIDLE, CPU_IDLE_GOV_HALTPOLL, and X86_HV_CALLBACK_VECTOR; the Hyper-V platform code installs that vector only under CONFIG_HYPERV. ARCH_CPUIDLE_HALTPOLL has a prompt, so it is turned off explicitly. HYPERVISOR_GUEST and PARAVIRT stay on for the Hyper-V platform code. The build now requires KVM_GUEST to be off. This must land together with the OpenVMM change that gives KVM guests the Hyper-V identity and the harness change that stops forcing clocksource=kvm-clock: on today's KVM backend the guest would lose kvmclock, which the snapshot-core test still requires. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 18 ++++++++++-------- kernel/config-microvm | 8 ++------ scripts/nvx_tools/build_constants.py | 1 + scripts/test_nvx_tools.py | 4 ++++ 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/doc/build.md b/doc/build.md index 7b90db740..4f61a1c8b 100644 --- a/doc/build.md +++ b/doc/build.md @@ -179,14 +179,16 @@ The native kernel build caches the verified and patched source under generated config as `build/vmlinux.config`, and fails if ACPI is enabled, PVH remains enabled, or the MP-table, APIC, IOAPIC, and command-line virtio-mmio requirements are missing. It also enforces the time-ABI settings: -`CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` stay on, while `CONFIG_HYPERV` -and `CONFIG_CPU_FREQ` stay off. Without cpufreq, `intel_pstate` cannot probe -MSRs the microVM does not implement, so no `#GP` traces are printed with -interrupts disabled during boot. `CONFIG_SCHED_MC_PRIO` is off as well because -it selects both cpufreq and `intel_pstate`. Production kernels must not enable -the soft-lockup or hung-task detectors. Changing an archive hash or patch -invalidates both source and object caches; changing the input configuration -invalidates the object cache. +`CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` stay on, while `CONFIG_HYPERV`, +`CONFIG_KVM_GUEST`, and `CONFIG_CPU_FREQ` stay off. The guest identifies the +hypervisor only through the time ABI's Hyper-V identity, so the KVM guest code, +kvmclock, and the haltpoll idle driver would be dead code. Without cpufreq, +`intel_pstate` cannot probe MSRs the microVM does not implement, so no `#GP` +traces are printed with interrupts disabled during boot. `CONFIG_SCHED_MC_PRIO` +is off as well because it selects both cpufreq and `intel_pstate`. Production +kernels must not enable the soft-lockup or hung-task detectors. Changing an +archive hash or patch invalidates both source and object caches; changing the +input configuration invalidates the object cache. The kernel keeps the code it generates at run time read-only. On x86, only `CONFIG_STRICT_MODULE_RWX` makes such memory read-only and executable diff --git a/kernel/config-microvm b/kernel/config-microvm index 6fdbbfa5b..5c9421b6f 100644 --- a/kernel/config-microvm +++ b/kernel/config-microvm @@ -319,13 +319,11 @@ CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_HYPERVISOR_GUEST=y CONFIG_PARAVIRT=y # CONFIG_PARAVIRT_SPINLOCKS is not set -CONFIG_X86_HV_CALLBACK_VECTOR=y # CONFIG_XEN is not set -CONFIG_KVM_GUEST=y -CONFIG_ARCH_CPUIDLE_HALTPOLL=y +# CONFIG_KVM_GUEST is not set +# CONFIG_ARCH_CPUIDLE_HALTPOLL is not set # CONFIG_PVH is not set # CONFIG_PARAVIRT_TIME_ACCOUNTING is not set -CONFIG_PARAVIRT_CLOCK=y # CONFIG_ACRN_GUEST is not set # CONFIG_BHYVE_GUEST is not set CONFIG_CC_HAS_MARCH_NATIVE=y @@ -498,8 +496,6 @@ CONFIG_CPU_IDLE=y # CONFIG_CPU_IDLE_GOV_LADDER is not set CONFIG_CPU_IDLE_GOV_MENU=y # CONFIG_CPU_IDLE_GOV_TEO is not set -CONFIG_CPU_IDLE_GOV_HALTPOLL=y -CONFIG_HALTPOLL_CPUIDLE=y # end of CPU Idle # CONFIG_INTEL_IDLE is not set diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 0d5168561..11dc7c3ab 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -83,6 +83,7 @@ class KernelBuildConstants: "CONFIG_HYPERVISOR_GUEST=y", "CONFIG_PARAVIRT=y", "# CONFIG_HYPERV is not set", + "# CONFIG_KVM_GUEST is not set", "# CONFIG_CPU_FREQ is not set", ) # x86 makes runtime-generated code, such as the ITS mitigation thunks, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index aac9fd9a2..502795757 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -6079,6 +6079,10 @@ def test_checked_in_config_meets_time_abi_kernel_contract(self): "CONFIG_X86_INTEL_PSTATE=y", "CONFIG_SCHED_MC_PRIO=y", "CONFIG_HYPERV=y", + "CONFIG_KVM_GUEST=y", + "CONFIG_PARAVIRT_CLOCK=y", + "CONFIG_HALTPOLL_CPUIDLE=y", + "CONFIG_ARCH_CPUIDLE_HALTPOLL=y", ): with self.subTest(removed=removed): self.assertNotIn(removed, configured) From 149dc68e14411aa8129f1663860d3c6ec2a2603e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 03:58:13 -0700 Subject: [PATCH 097/188] guest: run the time ABI through nvx-time at boot and restore Time ABI v1 moves the guest's clock duties into nvx-time. Wire it in: - init mounts /run and runs nvx-time boot right after /proc, /sys, and /dev, before any other guest work and in every guest mode, so the boot checks run and the time daemon starts exactly once. A boot check that cannot run powers the VM off with the conformance status. - nvx-init-agent reuses that /run mount instead of hiding the time state under a new tmpfs, and nvx-container-enter bind-mounts /run/nvx/time read-only into the workload at the same path, refusing image symlinks on the way. - nvx-snapshot follows the spec's snapshot-agent steps. nvx-time pre-capture runs the capture checks and suppresses the stall detectors before the barriers. nvx-time capture writes the request and, in a restored process, reads packet v4 and sets the wall clock. The agent then activates CPUs and memory from the packet metadata and repairs entropy and identity per tier as before, and nvx-time restore-finish runs the restore checks and acknowledges only when the packet sets ACK_REQUIRED. A rejected capture thaws the barriers and then restores the stall detectors. - nvx-reseed no longer polls the CMOS RTC, and nvx-port-io drops the restore packet v1 to v3 parser. Its generation-ID read now takes the portb lock that every guest user of the selectors shares. An untiered restore without targets still leaves entropy and identity to its caller; the entropy stays in /run/nvx/restore-entropy for it. This needs OpenVMM's restore packet v4, time-sample window, and Hyper-V identity, so it is held until they land. The harness's snapshot-core script still reads the version-1 packet itself and must change with them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/guests/ubuntu-guest.md | 9 +- guest/alpine/nvx-container-enter | 13 +++ guest/common/init | 10 ++ guest/common/nvx-init-agent | 3 +- guest/common/nvx-port-io.c | 162 ++++++++------------------- guest/common/nvx-reseed.c | 132 ---------------------- guest/common/nvx-snapshot | 184 ++++++++++++++----------------- scripts/test_microvm_tests.py | 32 ++++-- 8 files changed, 179 insertions(+), 366 deletions(-) diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index 00bd180d3..c81c88a6d 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -437,7 +437,8 @@ The Ubuntu initramfs boot remains identical to Alpine at the machine level: 3. OpenVMM prepends its xe9/hvc console parameters and fixed virtio-mmio discovery. 4. `/init` mounts procfs, sysfs, and devtmpfs, switches kernel module loading - off, and mounts tmpfs. + off, mounts `/run`, and runs the time ABI boot check (`nvx-time boot`), + which also starts the time daemon, before it mounts tmpfs. 5. `/init` configures loopback, optional static networking, and optional HostFs. 6. `/init` handles an explicit `nvx_exec` workload or opens a root shell. @@ -466,7 +467,11 @@ The first supported Ubuntu initramfs must provide: - HostFs mounting; - clean `/sbin/nvx-exit`; - SMP boot with 1, 2, 4, and 8 vCPUs; and -- snapshot and restore for blockless test scenarios. +- snapshot and restore for blockless test scenarios, including the time ABI + guest obligations that the common `nvx-time` and `nvx-snapshot` implement + for every guest: the boot conformance check, the violation watcher and + wall-clock discipline, restore packet v4 with CPU activation, and the RCU + grace-period release. Alpine-only benchmark scenarios that depend on extra shell tools may remain Alpine-only initially. The test selector must reject unsupported Ubuntu diff --git a/guest/alpine/nvx-container-enter b/guest/alpine/nvx-container-enter index 577137ceb..f4c72bdd5 100644 --- a/guest/alpine/nvx-container-enter +++ b/guest/alpine/nvx-container-enter @@ -46,6 +46,19 @@ rm -f "$rootfs/etc/machine-id" mount --bind "$workload_machine_id" "$rootfs/etc/machine-id" mount -o remount,bind,ro,nosuid,nodev "$rootfs/etc/machine-id" +# Workloads read the time ABI discontinuity state at the guest's path. No +# path component may be an image symlink, which would redirect the mount. +time_state=$rootfs +for component in run nvx time; do + time_state=$time_state/$component + if [ -L "$time_state" ] || { [ -e "$time_state" ] && [ ! -d "$time_state" ]; }; then + exit 125 + fi + mkdir -p "$time_state" +done +mount --bind /run/nvx/time "$time_state" +mount -o remount,bind,ro,nosuid,nodev,noexec "$time_state" + mknod -m 666 "$rootfs/dev/null" c 1 3 mknod -m 666 "$rootfs/dev/zero" c 1 5 mknod -m 666 "$rootfs/dev/full" c 1 7 diff --git a/guest/common/init b/guest/common/init index 294646f43..d8e85a626 100644 --- a/guest/common/init +++ b/guest/common/init @@ -15,6 +15,16 @@ if [ -e /proc/sys/kernel/modules_disabled ] && sleep 3600 done fi +# The time ABI boot check runs next, before any other guest work, in every +# guest mode. It powers the VM off itself on a violation and leaves its state +# and daemon under /run/nvx/time, so /run is mounted first and kept. +mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run +if ! /sbin/nvx-time boot; then + /sbin/nvx-exit 193 + while :; do + sleep 3600 + done +fi mount -t tmpfs none /tmp export PATH=/usr/sbin:/usr/bin:/sbin:/bin HOME=/root TERM=linux ifconfig lo 127.0.0.1 up diff --git a/guest/common/nvx-init-agent b/guest/common/nvx-init-agent index 4c10ceb80..ba69e4de9 100644 --- a/guest/common/nvx-init-agent +++ b/guest/common/nvx-init-agent @@ -230,7 +230,8 @@ teardown() { } mkdir -p "$layers" "$scratch" "$rootfs" -mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run || +# init mounts /run before the time ABI boot check, whose state lives there. +mountpoint -q /run || mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run || fatal "failed to mount runtime tmpfs" mkdir -p "$layers" "$scratch" "$rootfs" configure_cgroups diff --git a/guest/common/nvx-port-io.c b/guest/common/nvx-port-io.c index 5b55f2a78..478c3d1d1 100644 --- a/guest/common/nvx-port-io.c +++ b/guest/common/nvx-port-io.c @@ -1,4 +1,4 @@ -#define _POSIX_C_SOURCE 200809L +#define _DEFAULT_SOURCE #include #include @@ -7,27 +7,16 @@ #include #include #include +#include #include #include #define PORT_MAX UINT16_MAX -#define RESTORE_HEADER_SIZE 19 -#define RESTORE_ENTROPY_SIZE 64 -#define RESTORE_RANGE_SIZE 16 -#define RESTORE_PACKET_SELECT 0xa5 #define GENERATION_ID_SELECT 0xa6 #define GENERATION_ID_SIZE 16 #define STATUS_GENERATION_ID_AVAILABLE 32 -#define RESTORE_PACKET_MAX_SIZE \ - (RESTORE_HEADER_SIZE + 2 + UINT8_MAX * RESTORE_RANGE_SIZE + \ - RESTORE_ENTROPY_SIZE) - -static const unsigned char RESTORE_HEADER_V1[RESTORE_HEADER_SIZE] = - "OPENVMM_ENTROPY_V1"; -static const unsigned char RESTORE_HEADER_V2[RESTORE_HEADER_SIZE] = - "OPENVMM_ENTROPY_V2"; -static const unsigned char RESTORE_HEADER_V3[RESTORE_HEADER_SIZE] = - "OPENVMM_ENTROPY_V3"; +#define PORTB_LOCK_DIRECTORY "/run/nvx" +#define PORTB_LOCK_PATH PORTB_LOCK_DIRECTORY "/portb.lock" static int parse_u64(const char *text, uint64_t *value) { @@ -102,6 +91,33 @@ static int read_port_bytes(int port, uint64_t offset, unsigned char *buffer, return 0; } +// Selector transactions on portb are serialized with the time ABI's lock. +static int lock_portb(void) +{ + int lock; + + if (mkdir(PORTB_LOCK_DIRECTORY, 0755) != 0 && errno != EEXIST) { + fprintf(stderr, "nvx-port-io: mkdir %s: %s\n", PORTB_LOCK_DIRECTORY, + strerror(errno)); + return -1; + } + lock = open(PORTB_LOCK_PATH, O_RDWR | O_CREAT | O_CLOEXEC, S_IRUSR | S_IWUSR); + if (lock < 0) { + fprintf(stderr, "nvx-port-io: open %s: %s\n", PORTB_LOCK_PATH, + strerror(errno)); + return -1; + } + while (flock(lock, LOCK_EX) != 0) { + if (errno != EINTR) { + fprintf(stderr, "nvx-port-io: flock %s: %s\n", PORTB_LOCK_PATH, + strerror(errno)); + close(lock); + return -1; + } + } + return lock; +} + static int close_port(int port) { if (close(port) == 0) { @@ -126,113 +142,26 @@ static int print_hex(const unsigned char *bytes, size_t size) return 0; } -static uint64_t read_le_u64(const unsigned char *bytes) -{ - uint64_t value = 0; - for (size_t index = 0; index < sizeof(value); ++index) { - value |= (uint64_t)bytes[index] << (index * 8); - } - return value; -} - -static int read_restore_packet(uint64_t data_offset, uint64_t select_offset, - const char *path) -{ - unsigned char packet[RESTORE_PACKET_MAX_SIZE]; - unsigned int version; - unsigned int online_count = 0; - unsigned int range_count = 0; - size_t packet_size = RESTORE_HEADER_SIZE; - size_t payload_size; - int port = open("/dev/port", O_RDWR | O_CLOEXEC); - if (port < 0) { - fprintf(stderr, "nvx-port-io: open /dev/port: %s\n", strerror(errno)); - return 1; - } - - if (write_port_byte(port, select_offset, RESTORE_PACKET_SELECT) != 0 || - read_port_bytes(port, data_offset, packet, RESTORE_HEADER_SIZE) != 0) { - close(port); - return 1; - } - if (memcmp(packet, RESTORE_HEADER_V1, RESTORE_HEADER_SIZE) == 0) { - version = 1; - payload_size = RESTORE_ENTROPY_SIZE; - } else if (memcmp(packet, RESTORE_HEADER_V2, RESTORE_HEADER_SIZE) == 0) { - version = 2; - payload_size = 1 + RESTORE_ENTROPY_SIZE; - } else if (memcmp(packet, RESTORE_HEADER_V3, RESTORE_HEADER_SIZE) == 0) { - version = 3; - if (read_port_bytes(port, data_offset, packet + packet_size, 2) != 0) { - close(port); - return 1; - } - online_count = packet[packet_size]; - range_count = packet[packet_size + 1]; - packet_size += 2; - payload_size = - (size_t)range_count * RESTORE_RANGE_SIZE + RESTORE_ENTROPY_SIZE; - } else { - fprintf(stderr, "nvx-port-io: restore packet has an invalid header\n"); - close(port); - return 1; - } - - if (read_port_bytes(port, data_offset, packet + packet_size, payload_size) != - 0) { - close(port); - return 1; - } - packet_size += payload_size; - if (version == 2) { - online_count = packet[RESTORE_HEADER_SIZE]; - } - if (close_port(port) != 0) { - return 1; - } - - int output = - open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, S_IRUSR | S_IWUSR); - if (output < 0) { - fprintf(stderr, "nvx-port-io: open %s: %s\n", path, strerror(errno)); - return 1; - } - if (write_all(output, packet, packet_size) != 0) { - close(output); - return 1; - } - if (close(output) != 0) { - fprintf(stderr, "nvx-port-io: close %s: %s\n", path, strerror(errno)); - return 1; - } - printf("%u %u %u", version, online_count, range_count); - if (version == 3) { - size_t range_offset = RESTORE_HEADER_SIZE + 2; - for (unsigned int index = 0; index < range_count; ++index) { - uint64_t start = read_le_u64(packet + range_offset); - uint64_t length = - read_le_u64(packet + range_offset + sizeof(uint64_t)); - printf(" %" PRIu64 " %" PRIu64, start, length); - range_offset += RESTORE_RANGE_SIZE; - } - } - putchar('\n'); - return 0; -} - static int read_generation_id(uint64_t data_offset, uint64_t select_offset, const char *path) { unsigned char generation_id[GENERATION_ID_SIZE]; unsigned char status; - int port = open("/dev/port", O_RDWR | O_CLOEXEC); + int lock = lock_portb(); + int port; + if (lock < 0) { + return 1; + } + port = open("/dev/port", O_RDWR | O_CLOEXEC); if (port < 0) { fprintf(stderr, "nvx-port-io: open /dev/port: %s\n", strerror(errno)); + close(lock); return 1; } if (read_port_byte(port, select_offset, &status) != 0) { close(port); + close(lock); return 1; } if ((status & STATUS_GENERATION_ID_AVAILABLE) == 0) { @@ -240,14 +169,17 @@ static int read_generation_id(uint64_t data_offset, uint64_t select_offset, "nvx-port-io: VM generation ID is unavailable on port 0x%llx\n", (unsigned long long)select_offset); close(port); + close(lock); return 1; } if (write_port_byte(port, select_offset, GENERATION_ID_SELECT) != 0 || read_port_bytes(port, data_offset, generation_id, sizeof(generation_id)) != 0) { close(port); + close(lock); return 1; } + close(lock); if (close_port(port) != 0) { return 1; } @@ -312,9 +244,8 @@ static int write_u8(uint64_t offset, uint64_t value) static void usage(void) { fprintf(stderr, - "usage: nvx-port-io read-restore-packet DATA_PORT SELECT_PORT " - "OUTPUT | read-generation-id DATA_PORT SELECT_PORT [OUTPUT] | " - "read-u8 PORT | write-u8 PORT VALUE\n"); + "usage: nvx-port-io read-generation-id DATA_PORT SELECT_PORT " + "[OUTPUT] | read-u8 PORT | write-u8 PORT VALUE\n"); } int main(int argc, char **argv) @@ -331,11 +262,6 @@ int main(int argc, char **argv) parse_u64(argv[3], &value) && value <= UINT8_MAX) { return write_u8(offset, value); } - if (argc == 5 && strcmp(argv[1], "read-restore-packet") == 0 && - parse_u64(argv[3], &select_offset) && - select_offset <= PORT_MAX) { - return read_restore_packet(offset, select_offset, argv[4]); - } if ((argc == 4 || argc == 5) && strcmp(argv[1], "read-generation-id") == 0 && parse_u64(argv[3], &select_offset) && diff --git a/guest/common/nvx-reseed.c b/guest/common/nvx-reseed.c index f202b3e44..3d1142aab 100644 --- a/guest/common/nvx-reseed.c +++ b/guest/common/nvx-reseed.c @@ -3,25 +3,15 @@ #include #include #include -#include #include #include #include #include -#include #include #define SEED_BYTES 64 #define GENERATION_ID_BYTES 16 #define RANDOM_SAMPLE_BYTES 32 -#define CMOS_INDEX_PORT 0x70 -#define CMOS_DATA_PORT 0x71 -#define CMOS_STATUS_A 0x0a -#define CMOS_STATUS_B 0x0b -#define CMOS_UPDATE_IN_PROGRESS 0x80 -#define CMOS_24_HOUR 0x02 -#define CMOS_BINARY 0x04 -#define CMOS_CENTURY 0x32 struct seed_payload { int entropy_count; @@ -128,125 +118,6 @@ static int print_random_sample(void) return print_hex(sample, sizeof(sample)); } -static int read_cmos_register(int fd, uint8_t index, uint8_t *value) -{ - if (pwrite(fd, &index, 1, CMOS_INDEX_PORT) != 1 || - pread(fd, value, 1, CMOS_DATA_PORT) != 1) { - fprintf(stderr, "nvx-reseed: CMOS register 0x%02x: %s\n", index, - strerror(errno)); - return -1; - } - return 0; -} - -static uint8_t from_bcd(uint8_t value) -{ - return (uint8_t)((value & 0x0f) + ((value >> 4) * 10)); -} - -struct rtc_sample { - uint8_t second; - uint8_t minute; - uint8_t hour; - uint8_t day; - uint8_t month; - uint8_t year; - uint8_t century; - uint8_t status_b; -}; - -static int read_rtc_sample(int fd, struct rtc_sample *sample) -{ - uint8_t status_a; - for (int attempt = 0; attempt < 1000; ++attempt) { - if (read_cmos_register(fd, CMOS_STATUS_A, &status_a) != 0) { - return -1; - } - if ((status_a & CMOS_UPDATE_IN_PROGRESS) == 0) { - break; - } - if (attempt == 999) { - fprintf(stderr, "nvx-reseed: CMOS update did not complete\n"); - return -1; - } - } - - return read_cmos_register(fd, 0x00, &sample->second) || - read_cmos_register(fd, 0x02, &sample->minute) || - read_cmos_register(fd, 0x04, &sample->hour) || - read_cmos_register(fd, 0x07, &sample->day) || - read_cmos_register(fd, 0x08, &sample->month) || - read_cmos_register(fd, 0x09, &sample->year) || - read_cmos_register(fd, CMOS_CENTURY, &sample->century) || - read_cmos_register(fd, CMOS_STATUS_B, &sample->status_b); -} - -static int refresh_wall_clock(void) -{ - int port = open("/dev/port", O_RDWR | O_CLOEXEC); - if (port < 0) { - fprintf(stderr, "nvx-reseed: open /dev/port: %s\n", strerror(errno)); - return -1; - } - - struct rtc_sample first; - struct rtc_sample second; - int matched = 0; - for (int attempt = 0; attempt < 100; ++attempt) { - if (read_rtc_sample(port, &first) != 0 || - read_rtc_sample(port, &second) != 0) { - close(port); - return -1; - } - if (memcmp(&first, &second, sizeof(first)) == 0) { - matched = 1; - break; - } - } - close(port); - if (!matched) { - fprintf(stderr, "nvx-reseed: failed to obtain a stable CMOS sample\n"); - return -1; - } - - uint8_t hour = first.hour; - int is_pm = hour & 0x80; - hour &= 0x7f; - if ((first.status_b & CMOS_BINARY) == 0) { - first.second = from_bcd(first.second); - first.minute = from_bcd(first.minute); - hour = from_bcd(hour); - first.day = from_bcd(first.day); - first.month = from_bcd(first.month); - first.year = from_bcd(first.year); - first.century = from_bcd(first.century); - } - if ((first.status_b & CMOS_24_HOUR) == 0) { - hour = (uint8_t)((hour % 12) + (is_pm ? 12 : 0)); - } - - struct tm utc = { - .tm_sec = first.second, - .tm_min = first.minute, - .tm_hour = hour, - .tm_mday = first.day, - .tm_mon = first.month - 1, - .tm_year = first.century * 100 + first.year - 1900, - .tm_isdst = 0, - }; - time_t seconds = timegm(&utc); - if (seconds < 0) { - fprintf(stderr, "nvx-reseed: CMOS date is outside the supported range\n"); - return -1; - } - struct timespec time = {.tv_sec = seconds, .tv_nsec = 0}; - if (clock_settime(CLOCK_REALTIME, &time) != 0) { - fprintf(stderr, "nvx-reseed: clock_settime: %s\n", strerror(errno)); - return -1; - } - return 0; -} - int main(int argc, char **argv) { if (argc == 2 && strcmp(argv[1], "--sample") == 0) { @@ -299,9 +170,6 @@ int main(int argc, char **argv) return 1; } close(random); - if (refresh_wall_clock() != 0) { - return 1; - } if (previous_generation_id != NULL && print_hex(seed.bytes, GENERATION_ID_BYTES) != 0) { return 1; diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 6ac3430da..50871b15f 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -26,13 +26,14 @@ esac runtime=/run/nvx container_cgroup=/sys/fs/cgroup/container scratch_mount=$runtime/scratch -RESTORE_PACKET_AVAILABLE=2 -RESTORE_PROCESSOR_TARGET_AVAILABLE=4 -RESTORE_MEMORY_TARGET_AVAILABLE=8 -RESTORE_MEMORY_EXPANSION_AVAILABLE=16 +# Restore packet v4 flags (doc/design/time-abi.md). +PACKET_MEMORY_TARGET=2 +PACKET_ACK_REQUIRED=4 container_frozen=false scratch_frozen=false +stall_detectors_suppressed=false post_restore_pending=false +restore_new_cpus=none mkdir -p "$runtime" console_status() { @@ -64,6 +65,11 @@ cleanup() { fail_closed "failed to thaw the workload cgroup" container_frozen=false fi + if [ "$stall_detectors_suppressed" = true ]; then + /sbin/nvx-time cancel-capture || + fail_closed "failed to restore the stall detectors" + stall_detectors_suppressed=false + fi exit "$status" } @@ -86,14 +92,6 @@ cmdline_value() { return 1 } -read_port_byte() { - /sbin/nvx-port-io read-u8 "$1" -} - -write_port_byte() { - /sbin/nvx-port-io write-u8 "$1" "$2" -} - read_memtotal_kib() { while IFS=' ' read -r key value _; do if [ "$key" = MemTotal: ]; then @@ -134,6 +132,9 @@ activate_restore_processors() { esac [ "$before_count" -le "$target" ] || fail_closed "restore target $target is below the online CPU count $before_count" + if [ "$before_count" -lt "$target" ]; then + restore_new_cpus=$before_count-$((target - 1)) + fi cpu=1 while [ "$cpu" -lt "$target" ]; do @@ -276,7 +277,7 @@ elif [ "$snapshot_tier" != legacy ]; then exit 1 fi -generation_id=$(/sbin/nvx-port-io read-generation-id 233 234) || +generation_id=$(/sbin/nvx-time generation-id) || fail_closed "failed to read the VM generation ID" [ "${#generation_id}" -eq 32 ] || fail_closed "VM generation ID is not 16 bytes" @@ -284,75 +285,49 @@ case "$generation_id" in *[!0-9a-f]*) fail_closed "VM generation ID is not hexadecimal" ;; esac export NVX_VM_GENERATION_ID="$generation_id" +entropy=$runtime/restore-entropy +# Restore steps 9 to 13 of the time ABI (doc/design/time-abi.md). nvx-time +# capture has already read restore packet v4, checked its generation, set +# the wall clock, and written the entropy. Its metadata is FLAGS, +# ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by START +# and LENGTH for each memory range. post_restore() { post_restore_pending=true console_status "NVX-POST-RESTORE-STAGE: packet" - restore_packet=$runtime/restore-packet - entropy=$runtime/restore-entropy - packet_metadata=$( - /sbin/nvx-port-io read-restore-packet 233 234 "$restore_packet" - ) || fail_closed "failed to read restore packet" - # Metadata starts with version, online CPU count, and range count. - # shellcheck disable=SC2086 - set -- $packet_metadata - [ "$#" -ge 3 ] || fail_closed "restore packet metadata is invalid" - packet_version=$1 - packet_online_count=$2 - packet_memory_range_count=$3 - shift 3 - restore_online_count= - restore_memory_range_count= - case "$packet_version" in - 1) - [ "$#" -eq 0 ] || - fail_closed "version-1 restore packet has unexpected metadata" - ;; - 2) - [ "$#" -eq 0 ] || - fail_closed "version-2 restore packet has unexpected metadata" - restore_online_count=$packet_online_count - ;; - 3) - restore_online_count=$packet_online_count - restore_memory_range_count=$packet_memory_range_count + [ "$#" -ge 4 ] || fail_closed "restore packet metadata is invalid" + packet_flags=$1 + restore_online_count=$2 + restore_memory_range_count=$3 + shift 4 + case "$packet_flags:$restore_online_count:$restore_memory_range_count" in + *[!0-9:]* | :* | *::* | *:) + fail_closed "restore packet metadata is invalid" ;; - *) fail_closed "restore packet has an invalid header" ;; esac - if [ "$packet_version" -eq 3 ]; then - case "$restore_online_count" in - 0) restore_online_count= ;; - 1 | 2 | 4 | 8) ;; - *) fail_closed "restore packet has an invalid online CPU count" ;; - esac - case "$restore_memory_range_count" in - '' | *[!0-9]*) fail_closed "restore packet has an invalid memory range count" ;; - esac - elif [ "$packet_version" -eq 2 ]; then - case "$restore_online_count" in - 1 | 2 | 4 | 8) ;; - *) fail_closed "restore packet has an invalid online CPU count" ;; - esac + case "$restore_online_count" in + 0 | 1 | 2 | 4 | 8) ;; + *) fail_closed "restore packet has an invalid online CPU count" ;; + esac + if [ "$restore_online_count" -ne 0 ]; then + activate_restore_processors "$restore_online_count" + fi + if [ $((packet_flags & PACKET_MEMORY_TARGET)) -ne 0 ]; then + activate_restore_memory "$restore_memory_range_count" "$@" + elif [ "$restore_memory_range_count" -ne 0 ]; then + fail_closed "restore packet has memory ranges without a memory target" fi + # An untiered restore without processors or memory to add leaves the + # entropy and identity to its caller, as before. if [ "$snapshot_tier" = legacy ] && - [ -z "$restore_online_count" ] && - [ -z "$restore_memory_range_count" ]; then - rm -f "$restore_packet" - post_restore_pending=false + [ "$restore_online_count" -eq 0 ] && + [ "$restore_memory_range_count" -eq 0 ]; then + finish_restore "$packet_flags" return fi - if [ -n "$restore_online_count" ]; then - activate_restore_processors "$restore_online_count" - fi - if [ -n "$restore_memory_range_count" ]; then - activate_restore_memory "$restore_memory_range_count" "$@" - fi - console_status "NVX-POST-RESTORE-STAGE: entropy" - tail -c 64 "$restore_packet" >"$entropy" || - fail_closed "failed to extract restore entropy" if [ "$snapshot_tier" = instance-checkpoint ]; then generation_id=$( /sbin/nvx-reseed --generation-only "$entropy" "$generation_id" @@ -402,10 +377,22 @@ post_restore() { fi fi - rm -f "$restore_packet" "$entropy" + rm -f "$entropy" + finish_restore "$packet_flags" +} + +# Restore steps 11 to 13: the restore checks, the acknowledgement when the +# host gated its input, and the daemon's grace-period release, which runs off +# the restore latency path. +finish_restore() { console_status "NVX-POST-RESTORE-STAGE: acknowledge" - write_port_byte 1541 2 || - fail_closed "failed to release the VMM restore gate" + if [ $(($1 & PACKET_ACK_REQUIRED)) -ne 0 ]; then + /sbin/nvx-time restore-finish --ack --new-cpus "$restore_new_cpus" || + fail_closed "failed to finish the restore" + else + /sbin/nvx-time restore-finish --new-cpus "$restore_new_cpus" || + fail_closed "failed to finish the restore" + fi post_restore_pending=false echo "NVX-POST-RESTORE-OK: tier=$snapshot_tier" } @@ -441,6 +428,11 @@ elif [ "$scratch_policy" = fresh ]; then exit 1 fi +# Capture steps 1 to 3: the capture checks and the stall-detector +# suppression. They never wait. +/sbin/nvx-time pre-capture || exit 1 +stall_detectors_suppressed=true + if [ "$has_scratch" = true ] && [ "$scratch_policy" = paired ]; then if [ ! -w "$container_cgroup/cgroup.freeze" ] || [ ! -r "$container_cgroup/cgroup.events" ]; then @@ -471,40 +463,26 @@ if [ "$has_scratch" = true ] && [ "$scratch_policy" = paired ]; then fi fi -# A successful capture terminates the source before this write returns. On a -# rejected capture, the trap thaws every guest-owned barrier before returning. +# A successful capture terminates the source before nvx-time's request +# returns. On a rejected capture, the trap thaws every guest-owned barrier +# and then restores the stall detectors. if [ "$scratch_policy" = fresh ]; then capture_request=0 else capture_request=1 fi -write_port_byte 1541 "$capture_request" || - fail_closed "failed to request snapshot capture" - -post_restore_pending=true -restore_status=$(read_port_byte 234) -case "$restore_status" in - '' | *[!0-9]*) fail_closed "failed to read restore status" ;; +capture_metadata=$( + /sbin/nvx-time capture "$capture_request" "$generation_id" "$entropy" +) || fail_closed "failed to request snapshot capture" +# The metadata is a list of integers and one hexadecimal ID. +# shellcheck disable=SC2086 +set -- $capture_metadata +case "${1:-}" in + 0) ;; + 1) + stall_detectors_suppressed=false + shift + post_restore "$@" + ;; + *) fail_closed "snapshot capture metadata is invalid" ;; esac -if [ $((restore_status & RESTORE_MEMORY_EXPANSION_AVAILABLE)) -ne 0 ] && - [ $((restore_status & RESTORE_MEMORY_TARGET_AVAILABLE)) -eq 0 ]; then - fail_closed "restore status advertises expansion without a memory target" -fi -if [ $((restore_status & RESTORE_PACKET_AVAILABLE)) -ne 0 ] && - [ "$snapshot_tier" = legacy ] && - [ $((restore_status & RESTORE_MEMORY_TARGET_AVAILABLE)) -ne 0 ] && - [ $((restore_status & RESTORE_PROCESSOR_TARGET_AVAILABLE)) -eq 0 ] && - [ $((restore_status & RESTORE_MEMORY_EXPANSION_AVAILABLE)) -eq 0 ]; then - memtotal_kib=$(read_memtotal_kib) || - fail_closed "failed to read MemTotal after memory activation" - echo "NVX-MEMORY-ONLINE-OK: added_bytes=0 memtotal_kib=$memtotal_kib elapsed_us=0" - echo "NVX-POST-RESTORE-OK: tier=$snapshot_tier" - post_restore_pending=false -elif [ $((restore_status & RESTORE_PACKET_AVAILABLE)) -ne 0 ] && - { [ "$snapshot_tier" != legacy ] || - [ $((restore_status & RESTORE_PROCESSOR_TARGET_AVAILABLE)) -ne 0 ] || - [ $((restore_status & RESTORE_MEMORY_EXPANSION_AVAILABLE)) -ne 0 ]; }; then - post_restore -else - post_restore_pending=false -fi diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 3c55b917e..db6c4ef46 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -1409,17 +1409,17 @@ def test_negative_output_read_delay_does_not_start_process(self): ) interaction.assert_not_called() - def test_snapshot_restore_uses_batched_port_io_and_zero_expansion_path(self): + def test_snapshot_restore_runs_the_time_abi_steps_through_nvx_time(self): snapshot = ( Path(__file__).parents[1] / "guest" / "common" / "nvx-snapshot" ).read_text(encoding="utf-8") self.assertIn( - '/sbin/nvx-port-io read-restore-packet 233 234 "$restore_packet"', + "generation_id=$(/sbin/nvx-time generation-id)", snapshot, ) self.assertIn( - "generation_id=$(/sbin/nvx-port-io read-generation-id 233 234)", + '/sbin/nvx-time capture "$capture_request" "$generation_id" "$entropy"', snapshot, ) self.assertIn( @@ -1431,10 +1431,14 @@ def test_snapshot_restore_uses_batched_port_io_and_zero_expansion_path(self): snapshot, ) self.assertIn('export NVX_VM_GENERATION_ID="$generation_id"', snapshot) - self.assertNotIn("dd if=/dev/port", snapshot) - self.assertNotIn("dd of=/dev/port", snapshot) + self.assertNotIn("nvx-port-io", snapshot) + self.assertNotIn("/dev/port", snapshot) self.assertIn('[ "$range_count" -eq 0 ]', snapshot) - self.assertIn("RESTORE_MEMORY_EXPANSION_AVAILABLE=16", snapshot) + self.assertIn("PACKET_ACK_REQUIRED=4", snapshot) + self.assertIn( + '/sbin/nvx-time restore-finish --ack --new-cpus "$restore_new_cpus"', + snapshot, + ) self.assertIn('console_status "NVX-SNAPSHOT-ERROR: $*"', snapshot) for stage in ("packet", "entropy", "identity", "runtime-hook", "acknowledge"): self.assertIn( @@ -1446,11 +1450,19 @@ def test_snapshot_restore_uses_batched_port_io_and_zero_expansion_path(self): 'memtotal_kib=$memtotal_kib elapsed_us=0"', snapshot, ) - zero_expansion_fast_path = snapshot.index( - "[ $((restore_status & RESTORE_MEMORY_EXPANSION_AVAILABLE)) -eq 0 ]" + # Capture steps 1 to 3 precede the barriers and the request. + pre_capture = snapshot.index("/sbin/nvx-time pre-capture || exit 1") + freeze = snapshot.index('echo 1 >"$container_cgroup/cgroup.freeze"') + capture = snapshot.index("/sbin/nvx-time capture") + self.assertLess(pre_capture, freeze) + self.assertLess(freeze, capture) + # A rejected capture thaws the barriers before it restores the stall + # detectors. + cleanup_start = snapshot.index("cleanup() {") + cleanup = snapshot[cleanup_start : snapshot.index("\n}\n", cleanup_start)] + self.assertLess( + cleanup.index("cgroup.freeze"), cleanup.index("nvx-time cancel-capture") ) - packet_restore = snapshot.index(" post_restore\n") - self.assertLess(zero_expansion_fast_path, packet_restore) def test_snapshot_console_diagnostics_are_nonfatal_and_ordered(self): shell = _posix_shell() From 87310ff8478f5c2cf8f329e951ab91d6561955c7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 11:47:38 -0700 Subject: [PATCH 098/188] guest: keep plain restores off the console A plain untiered restore, with no processors or memory to add, now runs post_restore, which printed two stage lines and a completion line that the release guest never printed on that path. Each console byte is a port exit, and the lines delayed the restored workload: on prometheus32 (KVM, 512 MiB) plain restores took 3.2 ms longer at 1 vCPU and 3.5 ms longer at 8 vCPUs (paired medians of 24 restores). Print them only when a restore activates processors or memory or runs a tier, as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-snapshot | 20 ++++++++-- scripts/test_microvm_tests.py | 73 +++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+), 4 deletions(-) diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 50871b15f..17d7201e9 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -294,7 +294,6 @@ entropy=$runtime/restore-entropy # and LENGTH for each memory range. post_restore() { post_restore_pending=true - console_status "NVX-POST-RESTORE-STAGE: packet" [ "$#" -ge 4 ] || fail_closed "restore packet metadata is invalid" packet_flags=$1 restore_online_count=$2 @@ -309,6 +308,17 @@ post_restore() { 0 | 1 | 2 | 4 | 8) ;; *) fail_closed "restore packet has an invalid online CPU count" ;; esac + # A plain untiered restore, with no processors or memory to add, prints + # nothing, as before: each console byte is a port exit on its latency + # path. + if [ "$snapshot_tier" = legacy ] && + [ "$restore_online_count" -eq 0 ] && + [ "$restore_memory_range_count" -eq 0 ] && + [ $((packet_flags & PACKET_MEMORY_TARGET)) -eq 0 ]; then + finish_restore "$packet_flags" quiet + return + fi + console_status "NVX-POST-RESTORE-STAGE: packet" if [ "$restore_online_count" -ne 0 ]; then activate_restore_processors "$restore_online_count" fi @@ -383,9 +393,11 @@ post_restore() { # Restore steps 11 to 13: the restore checks, the acknowledgement when the # host gated its input, and the daemon's grace-period release, which runs off -# the restore latency path. +# the restore latency path. A second argument of "quiet" omits the stage and +# completion lines of a plain untiered restore. finish_restore() { - console_status "NVX-POST-RESTORE-STAGE: acknowledge" + [ "${2:-}" = quiet ] || + console_status "NVX-POST-RESTORE-STAGE: acknowledge" if [ $(($1 & PACKET_ACK_REQUIRED)) -ne 0 ]; then /sbin/nvx-time restore-finish --ack --new-cpus "$restore_new_cpus" || fail_closed "failed to finish the restore" @@ -394,7 +406,7 @@ finish_restore() { fail_closed "failed to finish the restore" fi post_restore_pending=false - echo "NVX-POST-RESTORE-OK: tier=$snapshot_tier" + [ "${2:-}" = quiet ] || echo "NVX-POST-RESTORE-OK: tier=$snapshot_tier" } trap cleanup EXIT trap 'exit 129' HUP diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index db6c4ef46..aac9a22fa 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -1518,6 +1518,79 @@ def test_snapshot_console_diagnostics_are_nonfatal_and_ordered(self): "nvx-snapshot: synthetic restore failure; terminating the VM\n", ) + def test_plain_untiered_restore_prints_nothing(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + snapshot = ( + Path(__file__).parents[1] / "guest" / "common" / "nvx-snapshot" + ).read_text(encoding="utf-8") + helpers_start = snapshot.index("console_status() {") + helpers_end = snapshot.index("\n}\n\ncleanup()", helpers_start) + 3 + restore_start = snapshot.index("post_restore() {") + restore_end = snapshot.index("\n}\n", snapshot.index("finish_restore() {")) + 3 + functions = ( + snapshot[helpers_start:helpers_end] + snapshot[restore_start:restore_end] + ) + functions = functions.replace(">/dev/console", '>>"$console_log"') + functions = functions.replace("/sbin/nvx-exit", "nvx_exit") + functions = functions.replace("/sbin/nvx-time", "nvx_time") + generation_id = "0123456789abcdef0123456789abcdef" + + def restore(flags: int) -> tuple[str, str, str]: + with tempfile.TemporaryDirectory() as temporary: + result = subprocess.run( + [shell, "-s", "--", temporary], + input=( + "set -eu\n" + 'console_log="$1/console"\n' + 'calls="$1/nvx-time"\n' + ': >"$console_log"\n' + ': >"$calls"\n' + "snapshot_tier=legacy\n" + "PACKET_MEMORY_TARGET=2\n" + "PACKET_ACK_REQUIRED=4\n" + "restore_new_cpus=none\n" + "post_restore_pending=false\n" + "nvx_exit() { :; }\n" + 'nvx_time() { printf "%s\\n" "$*" >>"$calls"; }\n' + 'activate_restore_memory() { echo "memory $*"; }\n' + f"{functions}\n" + f"post_restore {flags} 0 0 {generation_id}\n" + 'echo "pending=$post_restore_pending"\n' + ), + text=True, + capture_output=True, + timeout=5, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + root = Path(temporary) + return ( + (root / "console").read_text(encoding="ascii"), + result.stdout, + (root / "nvx-time").read_text(encoding="ascii"), + ) + + # No processors or memory to add: no console bytes on the restore path. + self.assertEqual( + restore(0), ("", "pending=false\n", "restore-finish --new-cpus none\n") + ) + self.assertEqual( + restore(4), + ("", "pending=false\n", "restore-finish --ack --new-cpus none\n"), + ) + # A memory target keeps its stage and completion lines. + console, stdout, calls = restore(2) + self.assertEqual( + console, + "NVX-POST-RESTORE-STAGE: packet\nNVX-POST-RESTORE-STAGE: acknowledge\n", + ) + self.assertEqual( + stdout, "memory 0\nNVX-POST-RESTORE-OK: tier=legacy\npending=false\n" + ) + self.assertEqual(calls, "restore-finish --new-cpus none\n") + def test_console_log_persists_buffered_and_completed_output(self): with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) From 0e03977dc28dc03880c133643d6272b4ce9be429 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:19:14 -0700 Subject: [PATCH 099/188] guest: keep the time ABI checks off the restore and boot paths The time ABI guest was 3 to 5 ms slower than the release guest per restore on prometheus32 (KVM), and its boot check took 9.6 ms at 8 vCPUs. After a restore, every process the guest starts costs 2 to 4 ms (fork and exec fault in their pages), so the restore path now does only what must precede the workload: read the packet, set the clock, write the entropy, activate processors and memory, and acknowledge. - The restore checks (step 11) run in the time daemon after the acknowledgement, at SCHED_IDLE, before the grace-period release. A failed check still emits its event and powers off with status 193, and the restore marker still reports the checks. - An untiered restore with nothing to activate needs no shell work, so nvx-time capture --finish acknowledges the gate and signals the daemon itself and reports 2; no second helper process starts. - The boot check runs each CPU's checks in a thread pinned to it, beside the kernel-log, tick-mode, and cpuinfo checks, while the main thread runs the remaining checks and the initial sample. One online CPU runs them serially. prometheus32 (KVM) against the full release bundle, 512 MiB, 30 interleaved restores per cell, p50 at 1/2/4/8 vCPUs: restore -0.1/-2.1/-3.6/-2.9 ms, cold boot -3/-1/+4/+2 ms. Boot check: 1.5/1.9-2.5/4.3/5.0 ms, from 1.2/2.2/5.8/9.6 ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-snapshot | 13 +- guest/common/nvx-time.c | 568 ++++++++++++++++++++++++++++------ scripts/test_guest_time.py | 63 ++++ scripts/test_microvm_tests.py | 53 ++++ 4 files changed, 601 insertions(+), 96 deletions(-) diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 17d7201e9..57d756abb 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -391,8 +391,8 @@ post_restore() { finish_restore "$packet_flags" } -# Restore steps 11 to 13: the restore checks, the acknowledgement when the -# host gated its input, and the daemon's grace-period release, which runs off +# Restore steps 12 and 13: the acknowledgement when the host gated its input, +# then the daemon's restore checks and grace-period release, which run off # the restore latency path. A second argument of "quiet" omits the stage and # completion lines of a plain untiered restore. finish_restore() { @@ -483,8 +483,14 @@ if [ "$scratch_policy" = fresh ]; then else capture_request=1 fi +# An untiered restore with nothing to activate needs no shell work before +# the acknowledgement, so nvx-time finishes it and reports 2: no second +# process starts on the restore path. +finish_option= +[ "$snapshot_tier" = legacy ] && finish_option=--finish capture_metadata=$( - /sbin/nvx-time capture "$capture_request" "$generation_id" "$entropy" + /sbin/nvx-time capture "$capture_request" "$generation_id" "$entropy" \ + ${finish_option:+"$finish_option"} ) || fail_closed "failed to request snapshot capture" # The metadata is a list of integers and one hexadecimal ID. # shellcheck disable=SC2086 @@ -496,5 +502,6 @@ case "${1:-}" in shift post_restore "$@" ;; + 2) stall_detectors_suppressed=false ;; *) fail_closed "snapshot capture metadata is invalid" ;; esac diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 41fa0d23b..43c032137 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -44,6 +45,7 @@ #include #include #include +#include #include #include #include @@ -514,7 +516,8 @@ static int enable_ports(void) } // Reads SIZE bytes from PORT four bytes per exit; the device zero-fills -// past the end of the selected record. +// past the end of the selected record. A string read (rep insl) would take one +// exit, but OpenVMM's MSHV and WHP emulation raises #GP for it in user mode. static void portb_read(uint16_t port, uint8_t *buffer, size_t size) { for (size_t offset = 0; offset < size; offset += 4) { @@ -1417,25 +1420,16 @@ static void boot_log_scan(char *buffer, struct boot_log *log) // Reads the whole kernel log for C4 with one SYSLOG_ACTION_READ_ALL. The // caller has already positioned the daemon's /dev/kmsg descriptor at the end // of the log, so records logged in between are seen twice, never missed. -static int read_boot_log(struct boot_log *log) +// BUFFER holds SIZE + 1 bytes, the size SYSLOG_ACTION_SIZE_BUFFER reported. +static int read_boot_log_into(struct boot_log *log, char *buffer, int size) { - int size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0); - char *buffer; - int count; - int error; + int count = klogctl(SYSLOG_ACTION_READ_ALL, buffer, size); + int error = errno; - if (size <= 0) - return -1; - buffer = malloc((size_t)size + 1); - if (buffer == NULL) - return -1; - count = klogctl(SYSLOG_ACTION_READ_ALL, buffer, size); - error = errno; if (count >= 0) { buffer[count] = '\0'; boot_log_scan(buffer, log); } - free(buffer); errno = error; return count < 0 ? -1 : 0; } @@ -1460,6 +1454,10 @@ static bool check_failed_code(struct checks *checks, const char *id, const char *code, const char *format, ...) __attribute__((format(printf, 4, 5))); +// Serializes failure reports from the boot check's per-CPU threads, which +// share the console and the violation count. +static pthread_mutex_t g_report_lock = PTHREAD_MUTEX_INITIALIZER; + // Prints the failure line, emits CODE, and powers off with status 193. In // report-only mode it returns false and the run continues. static bool check_failed_code(struct checks *checks, const char *id, @@ -1477,11 +1475,13 @@ static bool check_failed_code(struct checks *checks, const char *id, snprintf(line, sizeof(line), "%s: v=1 phase=%s status=fail check=%s detail=\"%s\"\n", abi_prefix(), k_phase_names[checks->phase], id, escaped); + pthread_mutex_lock(&g_report_lock); console_write(line); checks->failures++; emit_event(code, "conformance", checks->phase, detail); if (!g_report_only) power_off(STATUS_CONFORMANCE); + pthread_mutex_unlock(&g_report_lock); return false; } @@ -2395,6 +2395,32 @@ static pid_t daemon_pid(void) return (pid_t)pid; } +// The daemon finishes a restore (restore step 13 and the deferred step 11) +// at idle priority, so that on few vCPUs it never takes the CPU from the +// restored workload. musl does not implement sched_setscheduler(), and the +// daemon is single-threaded, so the per-thread system call applies to it. +static void set_idle_priority(pid_t pid, bool idle) +{ + struct sched_param param = {0}; + + (void)syscall(SYS_sched_setscheduler, pid, idle ? SCHED_IDLE : SCHED_OTHER, + ¶m); +} + +// Restore steps 12 and 13 on the readiness path: the acknowledgement when the +// host gated its input, then hand the rest of the restore to DAEMON. Returns +// -1, without acknowledging, when no daemon runs; 1 when the daemon vanished +// after the acknowledgement. The caller has enabled the ports when ACK is set. +static int signal_restore(pid_t daemon, bool ack) +{ + if (daemon <= 0 || kill(daemon, 0) != 0) + return -1; + if (ack) + outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); + set_idle_priority(daemon, true); + return kill(daemon, SIGUSR1) == 0 ? 0 : 1; +} + // C10: the daemon runs, has recorded no violation, and no stall was counted. static bool check_daemon(struct checks *checks) { @@ -2465,6 +2491,156 @@ static bool check_cpus(struct checks *checks, const bool *selected, return ok; } +struct cpu_job { + pthread_t thread; + struct checks checks; + bool identity; + bool msrs; + bool running; + int cpu; +}; + +static void *cpu_job_run(void *argument) +{ + struct cpu_job *job = argument; + + if (pin_to_cpu(job->cpu) != 0) { + check_failed(&job->checks, "C1", "cannot run on cpu %d: %s", job->cpu, + strerror(errno)); + return NULL; + } + if (job->identity) { + check_identity(&job->checks, job->cpu); + check_time_bits(&job->checks, job->cpu); + } + if (job->msrs) + check_msrs(&job->checks, job->cpu, true); + return NULL; +} + +// A boot check that runs in its own thread beside the per-CPU checks, on a +// private copy of the checks. +struct side_job { + pthread_t thread; + struct checks checks; + void (*run)(struct checks *checks, void *context); + void *context; + bool running; +}; + +static void *side_job_run(void *argument) +{ + struct side_job *job = argument; + + job->run(&job->checks, job->context); + return NULL; +} + +// check_cpus() with every CPU in its own thread pinned to it, beside the +// SIDES checks, each in a thread of its own, while the caller runs FOREGROUND +// (when set). Running them in parallel keeps the cost from growing with the +// CPU count and leaves the caller on its warm CPU. The threads share one +// stack mapping, so creating one maps nothing. Each job works on a copy of +// CHECKS whose failures are added back; a job whose thread cannot start runs +// in the caller afterwards. +static void check_cpus_parallel(struct checks *checks, const bool *selected, + bool identity, bool msrs, + struct side_job *sides, int side_count, + void (*foreground)(struct checks *, void *), + void *foreground_context) +{ + enum { STACK_SIZE = 128 * 1024 }; + static struct cpu_job jobs[MAX_CPUS]; + bool serial[MAX_CPUS] = {false}; + bool any_serial = false; + pthread_attr_t attributes; + pthread_attr_t *attr = NULL; + size_t threads = (size_t)side_count; + size_t next = 0; + uint8_t *stacks; + + for (int cpu = 0; cpu < MAX_CPUS; cpu++) + threads += selected[cpu] ? 1 : 0; + // With one CPU, threads only add their own cost. + if (threads <= (size_t)side_count + 1) { + if (foreground != NULL) + foreground(checks, foreground_context); + for (int index = 0; index < side_count; index++) { + struct side_job *side = &sides[index]; + + side->checks = *checks; + side->checks.failures = 0; + side->run(&side->checks, side->context); + checks->failures += side->checks.failures; + } + check_cpus(checks, selected, identity, msrs); + return; + } + stacks = mmap(NULL, threads * STACK_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stacks != MAP_FAILED && pthread_attr_init(&attributes) == 0) + attr = &attributes; + for (int index = 0; index < side_count; index++) { + struct side_job *side = &sides[index]; + + side->checks = *checks; + side->checks.failures = 0; + if (attr != NULL) + pthread_attr_setstack(attr, stacks + STACK_SIZE * next++, + STACK_SIZE); + side->running = + pthread_create(&side->thread, attr, side_job_run, side) == 0; + } + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + struct cpu_job *job = &jobs[cpu]; + + job->running = false; + if (!selected[cpu]) + continue; + job->checks = *checks; + job->checks.failures = 0; + job->identity = identity; + job->msrs = msrs; + job->cpu = cpu; + if (attr != NULL) + pthread_attr_setstack(attr, stacks + STACK_SIZE * next++, + STACK_SIZE); + if (pthread_create(&job->thread, attr, cpu_job_run, job) == 0) + job->running = true; + else + serial[cpu] = any_serial = true; + } + if (foreground != NULL) + foreground(checks, foreground_context); + for (int index = 0; index < side_count; index++) { + struct side_job *side = &sides[index]; + + if (side->running) + pthread_join(side->thread, NULL); + else + side->run(&side->checks, side->context); + checks->failures += side->checks.failures; + } + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + struct cpu_job *job = &jobs[cpu]; + + if (!job->running) + continue; + pthread_join(job->thread, NULL); + checks->failures += job->checks.failures; + // C3 reads the partition-wide rates on CPU 0 only. + if (cpu == 0 && msrs) { + checks->tsc_hz = job->checks.tsc_hz; + checks->lapic_hz = job->checks.lapic_hz; + } + } + if (attr != NULL) + pthread_attr_destroy(attr); + if (stacks != MAP_FAILED) + munmap(stacks, threads * STACK_SIZE); + if (any_serial) + check_cpus(checks, serial, identity, msrs); +} // --------------------------------------------------------------------------- // Host time samples and clock adjustment // --------------------------------------------------------------------------- @@ -2616,6 +2792,7 @@ struct daemon { int release_status; int64_t release_deadline_ns; int64_t restore_elapsed_us; + int restore_failures; }; static void watcher_consume(void *context, const char *message) @@ -2829,11 +3006,37 @@ static void print_restore_marker(int failures, int cpus, uint64_t tsc_hz, console_write(line); } +static void run_restore_checks(struct checks *checks, const bool *new_cpus, + int new_count); + +static int parse_new_cpus(const char *text, bool *cpus) +{ + if (strcmp(text, "") == 0 || strcmp(text, "none") == 0) { + memset(cpus, 0, sizeof(bool) * MAX_CPUS); + return 0; + } + return parse_cpu_list(text, cpus); +} + +// Parses the restore record: the generation and the repair time that the +// capture helper wrote, and the CPUs that restore-finish onlined, when it ran. +// Returns the number of those CPUs, or -1 when the record is malformed. +static int parse_restore_record(const char *text, unsigned long long *generation, + long long *elapsed_us, bool *new_cpus) +{ + char cpus[128] = "none"; + + if (sscanf(text, "generation=%llu elapsed_us=%lld new_cpus=%127s", + generation, elapsed_us, cpus) < 2) + return -1; + return parse_new_cpus(cpus, new_cpus); +} + // Step 13 after the grace period ended (or the stall timeout passed): put // the watchdog settings back, run the deferred C7, print the restore marker, // and restart the discipline at the fast cadence. static void complete_restore(bool released, uint64_t tsc_hz, uint64_t lapic_hz, - int64_t elapsed_us) + int64_t elapsed_us, int failures) { struct checks checks; char detail[DETAIL_MAX]; @@ -2848,35 +3051,50 @@ static void complete_restore(bool released, uint64_t tsc_hz, uint64_t lapic_hz, check_failed(&checks, "C7", "cannot read the online CPU set"); else check_timer_list(&checks, TIMER_LIST_WAIT_NS); - print_restore_marker(checks.failures, checks.online_count, tsc_hz, lapic_hz, - elapsed_us); + print_restore_marker(failures + checks.failures, checks.online_count, + tsc_hz, lapic_hz, elapsed_us); + set_idle_priority(0, false); } +// Restore step 13, and step 11 after the acknowledgement: the restore checks +// fail fast as before, then the grace-period release runs in a child. static void start_restore(struct daemon *daemon) { + int64_t start = clock_ns(CLOCK_MONOTONIC); + bool new_cpus[MAX_CPUS]; + struct checks checks; char text[256]; unsigned long long generation; long long elapsed; int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; + int new_count; pid_t pid; if (daemon->release_pid > 0) return; + set_idle_priority(0, true); if (read_text(RESTORE_PATH, text, sizeof(text)) < 0 || - sscanf(text, "generation=%llu elapsed_us=%lld", &generation, - &elapsed) != 2) { + (new_count = parse_restore_record(text, &generation, &elapsed, + new_cpus)) < 0) { fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", PHASE_RESTORE, "the restore record is unreadable"); return; } g_generation = (uint32_t)generation; - daemon->restore_elapsed_us = elapsed; + if (checks_init(&checks, PHASE_RESTORE) != 0) + check_failed(&checks, "C6", "cannot read the CPU sets"); + else + run_restore_checks(&checks, new_cpus, new_count); + daemon->restore_failures = checks.failures; + daemon->restore_elapsed_us = + elapsed + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC; (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); pid = fork(); if (pid == 0) release_child(); if (pid < 0) { - complete_restore(false, daemon->tsc_hz, daemon->lapic_hz, elapsed); + complete_restore(false, daemon->tsc_hz, daemon->lapic_hz, + daemon->restore_elapsed_us, daemon->restore_failures); return; } daemon->release_pid = pid; @@ -2980,7 +3198,8 @@ static void daemon_loop(struct daemon *daemon) daemon->release_pid = 0; complete_restore(released, daemon->tsc_hz, daemon->lapic_hz, - daemon->restore_elapsed_us); + daemon->restore_elapsed_us, + daemon->restore_failures); daemon->polls = 0; daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); arm_timer(daemon); @@ -3081,16 +3300,84 @@ static void load_generation(void) g_generation = (uint32_t)state.generation; } +struct boot_log_job { + struct boot_log *log; + char *buffer; + int size; + bool read_log; +}; + +// C4 and K1 over the kernel log, beside the per-CPU checks. The caller +// allocates the buffer before the threads start: freeing a large allocation +// unmaps it, and an unmap in a process with running threads waits for every +// CPU running one of them. +static void boot_kernel_log(struct checks *checks, void *context) +{ + struct boot_log_job *job = context; + char detail[DETAIL_MAX]; + + if (!job->read_log) + return; + if (job->buffer == NULL) { + check_failed(checks, "C4", "no buffer for the %d-byte kernel log", + job->size); + return; + } + if (read_boot_log_into(job->log, job->buffer, job->size) != 0) { + check_failed(checks, "C4", "read the kernel log: %s", strerror(errno)); + return; + } + if (!boot_log_verdict(job->log, detail, sizeof(detail))) + check_failed(checks, "C4", "%s", detail); + // K1 is kernel hardening, not time: a writable and executable kernel + // mapping, such as unprotected ITS thunk pages. + if (job->log->wx[0] != '\0') + check_failed_code(checks, "K1", "G_KERNEL_WX", "%s", job->log->wx); +} + +// C7 beside the per-CPU checks. +static void boot_timer_list(struct checks *checks, void *context) +{ + (void)context; + check_timer_list(checks, TIMER_LIST_WAIT_NS); +} + +// C5 beside the per-CPU checks. +static void boot_cpuinfo(struct checks *checks, void *context) +{ + (void)context; + check_cpuinfo(checks, checks->online, true); +} + +struct boot_sample { + int64_t theta; + int64_t epsilon; + int64_t accepted_ns; +}; + +// The boot checks that need no CPU of their own, C12 included, run in the +// caller while the parallel jobs run. +static void boot_foreground(struct checks *checks, void *context) +{ + struct boot_sample *sample = context; + + check_cmdline(checks); + check_platform(checks); + check_clocksource(checks); + check_debug_watchdogs(checks); + if (check_initial_sample(checks, &sample->theta, &sample->epsilon)) + sample->accepted_ns = clock_ns(CLOCK_MONOTONIC); +} + static int cmd_boot(void) { int64_t start = clock_ns(CLOCK_MONOTONIC); + struct boot_log_job log_job; + struct side_job sides[3]; struct checks checks; struct boot_log log; struct time_state state; - char detail[DETAIL_MAX]; - int64_t theta = 0; - int64_t epsilon = 0; - int64_t accepted_ns = 0; + struct boot_sample sample = {0, 0, 0}; int kmsg; g_generation = 0; @@ -3099,46 +3386,50 @@ static int cmd_boot(void) check_failed(&checks, "C1", "cannot read the CPU sets"); return 1; } - check_cmdline(&checks); - check_platform(&checks); - check_clocksource(&checks); memset(&log, 0, sizeof(log)); + log_job.log = &log; + log_job.read_log = false; + log_job.size = klogctl(SYSLOG_ACTION_SIZE_BUFFER, NULL, 0); + log_job.buffer = log_job.size > 0 ? malloc((size_t)log_job.size + 1) : NULL; kmsg = open("/dev/kmsg", O_RDONLY | O_NONBLOCK | O_CLOEXEC); if (kmsg < 0 || lseek(kmsg, 0, SEEK_END) < 0) { check_failed(&checks, "C4", "open /dev/kmsg: %s", strerror(errno)); if (kmsg >= 0) close(kmsg); kmsg = -1; - } else if (read_boot_log(&log) != 0) { - check_failed(&checks, "C4", "read the kernel log: %s", - strerror(errno)); } else { - if (!boot_log_verdict(&log, detail, sizeof(detail))) - check_failed(&checks, "C4", "%s", detail); - // K1 is kernel hardening, not time: a writable and executable kernel - // mapping, such as unprotected ITS thunk pages. - if (log.wx[0] != '\0') - check_failed_code(&checks, "K1", "G_KERNEL_WX", "%s", log.wx); - } - check_cpuinfo(&checks, checks.online, true); - check_cpus(&checks, checks.online, true, true); - check_timer_list(&checks, TIMER_LIST_WAIT_NS); - check_debug_watchdogs(&checks); - if (check_initial_sample(&checks, &theta, &epsilon)) - accepted_ns = clock_ns(CLOCK_MONOTONIC); + log_job.read_log = true; + } + // The kernel log, the tick mode, and cpuinfo run beside the per-CPU + // checks, longest first; the other checks run in the caller meanwhile. + sides[0] = (struct side_job){.run = boot_kernel_log, .context = &log_job}; + sides[1] = (struct side_job){.run = boot_timer_list}; + sides[2] = (struct side_job){.run = boot_cpuinfo}; + check_cpus_parallel(&checks, checks.online, true, true, sides, + ARRAY_SIZE(sides), boot_foreground, &sample); + free(log_job.buffer); + memcpy(checks.cpu_khz, sides[2].checks.cpu_khz, sizeof(checks.cpu_khz)); + memcpy(checks.cpu_khz_known, sides[2].checks.cpu_khz_known, + sizeof(checks.cpu_khz_known)); + // C3 compares CPU 0's declared rate with cpuinfo once both have run. + if (checks.tsc_hz != 0 && checks.cpu_khz_known[0] && + checks.tsc_hz / 1000 != checks.cpu_khz[0]) + check_failed(&checks, "C3", + "cpu 0: floor(F / 1000) is %" PRIu64 + " kHz but cpu MHz is %" PRIu64 " kHz", + checks.tsc_hz / 1000, checks.cpu_khz[0]); state_init(&state); - state.synchronized = accepted_ns != 0; - state.samples = accepted_ns != 0; - state.offset_ns = accepted_ns != 0 ? theta : 0; - state.uncertainty_ns = accepted_ns != 0 ? epsilon : 0; + state.synchronized = sample.accepted_ns != 0; + state.samples = sample.accepted_ns != 0; + state.offset_ns = sample.accepted_ns != 0 ? sample.theta : 0; + state.uncertainty_ns = sample.accepted_ns != 0 ? sample.epsilon : 0; state.frequency_ppb = frequency_to_ppb(current_frequency()); if (rates_store(checks.tsc_hz, checks.lapic_hz) != 0 || state_store(&state) != 0) check_failed(&checks, "C10", "cannot publish the time state: %s", strerror(errno)); - else if (kmsg >= 0 && - start_daemon(kmsg, checks.tsc_hz, checks.lapic_hz, accepted_ns) != - 0) + else if (kmsg >= 0 && start_daemon(kmsg, checks.tsc_hz, checks.lapic_hz, + sample.accepted_ns) != 0) check_failed(&checks, "C10", "cannot start the time daemon: %s", strerror(errno)); print_marker(&checks, start); @@ -3177,15 +3468,6 @@ static void run_restore_checks(struct checks *checks, const bool *new_cpus, check_daemon(checks); } -static int parse_new_cpus(const char *text, bool *cpus) -{ - if (strcmp(text, "") == 0 || strcmp(text, "none") == 0) { - memset(cpus, 0, sizeof(bool) * MAX_CPUS); - return 0; - } - return parse_cpu_list(text, cpus); -} - static int cmd_check(int argc, char **argv) { int64_t start = clock_ns(CLOCK_MONOTONIC); @@ -3302,9 +3584,12 @@ static int repair_failed(const char *code, const char *detail) // Restore steps 6 to 8: read the packet with four-byte reads, validate it, // and set the wall clock from its bracketed UTC. The caller holds the portb -// lock. +// lock. With DAEMON set, an untiered restore with no processors or memory to +// activate needs no shell work before the acknowledgement, so steps 12 and 13 +// run here too and no second helper process starts; the metadata then begins +// with 2 instead of 1. static int repair_restore(const uint8_t *previous_id, const char *entropy_path, - int64_t start_ns) + int64_t start_ns, pid_t daemon) { static uint8_t body[PACKET_MAX_SIZE]; static struct restore_packet packet; @@ -3318,6 +3603,7 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, int64_t t1; int64_t sample_theta; int64_t sample_epsilon; + int finished = -1; int fd; t0 = clock_ns(CLOCK_REALTIME); @@ -3388,8 +3674,16 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, return repair_failed("G_REPAIR_CLOCK", detail); } format_hex(packet.entropy, GENERATION_ID_SIZE, id); - printf("1 %u %u %u %s", packet.flags, packet.online_vp_count, - packet.range_count, id); + if (daemon > 0 && packet.online_vp_count == 0 && packet.range_count == 0 && + (packet.flags & PACKET_MEMORY_TARGET) == 0) { + finished = signal_restore(daemon, + (packet.flags & PACKET_ACK_REQUIRED) != 0); + if (finished > 0) + fail_fatal(STATUS_CONFORMANCE, "G_CONFORMANCE_C10", "conformance", + PHASE_RESTORE, "the time daemon is not running"); + } + printf("%d %u %u %u %s", finished >= 0 ? 2 : 1, packet.flags, + packet.online_vp_count, packet.range_count, id); for (unsigned index = 0; index < packet.range_count; index++) printf(" %" PRIu64 " %" PRIu64, packet.ranges[index][0], packet.ranges[index][1]); @@ -3411,14 +3705,16 @@ static int cmd_capture(int argc, char **argv) long request; int64_t start; unsigned status; + pid_t daemon = -1; int result; int lock; - if (argc != 3 || (request = strtol(argv[0], &end, 10), *end != '\0') || + if ((argc != 3 && (argc != 4 || strcmp(argv[3], "--finish") != 0)) || + (request = strtol(argv[0], &end, 10), *end != '\0') || (request != 0 && request != 1) || parse_hex(argv[1], previous_id, sizeof(previous_id)) != 0) { fprintf(stderr, "usage: nvx-time capture 0|1 GENERATION_ID " - "ENTROPY_FILE\n"); + "ENTROPY_FILE [--finish]\n"); return 2; } if (state_load(&(struct time_state){0}) != 0) { @@ -3430,6 +3726,10 @@ static int cmd_capture(int argc, char **argv) fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); return 1; } + // The daemon keeps its PID in the restored guest; looking it up now keeps + // the file reads off the restore path. + if (argc == 4) + daemon = daemon_pid(); lock = lock_portb(detail, sizeof(detail)); if (lock < 0) { fprintf(stderr, "nvx-time: %s\n", detail); @@ -3443,35 +3743,40 @@ static int cmd_capture(int argc, char **argv) puts("0"); return 0; } - result = repair_restore(previous_id, argv[2], start); + result = repair_restore(previous_id, argv[2], start, daemon); close(lock); return result; } -// Restore steps 11 to 13: checks, the acknowledgement when the packet asked -// for one, then the daemon finishes the restore off the latency path. +// Restore steps 12 and 13 after the shell's activation and identity work: the +// acknowledgement when the packet asked for one, then the daemon runs the +// restore checks (step 11, after the acknowledgement) and finishes the +// restore off the latency path. static int cmd_restore_finish(int argc, char **argv) { int64_t start = clock_ns(CLOCK_MONOTONIC); bool new_cpus[MAX_CPUS] = {false}; + const char *new_list = "none"; unsigned long long generation; long long elapsed; struct checks checks; - char text[128]; + char text[256]; int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; bool ack = false; int new_count = 0; - pid_t pid; + int signaled; for (int index = 0; index < argc; index++) { - if (strcmp(argv[index], "--ack") == 0) + if (strcmp(argv[index], "--ack") == 0) { ack = true; - else if (strcmp(argv[index], "--new-cpus") == 0 && index + 1 < argc) - new_count = parse_new_cpus(argv[++index], new_cpus); - else + } else if (strcmp(argv[index], "--new-cpus") == 0 && index + 1 < argc) { + new_list = argv[++index]; + new_count = parse_new_cpus(new_list, new_cpus); + } else { new_count = -1; + } } - if (new_count < 0) { + if (new_count < 0 || strlen(new_list) > 127) { fprintf(stderr, "usage: nvx-time restore-finish [--ack] " "[--new-cpus LIST]\n"); return 2; @@ -3481,32 +3786,32 @@ static int cmd_restore_finish(int argc, char **argv) sscanf(text, "generation=%llu elapsed_us=%lld", &generation, &elapsed) != 2) return repair_failed("G_REPAIR_PACKET", "no restore record"); - if (checks_init(&checks, PHASE_RESTORE) != 0) - check_failed(&checks, "C6", "cannot read the CPU sets"); - else - run_restore_checks(&checks, new_cpus, new_count); elapsed += (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC; - snprintf(text, sizeof(text), "generation=%llu\nelapsed_us=%lld\n", - generation, elapsed); + snprintf(text, sizeof(text), "generation=%llu\nelapsed_us=%lld\nnew_cpus=%s\n", + generation, elapsed, new_count > 0 ? new_list : "none"); (void)replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text); - if (ack) { - if (enable_ports() != 0) { - fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); - return 1; - } - outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); + if (ack && enable_ports() != 0) { + fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); + return 1; } - pid = daemon_pid(); - if (pid > 0 && kill(pid, SIGUSR1) == 0) + signaled = signal_restore(daemon_pid(), ack); + if (signaled == 0) return 0; - if (!g_report_only) { + if (checks_init(&checks, PHASE_RESTORE) != 0) { + check_failed(&checks, "C6", "cannot read the CPU sets"); + return 1; + } + if (signaled > 0 || !g_report_only) { check_failed(&checks, "C10", "the time daemon is not running"); return 1; } // Report-only runs may lack the daemon; finish the restore here. + if (ack) + outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); + run_restore_checks(&checks, new_cpus, new_count); (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); complete_restore(release_with_deadline(timeout_s), checks.tsc_hz, - checks.lapic_hz, elapsed); + checks.lapic_hz, elapsed, checks.failures); return 0; } @@ -3787,6 +4092,69 @@ static int test_state(int argc, char **argv) return 0; } +static int test_restore_record(int argc, char **argv) +{ + static char text[4096]; + unsigned long long generation; + long long elapsed_us; + bool cpus[MAX_CPUS]; + const char *separator = ""; + int count; + + if (argc != 1 || test_text(argv[0], text, sizeof(text)) != 0) + return 2; + count = parse_restore_record(text, &generation, &elapsed_us, cpus); + if (count < 0) { + printf("error\n"); + return 0; + } + printf("generation=%llu elapsed_us=%lld new_cpus=", generation, elapsed_us); + for (int cpu = 0; cpu < MAX_CPUS; cpu++) { + if (cpus[cpu]) { + printf("%s%d", separator, cpu); + separator = ","; + } + } + printf("%s count=%d\n", count == 0 ? "none" : "", count); + return 0; +} + +// A side job and a foreground that fail a known number of times, so the test +// sees whether check_cpus_parallel() merges every job's failures. +static void test_failing_job(struct checks *checks, void *context) +{ + const int *failures = context; + + for (int index = 0; index < *failures; index++) + check_failed(checks, "C4", "side job failure %d", index + 1); +} + +static void test_failing_foreground(struct checks *checks, void *context) +{ + (void)context; + check_failed(checks, "C9", "foreground failure"); +} + +static int test_parallel(int argc, char **argv) +{ + static int failures[3] = {1, 2, 3}; + struct side_job sides[ARRAY_SIZE(failures)]; + struct checks checks; + bool cpus[MAX_CPUS]; + + if (argc != 1 || parse_cpu_list(argv[0], cpus) <= 0) + return 2; + memset(&checks, 0, sizeof(checks)); + checks.phase = PHASE_BOOT; + for (size_t index = 0; index < ARRAY_SIZE(sides); index++) + sides[index] = (struct side_job){.run = test_failing_job, + .context = &failures[index]}; + check_cpus_parallel(&checks, cpus, false, false, sides, ARRAY_SIZE(sides), + test_failing_foreground, NULL); + printf("failures=%d\n", checks.failures); + return 0; +} + static int cmd_test(int argc, char **argv) { const char *name = argc > 0 ? argv[0] : ""; @@ -3882,6 +4250,20 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "state") == 0) return test_state(argc, argv); + if (strcmp(name, "restore-record") == 0) + return test_restore_record(argc, argv); + if (strcmp(name, "parallel") == 0) + return test_parallel(argc, argv); + if (strcmp(name, "idle-priority") == 0 && argc == 0) { + long idle; + + set_idle_priority(0, true); + idle = syscall(SYS_sched_getscheduler, 0); + set_idle_priority(0, false); + printf("idle=%ld normal=%ld\n", idle, + syscall(SYS_sched_getscheduler, 0)); + return 0; + } return test_checks(name, argc, argv); } diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 6edd6e657..37ac7356a 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -7,6 +7,7 @@ from __future__ import annotations +import os import re import shutil import struct @@ -211,6 +212,19 @@ def test_musl_build_matches_the_reference_build(self): self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.stdout, self.run_test(*arguments[1:])) + # musl has no sched_setscheduler(); the daemon must still reach + # SCHED_IDLE after a restore. Returning to SCHED_OTHER needs + # CAP_SYS_NICE, which the guest has and a container may not. + result = subprocess.run( + [str(binary), "test", "idle-priority"], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(result.stdout.startswith("idle=5 normal="), result.stdout) + self.assertTrue(self.run_test("idle-priority").startswith("idle=5 normal=")) def test_packet_memory_ranges_follow_the_header(self): packet = encode_packet( @@ -711,6 +725,55 @@ def test_state_file_round_trip(self): "error", ) + def restore_record(self, text: str) -> str: + return self.run_test( + "restore-record", self.fixture("restore.txt", text) + ).strip() + + def test_restore_record_names_the_cpus_that_restore_finish_onlined(self): + # The capture helper's record, which the daemon reads when the helper + # finished an untiered restore itself. + self.assertEqual( + self.restore_record("generation=3\nelapsed_us=1234\n"), + "generation=3 elapsed_us=1234 new_cpus=none count=0", + ) + # restore-finish adds the CPUs that activation brought online. + self.assertEqual( + self.restore_record("generation=3\nelapsed_us=1400\nnew_cpus=4-6,9\n"), + "generation=3 elapsed_us=1400 new_cpus=4,5,6,9 count=4", + ) + self.assertEqual( + self.restore_record("generation=3\nelapsed_us=1400\nnew_cpus=none\n"), + "generation=3 elapsed_us=1400 new_cpus=none count=0", + ) + for text in ( + "", + "generation=3\n", + "elapsed_us=1400\ngeneration=3\n", + "generation=3\nelapsed_us=1400\nnew_cpus=7-4\n", + "generation=3\nelapsed_us=1400\nnew_cpus=64\n", + ): + self.assertEqual(self.restore_record(text), "error", text) + + def test_parallel_checks_merge_every_failure(self): + if sys.platform != "linux": + self.skipTest("CPU affinity is Linux-only") + allowed: list[int] = sorted(os.sched_getaffinity(0)) + # Three side jobs fail 1, 2, and 3 times and the foreground once, + # whether one CPU runs them in the caller or each has a thread. + for size in sorted({1, min(2, len(allowed)), min(4, len(allowed))}): + output = self.run_test("parallel", ",".join(map(str, allowed[:size]))) + self.assertTrue(output.endswith("failures=7\n"), output) + self.assertEqual(output.count("check=C4"), 6, output) + self.assertEqual(output.count("check=C9"), 1, output) + # A CPU the checks cannot run on fails C1 in its job. + unusable = [cpu for cpu in range(64) if cpu not in allowed] + if not unusable: + self.skipTest("every CPU is usable") + output = self.run_test("parallel", f"{allowed[0]},{unusable[0]}") + self.assertTrue(output.endswith("failures=8\n"), output) + self.assertIn(f"cannot run on cpu {unusable[0]}", output) + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index aac9a22fa..c3afb19bd 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -1422,6 +1422,13 @@ def test_snapshot_restore_runs_the_time_abi_steps_through_nvx_time(self): '/sbin/nvx-time capture "$capture_request" "$generation_id" "$entropy"', snapshot, ) + # An untiered restore with nothing to activate finishes inside the + # capture helper (metadata 2), so no second process starts. + self.assertIn( + '[ "$snapshot_tier" = legacy ] && finish_option=--finish', snapshot + ) + self.assertIn('${finish_option:+"$finish_option"}', snapshot) + self.assertIn("2) stall_detectors_suppressed=false ;;", snapshot) self.assertIn( 'generation_id=$(/sbin/nvx-reseed "$entropy" "$generation_id")', snapshot, @@ -1591,6 +1598,52 @@ def restore(flags: int) -> tuple[str, str, str]: ) self.assertEqual(calls, "restore-finish --new-cpus none\n") + def test_capture_metadata_2_leaves_no_restore_work_to_the_shell(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + snapshot = ( + Path(__file__).parents[1] / "guest" / "common" / "nvx-snapshot" + ).read_text(encoding="utf-8") + start = snapshot.index('case "${1:-}" in\n 0) ;;') + dispatch = snapshot[start : snapshot.index("\nesac\n", start) + 6] + generation_id = "0123456789abcdef0123456789abcdef" + + def dispatch_metadata(metadata: str) -> tuple[int, str]: + result = subprocess.run( + [shell, "-s"], + input=( + "set -eu\n" + "stall_detectors_suppressed=true\n" + 'post_restore() { echo "post_restore $*"; }\n' + 'fail_closed() { echo "fail_closed $*"; exit 1; }\n' + f"set -- {metadata}\n" + f"{dispatch}" + 'echo "suppressed=$stall_detectors_suppressed"\n' + ), + text=True, + capture_output=True, + timeout=5, + check=False, + ) + return result.returncode, result.stdout + + # No restore: the cleanup trap restores the stall detectors. + self.assertEqual(dispatch_metadata("0"), (0, "suppressed=true\n")) + # nvx-time finished the restore and handed it to the daemon. + self.assertEqual( + dispatch_metadata(f"2 4 0 0 {generation_id}"), + (0, "suppressed=false\n"), + ) + self.assertEqual( + dispatch_metadata(f"1 4 2 0 {generation_id}"), + (0, f"post_restore 4 2 0 {generation_id}\nsuppressed=false\n"), + ) + self.assertEqual( + dispatch_metadata("3"), + (1, "fail_closed snapshot capture metadata is invalid\n"), + ) + def test_console_log_persists_buffered_and_completed_output(self): with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) From 414371de45be1629dc9dafd3103ced68aaddce8e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 15:59:49 -0700 Subject: [PATCH 100/188] guest: keep the time ABI quiet on the console Every console byte is a port exit, and after a restore every first touch of guest memory faults, so the time ABI's console lines and file writes cost milliseconds on the boot and restore paths. Following the spec's guest obligations: - Production paths print no time ABI marker or status line. Checks record their result in the state file (check_phase, check_status, check_cpus, check_elapsed_us, tsc_hz, lapic_hz), a failed check prints only its violation event, and the discipline records G_SAMPLE_UNCERTAIN without a console line. - nvx-time status waits at most 30 s for pending checks, then prints the last check's marker and the runtime line. - Before shell-ready, nvx-time boot only takes the initial time sample and steps the clock (C12). A child runs the other boot checks at SCHED_IDLE, fail-fast, then becomes the daemon at normal priority. pre-capture waits for pending boot checks, which start the daemon that C10 requires. - After the acknowledgement, the daemon forks a SCHED_IDLE child for the deferred restore work: the checks, the grace-period release, the saved watchdog settings, the deferred C7, and the record. The watcher and the discipline keep normal priority, and the daemon releases the stall suppression itself at the stall timeout. - The restore helper no longer writes a restore record; restore-finish writes one for the daemon after shell work. The entropy file is created empty before the capture, so the restored helper only writes its bytes: creating files right after a restore took 4 to 9 ms on WHP. prometheus32 (KVM) against the full release bundle, 512 MiB, 30 interleaved restores per cell, p50 at 1/2/4/8 vCPUs: restore -2.2/-1.5/-3.2/-2.9 ms, cold boot -4/-7/-3/+1 ms. Against the release guest under the same VMM and kernel: MSHV (prometheus30) -1.0/-5.4/-3.6/ -6.6 ms and WHP (prometheus28) +1.7/+1.4/-8.3/-10.2 ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 7 +- doc/guests/ubuntu-guest.md | 5 +- guest/common/init | 8 +- guest/common/nvx-snapshot | 7 +- guest/common/nvx-time.c | 811 +++++++++++++++++++++++-------------- scripts/test_guest_time.py | 86 +++- 6 files changed, 587 insertions(+), 337 deletions(-) diff --git a/doc/build.md b/doc/build.md index 4f61a1c8b..0192bacd5 100644 --- a/doc/build.md +++ b/doc/build.md @@ -148,7 +148,12 @@ hypervisor identity, and runs the cross-vCPU warp test that the skew bound. `nvx-time` implements the time ABI's [guest obligations](design/time-abi.md#guest-obligations): the conformance checks, the violation watcher, the restore packet v4 and time-sample parsers, -the snapshot agent's time steps, and the wall-clock discipline. Its boot check +the snapshot agent's time steps, and the wall-clock discipline. Before +shell-ready it only steps the clock to host UTC; the other boot checks run in +the background, and the production console shows no time ABI line except a +violation event. `nvx-time status` prints the last check's marker and the +runtime state on demand, after waiting for pending checks, for CI and test +harnesses. Its boot check also fails when the kernel's boot-time W+X audit reports a writable and executable mapping. `nvx-time exhaustive` runs the CI-only exhaustive check (checks X1 to X6 on every online CPU); it reports and exits, and production diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index c81c88a6d..c01d32a81 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -437,8 +437,9 @@ The Ubuntu initramfs boot remains identical to Alpine at the machine level: 3. OpenVMM prepends its xe9/hvc console parameters and fixed virtio-mmio discovery. 4. `/init` mounts procfs, sysfs, and devtmpfs, switches kernel module loading - off, mounts `/run`, and runs the time ABI boot check (`nvx-time boot`), - which also starts the time daemon, before it mounts tmpfs. + off, mounts `/run`, and runs the time ABI boot step (`nvx-time boot`), + which steps the clock to host UTC and leaves the other boot checks and the + time daemon running in the background, before it mounts tmpfs. 5. `/init` configures loopback, optional static networking, and optional HostFs. 6. `/init` handles an explicit `nvx_exec` workload or opens a root shell. diff --git a/guest/common/init b/guest/common/init index d8e85a626..0a0977a22 100644 --- a/guest/common/init +++ b/guest/common/init @@ -15,9 +15,11 @@ if [ -e /proc/sys/kernel/modules_disabled ] && sleep 3600 done fi -# The time ABI boot check runs next, before any other guest work, in every -# guest mode. It powers the VM off itself on a violation and leaves its state -# and daemon under /run/nvx/time, so /run is mounted first and kept. +# The time ABI boot step runs next, before any other guest work, in every +# guest mode: the initial clock step only. The other boot checks then run in +# the background and start the time daemon; a failure powers the VM off, and +# the state and daemon live under /run/nvx/time, so /run is mounted first and +# kept. mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run if ! /sbin/nvx-time boot; then /sbin/nvx-exit 193 diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 57d756abb..880465663 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -287,7 +287,7 @@ esac export NVX_VM_GENERATION_ID="$generation_id" entropy=$runtime/restore-entropy -# Restore steps 9 to 13 of the time ABI (doc/design/time-abi.md). nvx-time +# Restore steps 9 to 12 of the time ABI (doc/design/time-abi.md). nvx-time # capture has already read restore packet v4, checked its generation, set # the wall clock, and written the entropy. Its metadata is FLAGS, # ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by START @@ -391,7 +391,7 @@ post_restore() { finish_restore "$packet_flags" } -# Restore steps 12 and 13: the acknowledgement when the host gated its input, +# Restore steps 11 and 12: the acknowledgement when the host gated its input, # then the daemon's restore checks and grace-period release, which run off # the restore latency path. A second argument of "quiet" omits the stage and # completion lines of a plain untiered restore. @@ -441,7 +441,8 @@ elif [ "$scratch_policy" = fresh ]; then fi # Capture steps 1 to 3: the capture checks and the stall-detector -# suppression. They never wait. +# suppression. They never wait, though nvx-time first lets pending boot +# checks finish: they start the daemon that the capture checks require. /sbin/nvx-time pre-capture || exit 1 stall_detectors_suppressed=true diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 43c032137..23b06c7de 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -5,19 +5,25 @@ // // doc/design/time-abi.md ("Guest obligations") is the contract this helper // enforces. It runs the conformance checks, the violation watcher and the -// wall-clock discipline (one daemon), and the snapshot agent's time steps: +// wall-clock discipline (one daemon), and the snapshot agent's time steps. It +// prints nothing on success: checks record their result in the state file, +// and only violation events reach the console. // -// boot boot checks, initial clock step, daemon start +// boot initial clock step (C12), then the other boot +// checks asynchronously, which start the daemon // check --phase PHASE [--new-cpus LIST] // capture or restore checks, for diagnosis // pre-capture capture checks and watchdog suppression -// capture REQUEST GENERATION_ID ENTROPY_FILE +// capture REQUEST GENERATION_ID ENTROPY_FILE [--finish] // capture request, then restore repair // cancel-capture undo pre-capture after a rejected capture // restore-finish [--ack] [--new-cpus LIST] -// restore checks, acknowledgement, release +// acknowledgement, then the daemon's restore +// checks and release +// status print the last check and the runtime state // sample print one host time sample // generation-id print the VM generation ID +// exhaustive the CI-only exhaustive check // test NAME ARGUMENTS... pure decoders and checks, for unit tests // // --report-only, or nvx_time_abi=report-only on the kernel command line, @@ -116,6 +122,9 @@ #define SLOW_TIME_CONSTANT 6 #define TIMER_LIST_WAIT_NS (200 * NSEC_PER_MSEC) #define DEFAULT_STALL_TIMEOUT_S 21 +// nvx-time status and pre-capture wait at most 30 s for pending checks. +#define STATUS_WAIT_NS (30 * NSEC_PER_SEC) +#define STATUS_POLL_NS (10 * NSEC_PER_MSEC) #define EVENT_MAX 512 #define DETAIL_MAX 384 #define TEXT_MAX 65536 @@ -125,9 +134,10 @@ #define STATE_PATH TIME_DIR "/state" #define STATE_TEMP_PATH TIME_DIR "/.state" #define STATE_LOCK_PATH TIME_DIR "/state.lock" -#define RATES_PATH TIME_DIR "/rates" +#define REPORT_PATH TIME_DIR "/report" #define DAEMON_PID_PATH TIME_DIR "/daemon.pid" #define SUPPRESSION_PATH TIME_DIR "/suppression" +#define SUPPRESSION_LOCK_PATH TIME_DIR "/suppression.lock" #define RESTORE_PATH TIME_DIR "/restore" #define PORTB_LOCK_PATH RUN_DIR "/portb.lock" #define RCU_SYNC_PATH RUN_DIR "/rcu-sync" @@ -639,6 +649,12 @@ static void fail_fatal(int status, const char *code, const char *source, struct time_state { uint64_t generation; + char check_phase[16]; + char check_status[16]; + uint64_t check_cpus; + int64_t check_elapsed_us; + uint64_t tsc_hz; + uint64_t lapic_hz; uint64_t discontinuities; char last_discontinuity[16]; int64_t last_step_ns; @@ -655,9 +671,13 @@ struct time_state { uint64_t violations; }; +// A fresh state is published at the start of the boot checks, which are then +// pending. static void state_init(struct time_state *state) { memset(state, 0, sizeof(*state)); + snprintf(state->check_phase, sizeof(state->check_phase), "boot"); + snprintf(state->check_status, sizeof(state->check_status), "pending"); snprintf(state->last_discontinuity, sizeof(state->last_discontinuity), "none"); snprintf(state->last_downtime_source, sizeof(state->last_downtime_source), @@ -681,6 +701,10 @@ static int state_parse(char *text, struct time_state *state) int64_t *signed_value; } const numbers[] = { {"generation", &state->generation, NULL}, + {"check_cpus", &state->check_cpus, NULL}, + {"check_elapsed_us", NULL, &state->check_elapsed_us}, + {"tsc_hz", &state->tsc_hz, NULL}, + {"lapic_hz", &state->lapic_hz, NULL}, {"discontinuities", &state->discontinuities, NULL}, {"last_step_ns", NULL, &state->last_step_ns}, {"last_step_realtime_ns", NULL, &state->last_step_realtime_ns}, @@ -698,6 +722,8 @@ static int state_parse(char *text, struct time_state *state) char *value; size_t size; } const words[] = { + {"check_phase", state->check_phase, sizeof(state->check_phase)}, + {"check_status", state->check_status, sizeof(state->check_status)}, {"last_discontinuity", state->last_discontinuity, sizeof(state->last_discontinuity)}, {"last_downtime_source", state->last_downtime_source, @@ -743,6 +769,12 @@ static void state_format(const struct time_state *state, char *text, snprintf(text, size, "version=1\n" "generation=%" PRIu64 "\n" + "check_phase=%s\n" + "check_status=%s\n" + "check_cpus=%" PRIu64 "\n" + "check_elapsed_us=%" PRId64 "\n" + "tsc_hz=%" PRIu64 "\n" + "lapic_hz=%" PRIu64 "\n" "discontinuities=%" PRIu64 "\n" "last_discontinuity=%s\n" "last_step_ns=%" PRId64 "\n" @@ -757,7 +789,9 @@ static void state_format(const struct time_state *state, char *text, "rejected_samples=%" PRIu64 "\n" "last_sample_error=%s\n" "violations=%" PRIu64 "\n", - state->generation, state->discontinuities, + state->generation, state->check_phase, state->check_status, + state->check_cpus, state->check_elapsed_us, state->tsc_hz, + state->lapic_hz, state->discontinuities, state->last_discontinuity, state->last_step_ns, state->last_step_realtime_ns, state->last_downtime_ns, state->last_downtime_source, state->synchronized, @@ -822,27 +856,61 @@ static int64_t frequency_to_ppb(int64_t frequency) return (scaled + (scaled >= 0 ? 32768 : -32768)) / 65536; } +// F and L are published in the state file by the boot check. static int rates_load(uint64_t *tsc_hz, uint64_t *lapic_hz) { - char text[256]; - unsigned long long tsc; - unsigned long long lapic; + struct time_state state; - if (read_text(RATES_PATH, text, sizeof(text)) < 0 || - sscanf(text, "tsc_hz=%llu lapic_hz=%llu", &tsc, &lapic) != 2) + if (state_load(&state) != 0 || state.tsc_hz == 0 || state.lapic_hz == 0) return -1; - *tsc_hz = tsc; - *lapic_hz = lapic; + *tsc_hz = state.tsc_hz; + *lapic_hz = state.lapic_hz; return 0; } -static int rates_store(uint64_t tsc_hz, uint64_t lapic_hz) +struct check_record { + enum phase phase; + const char *status; + int cpus; + int64_t elapsed_us; + uint64_t tsc_hz; + uint64_t lapic_hz; +}; + +static void apply_check(struct time_state *state, const void *context) { - char text[128]; + const struct check_record *record = context; - snprintf(text, sizeof(text), "tsc_hz=%" PRIu64 "\nlapic_hz=%" PRIu64 "\n", - tsc_hz, lapic_hz); - return replace_file(RATES_PATH, RATES_PATH ".tmp", text); + snprintf(state->check_phase, sizeof(state->check_phase), "%s", + k_phase_names[record->phase]); + snprintf(state->check_status, sizeof(state->check_status), "%s", + record->status); + state->check_cpus = (uint64_t)record->cpus; + state->check_elapsed_us = record->elapsed_us; + if (record->tsc_hz != 0) + state->tsc_hz = record->tsc_hz; + if (record->lapic_hz != 0) + state->lapic_hz = record->lapic_hz; +} + +// Records a conformance check in the state file: "pending" while it runs, +// then "ok" (a failed check powers the guest off instead). Rates of 0 keep +// the published ones. In report-only mode the failure count goes beside the +// state, for nvx-time status. +static int record_check(enum phase phase, const char *status, int cpus, + int64_t elapsed_us, uint64_t tsc_hz, uint64_t lapic_hz, + int failures) +{ + struct check_record record = {phase, status, cpus, + elapsed_us, tsc_hz, lapic_hz}; + + if (g_report_only) { + char text[32]; + + snprintf(text, sizeof(text), "failures=%d\n", failures); + (void)replace_file(REPORT_PATH, REPORT_PATH ".tmp", text); + } + return state_apply(apply_check, &record); } // The watchdog settings saved before a capture, in the order they are @@ -888,17 +956,30 @@ static int suppression_save(void) return 0; } -// Writes back every value saved by suppression_save, then forgets them. -static int suppression_restore(char *detail, size_t size) +// Writes back every value saved by suppression_save, then forgets them. A +// lock serializes the daemon's release at the stall timeout with the restore +// worker's; with MISSING_OK, values already written back count as restored. +static int suppression_restore(char *detail, size_t size, bool missing_ok) { char text[512]; char *line = text; + int lock = lock_file(SUPPRESSION_LOCK_PATH); + int result = -1; - if (read_text(SUPPRESSION_PATH, text, sizeof(text)) < 0) { - snprintf(detail, size, "read %s: %s", SUPPRESSION_PATH, + if (lock < 0) { + snprintf(detail, size, "lock %s: %s", SUPPRESSION_LOCK_PATH, strerror(errno)); return -1; } + if (read_text(SUPPRESSION_PATH, text, sizeof(text)) < 0) { + if (missing_ok && errno == ENOENT) + result = 0; + else + snprintf(detail, size, "read %s: %s", SUPPRESSION_PATH, + strerror(errno)); + close(lock); + return result; + } while (*line != '\0') { char *next = strchr(line, '\n'); char *value = strchr(line, '='); @@ -911,16 +992,18 @@ static int suppression_restore(char *detail, size_t size) if (write_text(line, value) != 0) { snprintf(detail, size, "restore %.200s: %s", line, strerror(errno)); + close(lock); return -1; } line = next != NULL ? next : line + strlen(line); } - if (unlink(SUPPRESSION_PATH) != 0) { + if (unlink(SUPPRESSION_PATH) != 0) snprintf(detail, size, "remove %s: %s", SUPPRESSION_PATH, strerror(errno)); - return -1; - } - return 0; + else + result = 0; + close(lock); + return result; } // --------------------------------------------------------------------------- @@ -1450,33 +1533,26 @@ struct checks { uint64_t lapic_hz; }; -static bool check_failed_code(struct checks *checks, const char *id, - const char *code, const char *format, ...) - __attribute__((format(printf, 4, 5))); +static bool check_failed_code(struct checks *checks, const char *code, + const char *format, ...) + __attribute__((format(printf, 3, 4))); // Serializes failure reports from the boot check's per-CPU threads, which // share the console and the violation count. static pthread_mutex_t g_report_lock = PTHREAD_MUTEX_INITIALIZER; -// Prints the failure line, emits CODE, and powers off with status 193. In -// report-only mode it returns false and the run continues. -static bool check_failed_code(struct checks *checks, const char *id, - const char *code, const char *format, ...) +// Emits CODE and powers off with status 193; a failed check prints no +// marker. In report-only mode it returns false and the run continues. +static bool check_failed_code(struct checks *checks, const char *code, + const char *format, ...) { char detail[DETAIL_MAX]; - char escaped[DETAIL_MAX]; - char line[EVENT_MAX + 64]; va_list arguments; va_start(arguments, format); vsnprintf(detail, sizeof(detail), format, arguments); va_end(arguments); - escape_text(detail, escaped, sizeof(escaped) - 1); - snprintf(line, sizeof(line), - "%s: v=1 phase=%s status=fail check=%s detail=\"%s\"\n", - abi_prefix(), k_phase_names[checks->phase], id, escaped); pthread_mutex_lock(&g_report_lock); - console_write(line); checks->failures++; emit_event(code, "conformance", checks->phase, detail); if (!g_report_only) @@ -1501,7 +1577,7 @@ static bool check_failed(struct checks *checks, const char *id, vsnprintf(detail, sizeof(detail), format, arguments); va_end(arguments); snprintf(code, sizeof(code), "G_CONFORMANCE_%s", id); - return check_failed_code(checks, id, code, "%s", detail); + return check_failed_code(checks, code, "%s", detail); } static int checks_init(struct checks *checks, enum phase phase) @@ -2395,10 +2471,11 @@ static pid_t daemon_pid(void) return (pid_t)pid; } -// The daemon finishes a restore (restore step 13 and the deferred step 11) -// at idle priority, so that on few vCPUs it never takes the CPU from the -// restored workload. musl does not implement sched_setscheduler(), and the -// daemon is single-threaded, so the per-thread system call applies to it. +// The asynchronous boot checks and restore step 12 run at idle priority, so +// that on few vCPUs they never take a CPU from the workload; the watcher and +// the discipline keep normal priority. musl does not implement +// sched_setscheduler(), so this uses the per-thread system call: PID 0 is +// the calling thread. static void set_idle_priority(pid_t pid, bool idle) { struct sched_param param = {0}; @@ -2407,17 +2484,16 @@ static void set_idle_priority(pid_t pid, bool idle) ¶m); } -// Restore steps 12 and 13 on the readiness path: the acknowledgement when the -// host gated its input, then hand the rest of the restore to DAEMON. Returns -// -1, without acknowledging, when no daemon runs; 1 when the daemon vanished -// after the acknowledgement. The caller has enabled the ports when ACK is set. +// Restore steps 11 and 12 on the readiness path: the acknowledgement when the +// host gated its input, then hand step 12 to DAEMON. Returns -1, without +// acknowledging, when no daemon runs; 1 when the daemon vanished after the +// acknowledgement. The caller has enabled the ports when ACK is set. static int signal_restore(pid_t daemon, bool ack) { if (daemon <= 0 || kill(daemon, 0) != 0) return -1; if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); - set_idle_priority(daemon, true); return kill(daemon, SIGUSR1) == 0 ? 0 : 1; } @@ -2785,14 +2861,9 @@ struct daemon { int timer; int polls; int64_t last_accepted_ns; - uint64_t tsc_hz; - uint64_t lapic_hz; - pid_t release_pid; - bool release_done; - int release_status; - int64_t release_deadline_ns; - int64_t restore_elapsed_us; - int restore_failures; + pid_t worker_pid; + int64_t worker_deadline_ns; + bool suppression_released; }; static void watcher_consume(void *context, const char *message) @@ -2908,24 +2979,14 @@ static void discipline_poll(struct daemon *daemon) (void)state_apply(apply_poll, &outcome); if (lock >= 0) close(lock); - if (!outcome.accepted) { - char epsilon[24] = "none"; - char line[192]; - - if (result == 1) - snprintf(epsilon, sizeof(epsilon), "%" PRId64, outcome.epsilon); - snprintf(line, sizeof(line), - "%s: v=1 phase=runtime status=uncertain " - "code=G_SAMPLE_UNCERTAIN epsilon_ns=%s\n", - abi_prefix(), epsilon); - console_write(line); - } } // Waits for a full normal RCU grace period: with rcu_normal set, the // synchronize_rcu_expedited() in every unmount waits for one that starts -// after the call, on any number of CPUs. Runs in a child of the daemon. -static void release_child(void) +// after the call, on any number of CPUs. The private mount namespace makes the +// caller a process of its own: the restore worker, or a child. Returns 0 when +// the release completed. +static int grace_period_release(void) { int64_t normal = 0; bool saved = read_int64(RCU_NORMAL, &normal) == 0 && @@ -2943,7 +3004,7 @@ static void release_child(void) } if (saved && write_int64(RCU_NORMAL, normal) != 0) status = 1; - _exit(status); + return status; } // Runs the grace-period release in a child and waits at most TIMEOUT_S @@ -2964,7 +3025,7 @@ static bool release_with_deadline(int64_t timeout_s) pid = fork(); if (pid == 0) { sigprocmask(SIG_SETMASK, &previous, NULL); - release_child(); + _exit(grace_period_release()); } while (pid > 0) { int status; @@ -2989,23 +3050,6 @@ static bool release_with_deadline(int64_t timeout_s) return released; } -static void print_restore_marker(int failures, int cpus, uint64_t tsc_hz, - uint64_t lapic_hz, int64_t elapsed_us) -{ - char suffix[32] = ""; - char line[256]; - - if (g_report_only) - snprintf(suffix, sizeof(suffix), " failures=%d", failures); - snprintf(line, sizeof(line), - "%s: v=1 phase=restore status=%s cpus=%d tsc_hz=%" PRIu64 - " lapic_hz=%" PRIu64 " generation=%" PRIu32 - " elapsed_us=%" PRId64 "%s\n", - abi_prefix(), failures == 0 ? "ok" : "fail", cpus, tsc_hz, - lapic_hz, g_generation, elapsed_us, suffix); - console_write(line); -} - static void run_restore_checks(struct checks *checks, const bool *new_cpus, int new_count); @@ -3018,89 +3062,125 @@ static int parse_new_cpus(const char *text, bool *cpus) return parse_cpu_list(text, cpus); } -// Parses the restore record: the generation and the repair time that the -// capture helper wrote, and the CPUs that restore-finish onlined, when it ran. +// Parses the restore record that restore-finish writes for the daemon: the +// generation it finished and the CPUs that activation brought online. // Returns the number of those CPUs, or -1 when the record is malformed. static int parse_restore_record(const char *text, unsigned long long *generation, - long long *elapsed_us, bool *new_cpus) + bool *new_cpus) { char cpus[128] = "none"; - if (sscanf(text, "generation=%llu elapsed_us=%lld new_cpus=%127s", - generation, elapsed_us, cpus) < 2) + if (sscanf(text, "generation=%llu new_cpus=%127s", generation, cpus) < 1) return -1; return parse_new_cpus(cpus, new_cpus); } -// Step 13 after the grace period ended (or the stall timeout passed): put -// the watchdog settings back, run the deferred C7, print the restore marker, -// and restart the discipline at the fast cadence. -static void complete_restore(bool released, uint64_t tsc_hz, uint64_t lapic_hz, - int64_t elapsed_us, int failures) +// Restore step 12, fail-fast: the restore checks, the grace-period release, +// the saved watchdog settings, the deferred C7, and the restore check's +// record. A child of the daemon runs it at SCHED_IDLE, and the daemon +// releases the suppression itself at the stall timeout; restore-finish runs it +// inline, with its own deadline (WITH_DEADLINE), in report-only mode when no +// daemon runs. Returns 0, or 1 when a report-only release failed. +static int restore_work(const bool *new_cpus, int new_count, bool with_deadline) { - struct checks checks; + int64_t start = clock_ns(CLOCK_MONOTONIC); + int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; char detail[DETAIL_MAX]; + struct checks checks; + int64_t elapsed; + bool released; + if (checks_init(&checks, PHASE_RESTORE) != 0) + check_failed(&checks, "C6", "cannot read the CPU sets"); + else + run_restore_checks(&checks, new_cpus, new_count); + elapsed = clock_ns(CLOCK_MONOTONIC) - start; + if (with_deadline) { + (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); + released = release_with_deadline(timeout_s); + } else { + released = grace_period_release() == 0; + } if (!released) fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", PHASE_RESTORE, "the RCU grace-period release failed"); - if (suppression_restore(detail, sizeof(detail)) != 0) + if (suppression_restore(detail, sizeof(detail), true) != 0) fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", PHASE_RESTORE, "%s", detail); - if (checks_init(&checks, PHASE_RESTORE) != 0) - check_failed(&checks, "C7", "cannot read the online CPU set"); - else - check_timer_list(&checks, TIMER_LIST_WAIT_NS); - print_restore_marker(failures + checks.failures, checks.online_count, - tsc_hz, lapic_hz, elapsed_us); - set_idle_priority(0, false); + start = clock_ns(CLOCK_MONOTONIC); + check_timer_list(&checks, TIMER_LIST_WAIT_NS); + elapsed += clock_ns(CLOCK_MONOTONIC) - start; + if (record_check(PHASE_RESTORE, "ok", checks.online_count, + elapsed / NSEC_PER_USEC, 0, 0, checks.failures) != 0) + check_failed(&checks, "C10", "cannot record the restore check: %s", + strerror(errno)); + return released ? 0 : 1; } -// Restore step 13, and step 11 after the acknowledgement: the restore checks -// fail fast as before, then the grace-period release runs in a child. +// Starts restore step 12 after the acknowledgement, in a child at +// SCHED_IDLE, so the watcher and the discipline keep normal priority; the +// fork itself also runs at SCHED_IDLE. static void start_restore(struct daemon *daemon) { - int64_t start = clock_ns(CLOCK_MONOTONIC); - bool new_cpus[MAX_CPUS]; - struct checks checks; - char text[256]; + bool new_cpus[MAX_CPUS] = {false}; + struct time_state state; unsigned long long generation; - long long elapsed; int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; - int new_count; + char text[256]; + int new_count = 0; pid_t pid; - if (daemon->release_pid > 0) + if (daemon->worker_pid > 0) return; - set_idle_priority(0, true); - if (read_text(RESTORE_PATH, text, sizeof(text)) < 0 || - (new_count = parse_restore_record(text, &generation, &elapsed, - new_cpus)) < 0) { + if (state_load(&state) != 0) { fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", - PHASE_RESTORE, "the restore record is unreadable"); + PHASE_RESTORE, "the time state is unreadable"); return; } - g_generation = (uint32_t)generation; - if (checks_init(&checks, PHASE_RESTORE) != 0) - check_failed(&checks, "C6", "cannot read the CPU sets"); - else - run_restore_checks(&checks, new_cpus, new_count); - daemon->restore_failures = checks.failures; - daemon->restore_elapsed_us = - elapsed + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC; + g_generation = (uint32_t)state.generation; + if (read_text(RESTORE_PATH, text, sizeof(text)) >= 0) { + new_count = parse_restore_record(text, &generation, new_cpus); + if (new_count < 0) { + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "the restore record is malformed"); + return; + } + if (generation != state.generation) { + memset(new_cpus, 0, sizeof(new_cpus)); + new_count = 0; + } + } (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); + // The repair paired the clock with the packet's UTC. + daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); + set_idle_priority(0, true); pid = fork(); if (pid == 0) - release_child(); + _exit(restore_work(new_cpus, new_count, false)); + set_idle_priority(0, false); if (pid < 0) { - complete_restore(false, daemon->tsc_hz, daemon->lapic_hz, - daemon->restore_elapsed_us, daemon->restore_failures); + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "cannot start the restore work: %s", + strerror(errno)); return; } - daemon->release_pid = pid; - daemon->release_done = false; - daemon->release_deadline_ns = + daemon->worker_pid = pid; + daemon->worker_deadline_ns = clock_ns(CLOCK_MONOTONIC) + timeout_s * NSEC_PER_SEC; + daemon->suppression_released = false; +} + +// Releases the stall suppression for a worker whose grace period is still +// pending at the stall timeout: the grace period is stuck, and the kernel +// then reports the real stall. +static void release_suppression(struct daemon *daemon) +{ + char detail[DETAIL_MAX]; + + daemon->suppression_released = true; + if (suppression_restore(detail, sizeof(detail), true) != 0) + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "%s", detail); } static void reap_children(struct daemon *daemon) @@ -3109,10 +3189,20 @@ static void reap_children(struct daemon *daemon) pid_t pid; while ((pid = waitpid(-1, &status, WNOHANG)) > 0) { - if (pid == daemon->release_pid) { - daemon->release_done = true; - daemon->release_status = status; + if (pid != daemon->worker_pid) + continue; + daemon->worker_pid = 0; + // A failing worker powered the guest off itself, or reported in + // report-only mode; a crash leaves the suppression to the daemon. + if (!WIFEXITED(status)) { + release_suppression(daemon); + fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", + PHASE_RESTORE, "the restore work ended with signal %d", + WTERMSIG(status)); } + // Step 12 ends by restarting the discipline at the fast cadence. + daemon->polls = 0; + (void)arm_timer(daemon); } } @@ -3162,11 +3252,12 @@ static void daemon_loop(struct daemon *daemon) struct pollfd fds[3] = {{daemon->kmsg, POLLIN, 0}, {daemon->signals, POLLIN, 0}, {daemon->timer, POLLIN, 0}}; + bool waiting = daemon->worker_pid > 0 && !daemon->suppression_released; int timeout = -1; - if (daemon->release_pid > 0 && !daemon->release_done) { + if (waiting) { int64_t remaining = - daemon->release_deadline_ns - clock_ns(CLOCK_MONOTONIC); + daemon->worker_deadline_ns - clock_ns(CLOCK_MONOTONIC); timeout = remaining <= 0 ? 0 : (int)((remaining + NSEC_PER_MSEC - 1) / @@ -3189,84 +3280,31 @@ static void daemon_loop(struct daemon *daemon) discipline_poll(daemon); arm_timer(daemon); } - if (daemon->release_pid > 0 && - (daemon->release_done || - clock_ns(CLOCK_MONOTONIC) >= daemon->release_deadline_ns)) { - bool released = !daemon->release_done || - (WIFEXITED(daemon->release_status) && - WEXITSTATUS(daemon->release_status) == 0); - - daemon->release_pid = 0; - complete_restore(released, daemon->tsc_hz, daemon->lapic_hz, - daemon->restore_elapsed_us, - daemon->restore_failures); - daemon->polls = 0; - daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); - arm_timer(daemon); - } + if (daemon->worker_pid > 0 && !daemon->suppression_released && + clock_ns(CLOCK_MONOTONIC) >= daemon->worker_deadline_ns) + release_suppression(daemon); } } -// Forks the daemon, which keeps reading the kernel log from where the boot -// check stopped, waits until the daemon reports its descriptors ready, and -// publishes its PID. -static int start_daemon(int kmsg, uint64_t tsc_hz, uint64_t lapic_hz, - int64_t last_accepted_ns) +// Turns the boot checker into the time daemon, at normal priority. It keeps +// reading the kernel log from where the boot check positioned KMSG and +// publishes its PID once its descriptors are ready. Returns 0 or -1. +static int become_daemon(struct daemon *daemon, int kmsg, + int64_t last_accepted_ns) { char text[32]; - int ready[2]; - int error = 0; - ssize_t count; - pid_t pid; + int error; - if (pipe2(ready, O_CLOEXEC) != 0) - return -1; - pid = fork(); - if (pid < 0) { - error = errno; - close(ready[0]); - close(ready[1]); + memset(daemon, 0, sizeof(*daemon)); + daemon->kmsg = kmsg; + daemon->last_accepted_ns = last_accepted_ns; + set_idle_priority(0, false); + error = daemon_setup(daemon); + if (error != 0) { errno = error; return -1; } - if (pid == 0) { - struct daemon daemon; - int null = open("/dev/null", O_RDWR | O_CLOEXEC); - - close(ready[0]); - memset(&daemon, 0, sizeof(daemon)); - daemon.kmsg = kmsg; - daemon.tsc_hz = tsc_hz; - daemon.lapic_hz = lapic_hz; - daemon.last_accepted_ns = last_accepted_ns; - g_async_output = true; - (void)setsid(); - if (null >= 0) { - (void)dup2(null, STDIN_FILENO); - (void)dup2(null, STDOUT_FILENO); - (void)dup2(null, STDERR_FILENO); - close(null); - } - error = chdir("/") != 0 ? errno : daemon_setup(&daemon); - (void)write_all(ready[1], &error, sizeof(error)); - close(ready[1]); - if (error != 0) - _exit(1); - daemon_loop(&daemon); - _exit(0); - } - close(ready[1]); - do - count = read(ready[0], &error, sizeof(error)); - while (count < 0 && errno == EINTR); - close(ready[0]); - if (count != (ssize_t)sizeof(error) || error != 0) { - (void)kill(pid, SIGKILL); - (void)waitpid(pid, NULL, 0); - errno = count == (ssize_t)sizeof(error) ? error : EIO; - return -1; - } - snprintf(text, sizeof(text), "%d\n", (int)pid); + snprintf(text, sizeof(text), "%d\n", (int)getpid()); return replace_file(DAEMON_PID_PATH, DAEMON_PID_PATH ".tmp", text); } @@ -3274,24 +3312,6 @@ static int start_daemon(int kmsg, uint64_t tsc_hz, uint64_t lapic_hz, // Commands // --------------------------------------------------------------------------- -static void print_marker(const struct checks *checks, int64_t start_ns) -{ - char line[256]; - char failures[32] = ""; - - if (g_report_only) - snprintf(failures, sizeof(failures), " failures=%d", checks->failures); - snprintf(line, sizeof(line), - "%s: v=1 phase=%s status=%s cpus=%d tsc_hz=%" PRIu64 - " lapic_hz=%" PRIu64 " generation=%" PRIu32 - " elapsed_us=%" PRId64 "%s\n", - abi_prefix(), k_phase_names[checks->phase], - checks->failures == 0 ? "ok" : "fail", checks->online_count, - checks->tsc_hz, checks->lapic_hz, g_generation, - (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC, failures); - console_write(line); -} - static void load_generation(void) { struct time_state state; @@ -3332,7 +3352,7 @@ static void boot_kernel_log(struct checks *checks, void *context) // K1 is kernel hardening, not time: a writable and executable kernel // mapping, such as unprotected ITS thunk pages. if (job->log->wx[0] != '\0') - check_failed_code(checks, "K1", "G_KERNEL_WX", "%s", job->log->wx); + check_failed_code(checks, "G_KERNEL_WX", "%s", job->log->wx); } // C7 beside the per-CPU checks. @@ -3355,37 +3375,50 @@ struct boot_sample { int64_t accepted_ns; }; -// The boot checks that need no CPU of their own, C12 included, run in the -// caller while the parallel jobs run. +// The boot checks that need no CPU of their own run in the caller while the +// parallel jobs run. static void boot_foreground(struct checks *checks, void *context) { - struct boot_sample *sample = context; - + (void)context; check_cmdline(checks); check_platform(checks); check_clocksource(checks); check_debug_watchdogs(checks); - if (check_initial_sample(checks, &sample->theta, &sample->epsilon)) - sample->accepted_ns = clock_ns(CLOCK_MONOTONIC); } -static int cmd_boot(void) +// The asynchronous boot checks: every boot check but C12, at SCHED_IDLE +// after shell-ready, and fail-fast. When they pass, the checker becomes the +// daemon and records the boot check, whose duration adds C12's (C12_NS). +// It never returns. +static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, + int c12_failures) { int64_t start = clock_ns(CLOCK_MONOTONIC); + int null = open("/dev/null", O_RDWR | O_CLOEXEC); struct boot_log_job log_job; struct side_job sides[3]; + struct daemon daemon; struct checks checks; struct boot_log log; - struct time_state state; - struct boot_sample sample = {0, 0, 0}; + bool started = false; int kmsg; - g_generation = 0; - make_runtime_directories(); + g_async_output = true; + (void)setsid(); + if (null >= 0) { + (void)dup2(null, STDIN_FILENO); + (void)dup2(null, STDOUT_FILENO); + (void)dup2(null, STDERR_FILENO); + close(null); + } + if (chdir("/") != 0) + _exit(1); + set_idle_priority(0, true); if (checks_init(&checks, PHASE_BOOT) != 0) { check_failed(&checks, "C1", "cannot read the CPU sets"); - return 1; + _exit(1); } + checks.failures = c12_failures; memset(&log, 0, sizeof(log)); log_job.log = &log; log_job.read_log = false; @@ -3406,7 +3439,7 @@ static int cmd_boot(void) sides[1] = (struct side_job){.run = boot_timer_list}; sides[2] = (struct side_job){.run = boot_cpuinfo}; check_cpus_parallel(&checks, checks.online, true, true, sides, - ARRAY_SIZE(sides), boot_foreground, &sample); + ARRAY_SIZE(sides), boot_foreground, NULL); free(log_job.buffer); memcpy(checks.cpu_khz, sides[2].checks.cpu_khz, sizeof(checks.cpu_khz)); memcpy(checks.cpu_khz_known, sides[2].checks.cpu_khz_known, @@ -3418,21 +3451,62 @@ static int cmd_boot(void) "cpu 0: floor(F / 1000) is %" PRIu64 " kHz but cpu MHz is %" PRIu64 " kHz", checks.tsc_hz / 1000, checks.cpu_khz[0]); + if (kmsg >= 0) { + started = become_daemon(&daemon, kmsg, sample->accepted_ns) == 0; + if (!started) + check_failed(&checks, "C10", "cannot start the time daemon: %s", + strerror(errno)); + } + if (record_check(PHASE_BOOT, "ok", checks.online_count, + (c12_ns + clock_ns(CLOCK_MONOTONIC) - start) / + NSEC_PER_USEC, + checks.tsc_hz, checks.lapic_hz, checks.failures) != 0) + check_failed(&checks, "C10", "cannot publish the time state: %s", + strerror(errno)); + if (!started) + _exit(1); + daemon_loop(&daemon); + _exit(0); +} + +// Before shell-ready, only C12: the initial time sample and clock step, then +// the published state. Every other boot check runs asynchronously in a child, +// which becomes the daemon. +static int cmd_boot(void) +{ + int64_t start = clock_ns(CLOCK_MONOTONIC); + struct boot_sample sample = {0, 0, 0}; + struct time_state state; + struct checks checks; + pid_t pid; + + g_generation = 0; + make_runtime_directories(); + memset(&checks, 0, sizeof(checks)); + checks.phase = PHASE_BOOT; + if (check_initial_sample(&checks, &sample.theta, &sample.epsilon)) + sample.accepted_ns = clock_ns(CLOCK_MONOTONIC); state_init(&state); state.synchronized = sample.accepted_ns != 0; state.samples = sample.accepted_ns != 0; state.offset_ns = sample.accepted_ns != 0 ? sample.theta : 0; state.uncertainty_ns = sample.accepted_ns != 0 ? sample.epsilon : 0; state.frequency_ppb = frequency_to_ppb(current_frequency()); - if (rates_store(checks.tsc_hz, checks.lapic_hz) != 0 || - state_store(&state) != 0) + if (state_store(&state) != 0) { check_failed(&checks, "C10", "cannot publish the time state: %s", strerror(errno)); - else if (kmsg >= 0 && start_daemon(kmsg, checks.tsc_hz, checks.lapic_hz, - sample.accepted_ns) != 0) - check_failed(&checks, "C10", "cannot start the time daemon: %s", + return 1; + } + fflush(NULL); + pid = fork(); + if (pid == 0) + boot_checker(&sample, clock_ns(CLOCK_MONOTONIC) - start, + checks.failures); + if (pid < 0) { + check_failed(&checks, "C10", "cannot start the boot checks: %s", strerror(errno)); - print_marker(&checks, start); + return 1; + } return 0; } @@ -3503,16 +3577,49 @@ static int cmd_check(int argc, char **argv) run_capture_checks(&checks); else run_restore_checks(&checks, new_cpus, new_count); - print_marker(&checks, start); + (void)record_check(phase, "ok", checks.online_count, + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, 0, + 0, checks.failures); return checks.failures == 0 ? 0 : 1; } +// Waits until no conformance check is pending, for at most STATUS_WAIT_NS, +// and loads the state into STATE. Returns 0, 1 when a check is still +// pending, or -1 when the state is unreadable. +static int wait_for_checks(struct time_state *state) +{ + int64_t deadline = clock_ns(CLOCK_MONOTONIC) + STATUS_WAIT_NS; + + for (;;) { + if (state_load(state) != 0) + return -1; + if (strcmp(state->check_status, "pending") != 0) + return 0; + if (clock_ns(CLOCK_MONOTONIC) >= deadline) + return 1; + (void)nanosleep(&(struct timespec){0, STATUS_POLL_NS}, NULL); + } +} + static int cmd_pre_capture(void) { - int64_t start = clock_ns(CLOCK_MONOTONIC); + struct time_state state; struct checks checks; - - load_generation(); + int64_t start; + int waited; + + // The asynchronous boot checks start the daemon that C10 requires. + waited = wait_for_checks(&state); + if (waited != 0) { + memset(&checks, 0, sizeof(checks)); + checks.phase = PHASE_CAPTURE; + check_failed(&checks, "C10", "%s", + waited > 0 ? "the boot checks are still pending" + : "the time state is unreadable"); + return 1; + } + start = clock_ns(CLOCK_MONOTONIC); + g_generation = (uint32_t)state.generation; if (checks_init(&checks, PHASE_CAPTURE) != 0) { check_failed(&checks, "C6", "cannot read the CPU sets"); return 1; @@ -3523,13 +3630,17 @@ static int cmd_pre_capture(void) int error = errno; if (error != EEXIST) - (void)suppression_restore(detail, sizeof(detail)); + (void)suppression_restore(detail, sizeof(detail), false); fprintf(stderr, "nvx-time: cannot suppress the stall detectors: %s\n", error == EEXIST ? "a capture is already pending" : strerror(error)); return 1; } - print_marker(&checks, start); + if (record_check(PHASE_CAPTURE, "ok", checks.online_count, + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, 0, 0, + checks.failures) != 0) + check_failed(&checks, "C10", "cannot record the capture check: %s", + strerror(errno)); return 0; } @@ -3537,7 +3648,7 @@ static int cmd_cancel_capture(void) { char detail[DETAIL_MAX]; - if (suppression_restore(detail, sizeof(detail)) != 0) { + if (suppression_restore(detail, sizeof(detail), false) != 0) { fprintf(stderr, "nvx-time: %s\n", detail); return 1; } @@ -3574,6 +3685,11 @@ static void apply_repair(struct time_state *state, const void *context) state->samples++; snprintf(state->last_sample_error, sizeof(state->last_sample_error), "none"); + // The daemon's step 12 runs the restore check after the acknowledgement. + snprintf(state->check_phase, sizeof(state->check_phase), "restore"); + snprintf(state->check_status, sizeof(state->check_status), "pending"); + state->check_cpus = 0; + state->check_elapsed_us = 0; } static int repair_failed(const char *code, const char *detail) @@ -3583,13 +3699,14 @@ static int repair_failed(const char *code, const char *detail) } // Restore steps 6 to 8: read the packet with four-byte reads, validate it, -// and set the wall clock from its bracketed UTC. The caller holds the portb -// lock. With DAEMON set, an untiered restore with no processors or memory to -// activate needs no shell work before the acknowledgement, so steps 12 and 13 -// run here too and no second helper process starts; the metadata then begins -// with 2 instead of 1. +// and set the wall clock from its bracketed UTC, then write the entropy to +// ENTROPY_FD, the caller's file at ENTROPY_PATH. The caller holds the portb +// lock. With FINISH (untiered) and a DAEMON, a restore with no processors or +// memory to activate needs no shell work before the acknowledgement, so +// steps 11 and 12 start here too and no second helper process starts; the +// metadata then begins with 2 instead of 1. static int repair_restore(const uint8_t *previous_id, const char *entropy_path, - int64_t start_ns, pid_t daemon) + int entropy_fd, bool finish, pid_t daemon) { static uint8_t body[PACKET_MAX_SIZE]; static struct restore_packet packet; @@ -3598,13 +3715,11 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, struct time_sample sample; char detail[DETAIL_MAX]; char id[GENERATION_ID_SIZE * 2 + 1]; - char text[128]; int64_t t0; int64_t t1; int64_t sample_theta; int64_t sample_epsilon; int finished = -1; - int fd; t0 = clock_ns(CLOCK_REALTIME); outb(SELECT_PACKET, PORTB_STATUS); @@ -3613,6 +3728,8 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, if (parse_packet_header(header, &packet, detail, sizeof(detail)) != 0) return repair_failed("G_REPAIR_PACKET", detail); if (packet.generation == g_generation) { + close(entropy_fd); + unlink(entropy_path); puts("0"); return 0; } @@ -3656,26 +3773,25 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, } repair.step_realtime_ns = clock_ns(CLOCK_REALTIME); repair.packet = &packet; - fd = open(entropy_path, O_WRONLY | O_CREAT | O_TRUNC | O_NOFOLLOW | O_CLOEXEC, - 0600); - if (fd < 0 || write_all(fd, packet.entropy, PACKET_ENTROPY_SIZE) != 0 || - close(fd) != 0) { + // The entropy goes to the caller in every case, an untiered restore's + // too; the file already exists, empty, so only its bytes are written. + if (pwrite(entropy_fd, packet.entropy, PACKET_ENTROPY_SIZE, 0) != + PACKET_ENTROPY_SIZE || + close(entropy_fd) != 0) { snprintf(detail, sizeof(detail), "write %.200s: %s", entropy_path, strerror(errno)); return repair_failed("G_REPAIR_PACKET", detail); } - snprintf(text, sizeof(text), "generation=%" PRIu32 "\nelapsed_us=%" PRId64 "\n", - packet.generation, - (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC); - if (state_apply(apply_repair, &repair) != 0 || - replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text) != 0) { + if (state_apply(apply_repair, &repair) != 0) { snprintf(detail, sizeof(detail), "record the restore: %s", strerror(errno)); return repair_failed("G_REPAIR_CLOCK", detail); } format_hex(packet.entropy, GENERATION_ID_SIZE, id); - if (daemon > 0 && packet.online_vp_count == 0 && packet.range_count == 0 && - (packet.flags & PACKET_MEMORY_TARGET) == 0) { + // A plain untiered restore needs no shell work: this helper also + // acknowledges and hands step 12 to the daemon. + if (finish && daemon > 0 && packet.online_vp_count == 0 && + packet.range_count == 0 && (packet.flags & PACKET_MEMORY_TARGET) == 0) { finished = signal_restore(daemon, (packet.flags & PACKET_ACK_REQUIRED) != 0); if (finished > 0) @@ -3703,9 +3819,9 @@ static int cmd_capture(int argc, char **argv) char detail[DETAIL_MAX]; char *end; long request; - int64_t start; unsigned status; pid_t daemon = -1; + int entropy_fd; int result; int lock; @@ -3730,38 +3846,48 @@ static int cmd_capture(int argc, char **argv) // the file reads off the restore path. if (argc == 4) daemon = daemon_pid(); + // The entropy file is created before the capture, empty, so that a + // restored helper only writes its bytes: creating a file right after a + // restore faults in the filesystem's metadata, 4 to 9 ms on WHP. + entropy_fd = open(argv[2], O_WRONLY | O_CREAT | O_TRUNC | O_NOFOLLOW | + O_CLOEXEC, + 0600); + if (entropy_fd < 0) { + fprintf(stderr, "nvx-time: open %s: %s\n", argv[2], strerror(errno)); + return 1; + } lock = lock_portb(detail, sizeof(detail)); if (lock < 0) { fprintf(stderr, "nvx-time: %s\n", detail); + close(entropy_fd); + unlink(argv[2]); return 1; } outb((unsigned char)request, PORT_SNAPSHOT); - start = clock_ns(CLOCK_MONOTONIC); status = inb(PORTB_STATUS); if ((status & PORTB_PACKET_AVAILABLE) == 0) { close(lock); + close(entropy_fd); + unlink(argv[2]); puts("0"); return 0; } - result = repair_restore(previous_id, argv[2], start, daemon); + result = repair_restore(previous_id, argv[2], entropy_fd, argc == 4, + daemon); close(lock); return result; } -// Restore steps 12 and 13 after the shell's activation and identity work: the -// acknowledgement when the packet asked for one, then the daemon runs the -// restore checks (step 11, after the acknowledgement) and finishes the -// restore off the latency path. +// Restore steps 11 and 12 after the shell's activation and identity work: the +// acknowledgement when the packet asked for one, then the daemon runs step 12 +// asynchronously. The record tells it which CPUs activation onlined. static int cmd_restore_finish(int argc, char **argv) { - int64_t start = clock_ns(CLOCK_MONOTONIC); bool new_cpus[MAX_CPUS] = {false}; const char *new_list = "none"; - unsigned long long generation; - long long elapsed; + struct time_state state; struct checks checks; char text[256]; - int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; bool ack = false; int new_count = 0; int signaled; @@ -3781,15 +3907,16 @@ static int cmd_restore_finish(int argc, char **argv) "[--new-cpus LIST]\n"); return 2; } - load_generation(); - if (read_text(RESTORE_PATH, text, sizeof(text)) < 0 || - sscanf(text, "generation=%llu elapsed_us=%lld", &generation, - &elapsed) != 2) - return repair_failed("G_REPAIR_PACKET", "no restore record"); - elapsed += (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC; - snprintf(text, sizeof(text), "generation=%llu\nelapsed_us=%lld\nnew_cpus=%s\n", - generation, elapsed, new_count > 0 ? new_list : "none"); - (void)replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text); + if (state_load(&state) != 0) + return repair_failed("G_REPAIR_CLOCK", "the time state is unreadable"); + g_generation = (uint32_t)state.generation; + snprintf(text, sizeof(text), "generation=%" PRIu64 "\nnew_cpus=%s\n", + state.generation, new_count > 0 ? new_list : "none"); + if (replace_file(RESTORE_PATH, RESTORE_PATH ".tmp", text) != 0) { + snprintf(text, sizeof(text), "record the restore: %s", + strerror(errno)); + return repair_failed("G_REPAIR_CLOCK", text); + } if (ack && enable_ports() != 0) { fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); return 1; @@ -3797,24 +3924,77 @@ static int cmd_restore_finish(int argc, char **argv) signaled = signal_restore(daemon_pid(), ack); if (signaled == 0) return 0; - if (checks_init(&checks, PHASE_RESTORE) != 0) { - check_failed(&checks, "C6", "cannot read the CPU sets"); - return 1; - } if (signaled > 0 || !g_report_only) { + memset(&checks, 0, sizeof(checks)); + checks.phase = PHASE_RESTORE; check_failed(&checks, "C10", "the time daemon is not running"); return 1; } // Report-only runs may lack the daemon; finish the restore here. if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); - run_restore_checks(&checks, new_cpus, new_count); - (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); - complete_restore(release_with_deadline(timeout_s), checks.tsc_hz, - checks.lapic_hz, elapsed, checks.failures); + (void)restore_work(new_cpus, new_count, true); return 0; } +// Formats nvx-time status's lines from STATE: the marker of the last +// conformance check, with status=pending and no elapsed_us while it runs, +// and the runtime line. Report-only mode adds the check's FAILURES. +static void format_status(const struct time_state *state, int failures, + char *text, size_t size) +{ + bool pending = strcmp(state->check_status, "pending") == 0; + char elapsed[40] = ""; + char suffix[32] = ""; + int length; + + if (!pending) + snprintf(elapsed, sizeof(elapsed), " elapsed_us=%" PRId64, + state->check_elapsed_us); + if (g_report_only) + snprintf(suffix, sizeof(suffix), " failures=%d", failures); + length = snprintf(text, size, + "%s: v=1 phase=%s status=%s cpus=%" PRIu64 + " tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 + " generation=%" PRIu64 "%s%s\n", + abi_prefix(), state->check_phase, state->check_status, + state->check_cpus, state->tsc_hz, state->lapic_hz, + state->generation, elapsed, suffix); + if (length < 0 || (size_t)length >= size) + return; + snprintf(text + length, size - (size_t)length, + "%s: v=1 phase=runtime status=%s generation=%" PRIu64 + " discontinuities=%" PRIu64 " offset_ns=%" PRId64 + " uncertainty_ns=%" PRId64 " rejected_samples=%" PRIu64 + " last_sample_error=%s\n", + abi_prefix(), + state->synchronized != 0 ? "synchronized" : "unsynchronized", + state->generation, state->discontinuities, state->offset_ns, + state->uncertainty_ns, state->rejected_samples, + state->last_sample_error); +} + +// Prints the time state on demand, after waiting for pending checks: CI and +// the matrix driver run it over the console; production paths never do. +static int cmd_status(void) +{ + struct time_state state; + char text[768]; + int failures = 0; + int waited = wait_for_checks(&state); + + if (waited < 0) { + fprintf(stderr, "nvx-time: the time state is unavailable\n"); + return 1; + } + if (g_report_only && read_text(REPORT_PATH, text, sizeof(text)) >= 0) + (void)sscanf(text, "failures=%d", &failures); + format_status(&state, failures, text, sizeof(text)); + fputs(text, stdout); + fflush(stdout); + return waited == 0 ? 0 : 1; +} + static int cmd_sample(void) { struct time_sample sample; @@ -4096,19 +4276,18 @@ static int test_restore_record(int argc, char **argv) { static char text[4096]; unsigned long long generation; - long long elapsed_us; bool cpus[MAX_CPUS]; const char *separator = ""; int count; if (argc != 1 || test_text(argv[0], text, sizeof(text)) != 0) return 2; - count = parse_restore_record(text, &generation, &elapsed_us, cpus); + count = parse_restore_record(text, &generation, cpus); if (count < 0) { printf("error\n"); return 0; } - printf("generation=%llu elapsed_us=%lld new_cpus=", generation, elapsed_us); + printf("generation=%llu new_cpus=", generation); for (int cpu = 0; cpu < MAX_CPUS; cpu++) { if (cpus[cpu]) { printf("%s%d", separator, cpu); @@ -4250,6 +4429,19 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "state") == 0) return test_state(argc, argv); + if (strcmp(name, "status") == 0 && argc == 3) { + static char text[4096]; + struct time_state state; + char out[768]; + + if (test_text(argv[0], text, sizeof(text)) != 0 || + state_parse(text, &state) != 0) + return 2; + g_report_only = strcmp(argv[1], "report-only") == 0; + format_status(&state, atoi(argv[2]), out, sizeof(out)); + fputs(out, stdout); + return 0; + } if (strcmp(name, "restore-record") == 0) return test_restore_record(argc, argv); if (strcmp(name, "parallel") == 0) @@ -4292,9 +4484,10 @@ static int usage(void) " boot\n" " check --phase capture|restore [--new-cpus LIST]\n" " pre-capture\n" - " capture 0|1 GENERATION_ID ENTROPY_FILE\n" + " capture 0|1 GENERATION_ID ENTROPY_FILE [--finish]\n" " cancel-capture\n" " restore-finish [--ack] [--new-cpus LIST]\n" + " status\n" " sample\n" " generation-id\n" " exhaustive\n" @@ -4338,6 +4531,8 @@ int main(int argc, char **argv) return cmd_cancel_capture(); if (strcmp(command, "restore-finish") == 0) return cmd_restore_finish(count - 1, arguments + 1); + if (strcmp(command, "status") == 0 && count == 1) + return cmd_status(); if (strcmp(command, "sample") == 0 && count == 1) return cmd_sample(); if (strcmp(command, "generation-id") == 0 && count == 1) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 37ac7356a..5e3a9b048 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -645,8 +645,8 @@ def cpuinfo(flags: str, cpus: int = 2) -> str: "adjust": GOOD_FLAGS + " tsc_adjust", }.items(): with self.subTest(case=name): - self.assertIn("status=fail check=C5", cpuinfo(flags)) - self.assertIn("status=fail check=C5", cpuinfo(GOOD_FLAGS, cpus=1)) + self.assertIn("code=G_CONFORMANCE_C5", cpuinfo(flags)) + self.assertIn("code=G_CONFORMANCE_C5", cpuinfo(GOOD_FLAGS, cpus=1)) def test_command_line_rejects_clock_workarounds(self): accepted = "earlycon=xe9 console=hvc0 quiet clocksource=tsc tsc=reliable" @@ -663,7 +663,7 @@ def test_command_line_rejects_clock_workarounds(self): ): with self.subTest(token=token): self.assertIn( - "status=fail check=C9", + "code=G_CONFORMANCE_C9", self.run_test("cmdline", f"{accepted} {token}"), ) @@ -682,7 +682,7 @@ def test_clocksource_must_be_tsc_with_only_jiffies_fallbacks(self): ): with self.subTest(current=current, available=available): self.assertIn( - "status=fail check=C6", + "code=G_CONFORMANCE_C6", self.run_test("clocksource", current, available), ) @@ -709,8 +709,11 @@ def test_violation_event_format_escaping_and_limit(self): self.assertTrue(long_line.endswith('\\x02"\n')) def test_state_file_round_trip(self): + # The keys and their order follow the spec's state file table. text = ( - "version=1\ngeneration=2\ndiscontinuities=3\nlast_discontinuity=step\n" + "version=1\ngeneration=2\ncheck_phase=restore\ncheck_status=ok\n" + "check_cpus=4\ncheck_elapsed_us=1830\ntsc_hz=2194843000\n" + "lapic_hz=200000000\ndiscontinuities=3\nlast_discontinuity=step\n" "last_step_ns=-200000000\nlast_step_realtime_ns=1790841600000000000\n" "last_downtime_ns=31000000000\nlast_downtime_source=utc\n" "synchronized=1\noffset_ns=-42\nuncertainty_ns=1800\n" @@ -718,6 +721,13 @@ def test_state_file_round_trip(self): "last_sample_error=none\nviolations=0\n" ) self.assertEqual(self.run_test("state", self.fixture("state.txt", text)), text) + # A fresh state is the pending boot check. + self.assertTrue( + self.run_test("state", self.fixture("state.txt", "version=1\n")).startswith( + "version=1\ngeneration=0\ncheck_phase=boot\ncheck_status=pending\n" + "check_cpus=0\ncheck_elapsed_us=0\ntsc_hz=0\nlapic_hz=0\n" + ) + ) self.assertEqual( self.run_test( "state", self.fixture("state.txt", "unknown=x\nsamples=bad\n") @@ -725,33 +735,69 @@ def test_state_file_round_trip(self): "error", ) + def status(self, text: str, mode: str = "enforcing", failures: int = 0) -> str: + return self.run_test( + "status", self.fixture("state.txt", text), mode, str(failures) + ) + + def test_status_prints_the_last_check_and_the_runtime_line(self): + state = ( + "version=1\ngeneration=1\ncheck_phase=restore\ncheck_status=ok\n" + "check_cpus=4\ncheck_elapsed_us=1830\ntsc_hz=2194843000\n" + "lapic_hz=200000000\ndiscontinuities=1\nsynchronized=1\n" + "offset_ns=-42\nuncertainty_ns=1800\nrejected_samples=2\n" + "last_sample_error=G_SAMPLE_UNCERTAIN\n" + ) + self.assertEqual( + self.status(state), + "NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 tsc_hz=2194843000 " + "lapic_hz=200000000 generation=1 elapsed_us=1830\n" + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=1 " + "discontinuities=1 offset_ns=-42 uncertainty_ns=1800 " + "rejected_samples=2 last_sample_error=G_SAMPLE_UNCERTAIN\n", + ) + # A check still pending after the wait has no elapsed_us. + pending = state.replace("check_status=ok", "check_status=pending") + pending = pending.replace("synchronized=1", "synchronized=0") + self.assertEqual( + self.status(pending).splitlines(), + [ + "NVX-TIME-ABI: v=1 phase=restore status=pending cpus=4 " + "tsc_hz=2194843000 lapic_hz=200000000 generation=1", + "NVX-TIME-ABI: v=1 phase=runtime status=unsynchronized " + "generation=1 discontinuities=1 offset_ns=-42 uncertainty_ns=1800 " + "rejected_samples=2 last_sample_error=G_SAMPLE_UNCERTAIN", + ], + ) + # Report-only mode marks both lines and counts the check's failures. + report = self.status(state, "report-only", 3).splitlines() + self.assertTrue(report[0].startswith("NVX-TIME-REPORT: v=1 phase=restore ")) + self.assertTrue(report[0].endswith(" elapsed_us=1830 failures=3")) + self.assertTrue(report[1].startswith("NVX-TIME-REPORT: v=1 phase=runtime ")) + def restore_record(self, text: str) -> str: return self.run_test( "restore-record", self.fixture("restore.txt", text) ).strip() def test_restore_record_names_the_cpus_that_restore_finish_onlined(self): - # The capture helper's record, which the daemon reads when the helper - # finished an untiered restore itself. + # restore-finish records the CPUs that activation brought online. self.assertEqual( - self.restore_record("generation=3\nelapsed_us=1234\n"), - "generation=3 elapsed_us=1234 new_cpus=none count=0", + self.restore_record("generation=3\nnew_cpus=4-6,9\n"), + "generation=3 new_cpus=4,5,6,9 count=4", ) - # restore-finish adds the CPUs that activation brought online. self.assertEqual( - self.restore_record("generation=3\nelapsed_us=1400\nnew_cpus=4-6,9\n"), - "generation=3 elapsed_us=1400 new_cpus=4,5,6,9 count=4", + self.restore_record("generation=3\nnew_cpus=none\n"), + "generation=3 new_cpus=none count=0", ) self.assertEqual( - self.restore_record("generation=3\nelapsed_us=1400\nnew_cpus=none\n"), - "generation=3 elapsed_us=1400 new_cpus=none count=0", + self.restore_record("generation=3\n"), "generation=3 new_cpus=none count=0" ) for text in ( "", - "generation=3\n", - "elapsed_us=1400\ngeneration=3\n", - "generation=3\nelapsed_us=1400\nnew_cpus=7-4\n", - "generation=3\nelapsed_us=1400\nnew_cpus=64\n", + "new_cpus=4\ngeneration=3\n", + "generation=3\nnew_cpus=7-4\n", + "generation=3\nnew_cpus=64\n", ): self.assertEqual(self.restore_record(text), "error", text) @@ -764,8 +810,8 @@ def test_parallel_checks_merge_every_failure(self): for size in sorted({1, min(2, len(allowed)), min(4, len(allowed))}): output = self.run_test("parallel", ",".join(map(str, allowed[:size]))) self.assertTrue(output.endswith("failures=7\n"), output) - self.assertEqual(output.count("check=C4"), 6, output) - self.assertEqual(output.count("check=C9"), 1, output) + self.assertEqual(output.count("code=G_CONFORMANCE_C4"), 6, output) + self.assertEqual(output.count("code=G_CONFORMANCE_C9"), 1, output) # A CPU the checks cannot run on fails C1 in its job. unusable = [cpu for cpu in range(64) if cpu not in allowed] if not unusable: From 69b9a063784f2ad9f8ce1ac3e97ccbc8c4372a6f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 17:46:55 -0700 Subject: [PATCH 101/188] guest: bound the deferred time ABI work and time the readiness path Following spec f4f7305a and a9e1ede3: - The asynchronous boot checks and restore step 13 start at SCHED_IDLE and continue at normal priority 100 ms after shell-ready or after the acknowledgement, which bounds how long a CPU-bound workload runs before a failing check powers the guest off. A normal-priority thread in the boot checker raises the checker's threads; the daemon raises the restore worker. - A restore's elapsed_us covers the readiness path, from the capture request's return through the acknowledgement (steps 6 to 12), and the step 11 checks, but no longer the grace-period wait or the deferred C7. The helper records its part in the state file, and the time through the acknowledgement travels in the daemon's signal. - The daemon reaps the previous restore's worker before it starts the next one: the signalfd returns SIGUSR1 before SIGCHLD, so a worker that ended just before a capture made the daemon skip step 13. - The watcher reports "APIC ID mismatch" as G_APIC_ID_MISMATCH. - The step numbers follow the spec again: step 11 defers the checks, step 12 acknowledges, and step 13 is the daemon's. On prometheus32 (KVM), a restore's worker reads SCHED_IDLE 20 ms after the signal and SCHED_OTHER 220 ms after it at 1, 2, and 8 vCPUs; the previous build keeps it at SCHED_IDLE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-snapshot | 10 +- guest/common/nvx-time.c | 342 ++++++++++++++++++++++++++++++------- scripts/test_guest_time.py | 58 +++++-- 3 files changed, 331 insertions(+), 79 deletions(-) diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 880465663..21c641c89 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -287,7 +287,7 @@ esac export NVX_VM_GENERATION_ID="$generation_id" entropy=$runtime/restore-entropy -# Restore steps 9 to 12 of the time ABI (doc/design/time-abi.md). nvx-time +# Restore steps 9 to 13 of the time ABI (doc/design/time-abi.md). nvx-time # capture has already read restore packet v4, checked its generation, set # the wall clock, and written the entropy. Its metadata is FLAGS, # ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by START @@ -391,10 +391,10 @@ post_restore() { finish_restore "$packet_flags" } -# Restore steps 11 and 12: the acknowledgement when the host gated its input, -# then the daemon's restore checks and grace-period release, which run off -# the restore latency path. A second argument of "quiet" omits the stage and -# completion lines of a plain untiered restore. +# Restore steps 12 and 13: the acknowledgement when the host gated its input, +# then the daemon's step 11 restore checks and grace-period release, which +# run off the restore latency path. A second argument of "quiet" omits the +# stage and completion lines of a plain untiered restore. finish_restore() { [ "${2:-}" = quiet ] || console_status "NVX-POST-RESTORE-STAGE: acknowledge" diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 23b06c7de..478e6e90a 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -34,6 +34,7 @@ #define _GNU_SOURCE +#include #include #include #include @@ -122,6 +123,10 @@ #define SLOW_TIME_CONSTANT 6 #define TIMER_LIST_WAIT_NS (200 * NSEC_PER_MSEC) #define DEFAULT_STALL_TIMEOUT_S 21 +// The asynchronous boot checks and restore step 13 run at SCHED_IDLE for +// 100 ms, then at normal priority, which bounds the fail-fast window under a +// CPU-bound workload. +#define IDLE_BOUND_NS (100 * NSEC_PER_MSEC) // nvx-time status and pre-capture wait at most 30 s for pending checks. #define STATUS_WAIT_NS (30 * NSEC_PER_SEC) #define STATUS_POLL_NS (10 * NSEC_PER_MSEC) @@ -1240,6 +1245,7 @@ static const struct watch_rule k_watch_rules[] = { {"G_HARD_LOCKUP", {"Watchdog detected hard LOCKUP", NULL}, {NULL}}, {"G_HUNG_TASK", {"INFO: task ", " blocked for more than "}, {NULL}}, {"G_UNCHECKED_MSR", {"unchecked MSR access error", NULL}, {NULL}}, + {"G_APIC_ID_MISMATCH", {"APIC ID mismatch", NULL}, {NULL}}, }; // Returns the code of the first rule whose substrings all appear in MESSAGE. @@ -2471,11 +2477,11 @@ static pid_t daemon_pid(void) return (pid_t)pid; } -// The asynchronous boot checks and restore step 12 run at idle priority, so -// that on few vCPUs they never take a CPU from the workload; the watcher and -// the discipline keep normal priority. musl does not implement -// sched_setscheduler(), so this uses the per-thread system call: PID 0 is -// the calling thread. +// The asynchronous boot checks and restore step 13 start at idle priority, so +// that on few vCPUs they never take a CPU from the workload, and continue at +// normal priority after IDLE_BOUND_NS; the watcher and the discipline keep +// normal priority. musl does not implement sched_setscheduler(), so this uses +// the per-thread system call: PID 0 is the calling thread. static void set_idle_priority(pid_t pid, bool idle) { struct sched_param param = {0}; @@ -2484,17 +2490,103 @@ static void set_idle_priority(pid_t pid, bool idle) ¶m); } -// Restore steps 11 and 12 on the readiness path: the acknowledgement when the -// host gated its input, then hand step 12 to DAEMON. Returns -1, without -// acknowledging, when no daemon runs; 1 when the daemon vanished after the -// acknowledgement. The caller has enabled the ports when ACK is set. -static int signal_restore(pid_t daemon, bool ack) +// Raises every thread of this process to normal priority, twice, so that a +// thread an idle thread created during the first pass is raised too. +static void promote_threads(void) { + for (int pass = 0; pass < 2; pass++) { + DIR *tasks = opendir("/proc/self/task"); + struct dirent *entry; + + if (tasks == NULL) + return; + while ((entry = readdir(tasks)) != NULL) { + long tid = strtol(entry->d_name, NULL, 10); + + if (tid > 0) + set_idle_priority((pid_t)tid, false); + } + closedir(tasks); + } +} + +// A normal-priority thread that raises the boot checker's idle threads when +// the checks still run IDLE_BOUND_NS after they started. +struct promoter { + pthread_t thread; + pthread_mutex_t lock; + pthread_cond_t stop; + struct timespec deadline; + bool stopped; +}; + +static void *promoter_run(void *argument) +{ + struct promoter *promoter = argument; + bool stopped; + int result = 0; + + pthread_mutex_lock(&promoter->lock); + while (!promoter->stopped && result == 0) + result = pthread_cond_timedwait(&promoter->stop, &promoter->lock, + &promoter->deadline); + stopped = promoter->stopped; + pthread_mutex_unlock(&promoter->lock); + if (!stopped) + promote_threads(); + return NULL; +} + +// Starts PROMOTER, at the caller's priority, for checks that started at +// START_NS. Returns true when it runs. +static bool promoter_start(struct promoter *promoter, int64_t start_ns) +{ + int64_t deadline = start_ns + IDLE_BOUND_NS; + pthread_condattr_t attributes; + bool ready; + + promoter->stopped = false; + promoter->deadline.tv_sec = deadline / NSEC_PER_SEC; + promoter->deadline.tv_nsec = deadline % NSEC_PER_SEC; + if (pthread_condattr_init(&attributes) != 0) + return false; + ready = pthread_condattr_setclock(&attributes, CLOCK_MONOTONIC) == 0 && + pthread_cond_init(&promoter->stop, &attributes) == 0; + pthread_condattr_destroy(&attributes); + return ready && pthread_mutex_init(&promoter->lock, NULL) == 0 && + pthread_create(&promoter->thread, NULL, promoter_run, promoter) == + 0; +} + +static void promoter_stop(struct promoter *promoter) +{ + pthread_mutex_lock(&promoter->lock); + promoter->stopped = true; + pthread_cond_signal(&promoter->stop); + pthread_mutex_unlock(&promoter->lock); + pthread_join(promoter->thread, NULL); +} + +// Restore steps 12 and 13 on the readiness path: the acknowledgement when the +// host gated its input, then hand step 13 to DAEMON, with the readiness +// path's duration since START_NS (CLOCK_MONOTONIC), in microseconds, in the +// signal. Returns -1, without acknowledging, when no daemon runs; 1 when the +// daemon vanished after the acknowledgement. The caller has enabled the ports +// when ACK is set. +static int signal_restore(pid_t daemon, bool ack, int64_t start_ns) +{ + union sigval value; + int64_t elapsed_us; + if (daemon <= 0 || kill(daemon, 0) != 0) return -1; if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); - return kill(daemon, SIGUSR1) == 0 ? 0 : 1; + elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; + value.sival_int = (int)(elapsed_us < 0 ? 0 + : elapsed_us > INT_MAX ? INT_MAX + : elapsed_us); + return sigqueue(daemon, SIGUSR1, value) == 0 ? 0 : 1; } // C10: the daemon runs, has recorded no violation, and no stall was counted. @@ -2862,8 +2954,11 @@ struct daemon { int polls; int64_t last_accepted_ns; pid_t worker_pid; + int64_t worker_promote_ns; int64_t worker_deadline_ns; bool suppression_released; + bool restore_pending; + int64_t readiness_us; }; static void watcher_consume(void *context, const char *message) @@ -3075,13 +3170,17 @@ static int parse_restore_record(const char *text, unsigned long long *generation return parse_new_cpus(cpus, new_cpus); } -// Restore step 12, fail-fast: the restore checks, the grace-period release, -// the saved watchdog settings, the deferred C7, and the restore check's -// record. A child of the daemon runs it at SCHED_IDLE, and the daemon -// releases the suppression itself at the stall timeout; restore-finish runs it -// inline, with its own deadline (WITH_DEADLINE), in report-only mode when no -// daemon runs. Returns 0, or 1 when a report-only release failed. -static int restore_work(const bool *new_cpus, int new_count, bool with_deadline) +// Restore step 13, fail-fast: the step 11 restore checks, the grace-period +// release, the saved watchdog settings, the deferred C7, and the restore +// check's record. A child of the daemon runs it at SCHED_IDLE, and the daemon +// raises it to normal priority after 100 ms and releases the suppression +// itself at the stall timeout; restore-finish runs it inline, with its own +// deadline (WITH_DEADLINE), in report-only mode when no daemon runs. The +// recorded duration is READINESS_US, the readiness path's, plus the checks', +// without the grace period wait or C7. Returns 0, or 1 when a report-only +// release failed. +static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, + int64_t readiness_us) { int64_t start = clock_ns(CLOCK_MONOTONIC); int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; @@ -3107,20 +3206,19 @@ static int restore_work(const bool *new_cpus, int new_count, bool with_deadline) if (suppression_restore(detail, sizeof(detail), true) != 0) fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", PHASE_RESTORE, "%s", detail); - start = clock_ns(CLOCK_MONOTONIC); check_timer_list(&checks, TIMER_LIST_WAIT_NS); - elapsed += clock_ns(CLOCK_MONOTONIC) - start; if (record_check(PHASE_RESTORE, "ok", checks.online_count, - elapsed / NSEC_PER_USEC, 0, 0, checks.failures) != 0) + readiness_us + elapsed / NSEC_PER_USEC, 0, 0, + checks.failures) != 0) check_failed(&checks, "C10", "cannot record the restore check: %s", strerror(errno)); return released ? 0 : 1; } -// Starts restore step 12 after the acknowledgement, in a child at +// Starts restore step 13 after the acknowledgement, in a child at // SCHED_IDLE, so the watcher and the discipline keep normal priority; the -// fork itself also runs at SCHED_IDLE. -static void start_restore(struct daemon *daemon) +// fork itself also runs at SCHED_IDLE. READINESS_US comes with the signal. +static void start_restore(struct daemon *daemon, int64_t readiness_us) { bool new_cpus[MAX_CPUS] = {false}; struct time_state state; @@ -3156,7 +3254,7 @@ static void start_restore(struct daemon *daemon) set_idle_priority(0, true); pid = fork(); if (pid == 0) - _exit(restore_work(new_cpus, new_count, false)); + _exit(restore_work(new_cpus, new_count, false, readiness_us)); set_idle_priority(0, false); if (pid < 0) { fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", @@ -3165,6 +3263,7 @@ static void start_restore(struct daemon *daemon) return; } daemon->worker_pid = pid; + daemon->worker_promote_ns = clock_ns(CLOCK_MONOTONIC) + IDLE_BOUND_NS; daemon->worker_deadline_ns = clock_ns(CLOCK_MONOTONIC) + timeout_s * NSEC_PER_SEC; daemon->suppression_released = false; @@ -3192,6 +3291,7 @@ static void reap_children(struct daemon *daemon) if (pid != daemon->worker_pid) continue; daemon->worker_pid = 0; + daemon->worker_promote_ns = 0; // A failing worker powered the guest off itself, or reported in // report-only mode; a crash leaves the suppression to the daemon. if (!WIFEXITED(status)) { @@ -3200,7 +3300,7 @@ static void reap_children(struct daemon *daemon) PHASE_RESTORE, "the restore work ended with signal %d", WTERMSIG(status)); } - // Step 12 ends by restarting the discipline at the fast cadence. + // Step 13 ends by restarting the discipline at the fast cadence. daemon->polls = 0; (void)arm_timer(daemon); } @@ -3213,10 +3313,19 @@ static void handle_signals(struct daemon *daemon) while (read(daemon->signals, &info, sizeof(info)) == sizeof(info)) { if (info.ssi_signo == SIGTERM) _exit(0); - if (info.ssi_signo == SIGUSR1) - start_restore(daemon); - else if (info.ssi_signo == SIGCHLD) - reap_children(daemon); + // sigqueue() carries the readiness path's duration in microseconds. + if (info.ssi_signo == SIGUSR1) { + daemon->restore_pending = true; + daemon->readiness_us = + info.ssi_code == SI_QUEUE ? info.ssi_int : 0; + } + } + // The previous restore's worker can end just before a capture, and the + // signalfd returns SIGUSR1 before SIGCHLD: reap before starting. + reap_children(daemon); + if (daemon->restore_pending && daemon->worker_pid == 0) { + daemon->restore_pending = false; + start_restore(daemon, daemon->readiness_us); } } @@ -3246,6 +3355,20 @@ static int daemon_setup(struct daemon *daemon) return 0; } +// Returns the poll timeout in milliseconds until DEADLINE_NS, or -1 when +// DEADLINE_NS is 0, for none. +static int poll_timeout_until(int64_t deadline_ns) +{ + int64_t remaining; + + if (deadline_ns == 0) + return -1; + remaining = deadline_ns - clock_ns(CLOCK_MONOTONIC); + return remaining <= 0 + ? 0 + : (int)((remaining + NSEC_PER_MSEC - 1) / NSEC_PER_MSEC); +} + static void daemon_loop(struct daemon *daemon) { for (;;) { @@ -3253,16 +3376,14 @@ static void daemon_loop(struct daemon *daemon) {daemon->signals, POLLIN, 0}, {daemon->timer, POLLIN, 0}}; bool waiting = daemon->worker_pid > 0 && !daemon->suppression_released; - int timeout = -1; - - if (waiting) { - int64_t remaining = - daemon->worker_deadline_ns - clock_ns(CLOCK_MONOTONIC); - - timeout = remaining <= 0 ? 0 - : (int)((remaining + NSEC_PER_MSEC - 1) / - NSEC_PER_MSEC); - } + bool idle = daemon->worker_pid > 0 && daemon->worker_promote_ns != 0; + int timeout = poll_timeout_until(idle ? daemon->worker_promote_ns : 0); + int deadline = + poll_timeout_until(waiting ? daemon->worker_deadline_ns : 0); + int64_t now; + + if (timeout < 0 || (deadline >= 0 && deadline < timeout)) + timeout = deadline; if (poll(fds, ARRAY_SIZE(fds), timeout) < 0 && errno != EINTR) { sleep(1); continue; @@ -3280,8 +3401,15 @@ static void daemon_loop(struct daemon *daemon) discipline_poll(daemon); arm_timer(daemon); } + now = clock_ns(CLOCK_MONOTONIC); + // Step 13 continues at normal priority 100 ms after the ack. + if (daemon->worker_pid > 0 && daemon->worker_promote_ns != 0 && + now >= daemon->worker_promote_ns) { + daemon->worker_promote_ns = 0; + set_idle_priority(daemon->worker_pid, false); + } if (daemon->worker_pid > 0 && !daemon->suppression_released && - clock_ns(CLOCK_MONOTONIC) >= daemon->worker_deadline_ns) + now >= daemon->worker_deadline_ns) release_suppression(daemon); } } @@ -3386,21 +3514,23 @@ static void boot_foreground(struct checks *checks, void *context) check_debug_watchdogs(checks); } -// The asynchronous boot checks: every boot check but C12, at SCHED_IDLE -// after shell-ready, and fail-fast. When they pass, the checker becomes the -// daemon and records the boot check, whose duration adds C12's (C12_NS). -// It never returns. +// The asynchronous boot checks: every boot check but C12, after shell-ready, +// fail-fast, at SCHED_IDLE for at most IDLE_BOUND_NS. When they pass, the +// checker becomes the daemon and records the boot check, whose duration adds +// C12's (C12_NS) and the daemon's start. It never returns. static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, int c12_failures) { int64_t start = clock_ns(CLOCK_MONOTONIC); int null = open("/dev/null", O_RDWR | O_CLOEXEC); struct boot_log_job log_job; + struct promoter promoter; struct side_job sides[3]; struct daemon daemon; struct checks checks; struct boot_log log; bool started = false; + bool promoting; int kmsg; g_async_output = true; @@ -3413,6 +3543,7 @@ static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, } if (chdir("/") != 0) _exit(1); + promoting = promoter_start(&promoter, start); set_idle_priority(0, true); if (checks_init(&checks, PHASE_BOOT) != 0) { check_failed(&checks, "C1", "cannot read the CPU sets"); @@ -3451,6 +3582,9 @@ static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, "cpu 0: floor(F / 1000) is %" PRIu64 " kHz but cpu MHz is %" PRIu64 " kHz", checks.tsc_hz / 1000, checks.cpu_khz[0]); + // The daemon blocks its signals in its only thread. + if (promoting) + promoter_stop(&promoter); if (kmsg >= 0) { started = become_daemon(&daemon, kmsg, sample->accepted_ns) == 0; if (!started) @@ -3661,6 +3795,7 @@ struct repair_record { int64_t epsilon; int64_t step_realtime_ns; int64_t frequency; + int64_t elapsed_us; }; static void apply_repair(struct time_state *state, const void *context) @@ -3685,11 +3820,13 @@ static void apply_repair(struct time_state *state, const void *context) state->samples++; snprintf(state->last_sample_error, sizeof(state->last_sample_error), "none"); - // The daemon's step 12 runs the restore check after the acknowledgement. + // The daemon's step 13 runs the restore checks after the acknowledgement. + // Until then, check_elapsed_us is the readiness path up to the clock + // step, at last_step_realtime_ns. snprintf(state->check_phase, sizeof(state->check_phase), "restore"); snprintf(state->check_status, sizeof(state->check_status), "pending"); state->check_cpus = 0; - state->check_elapsed_us = 0; + state->check_elapsed_us = repair->elapsed_us; } static int repair_failed(const char *code, const char *detail) @@ -3700,13 +3837,15 @@ static int repair_failed(const char *code, const char *detail) // Restore steps 6 to 8: read the packet with four-byte reads, validate it, // and set the wall clock from its bracketed UTC, then write the entropy to -// ENTROPY_FD, the caller's file at ENTROPY_PATH. The caller holds the portb -// lock. With FINISH (untiered) and a DAEMON, a restore with no processors or -// memory to activate needs no shell work before the acknowledgement, so -// steps 11 and 12 start here too and no second helper process starts; the -// metadata then begins with 2 instead of 1. +// ENTROPY_FD, the caller's file at ENTROPY_PATH. START_NS is when the capture +// request returned. The caller holds the portb lock. With FINISH (untiered) +// and a DAEMON, a restore with no processors or memory to activate needs no +// shell work before the acknowledgement, so steps 12 and 13 start here too +// and no second helper process starts; the metadata then begins with 2 +// instead of 1. static int repair_restore(const uint8_t *previous_id, const char *entropy_path, - int entropy_fd, bool finish, pid_t daemon) + int entropy_fd, int64_t start_ns, bool finish, + pid_t daemon) { static uint8_t body[PACKET_MAX_SIZE]; static struct restore_packet packet; @@ -3772,6 +3911,7 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, return repair_failed("G_REPAIR_CLOCK", detail); } repair.step_realtime_ns = clock_ns(CLOCK_REALTIME); + repair.elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; repair.packet = &packet; // The entropy goes to the caller in every case, an untiered restore's // too; the file already exists, empty, so only its bytes are written. @@ -3789,11 +3929,11 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, } format_hex(packet.entropy, GENERATION_ID_SIZE, id); // A plain untiered restore needs no shell work: this helper also - // acknowledges and hands step 12 to the daemon. + // acknowledges and hands step 13 to the daemon. if (finish && daemon > 0 && packet.online_vp_count == 0 && packet.range_count == 0 && (packet.flags & PACKET_MEMORY_TARGET) == 0) { - finished = signal_restore(daemon, - (packet.flags & PACKET_ACK_REQUIRED) != 0); + finished = signal_restore( + daemon, (packet.flags & PACKET_ACK_REQUIRED) != 0, start_ns); if (finished > 0) fail_fatal(STATUS_CONFORMANCE, "G_CONFORMANCE_C10", "conformance", PHASE_RESTORE, "the time daemon is not running"); @@ -3819,6 +3959,7 @@ static int cmd_capture(int argc, char **argv) char detail[DETAIL_MAX]; char *end; long request; + int64_t start; unsigned status; pid_t daemon = -1; int entropy_fd; @@ -3864,6 +4005,7 @@ static int cmd_capture(int argc, char **argv) return 1; } outb((unsigned char)request, PORT_SNAPSHOT); + start = clock_ns(CLOCK_MONOTONIC); status = inb(PORTB_STATUS); if ((status & PORTB_PACKET_AVAILABLE) == 0) { close(lock); @@ -3872,15 +4014,17 @@ static int cmd_capture(int argc, char **argv) puts("0"); return 0; } - result = repair_restore(previous_id, argv[2], entropy_fd, argc == 4, + result = repair_restore(previous_id, argv[2], entropy_fd, start, argc == 4, daemon); close(lock); return result; } -// Restore steps 11 and 12 after the shell's activation and identity work: the -// acknowledgement when the packet asked for one, then the daemon runs step 12 -// asynchronously. The record tells it which CPUs activation onlined. +// Restore steps 12 and 13 after the shell's activation and identity work: the +// acknowledgement when the packet asked for one, then the daemon runs step 13 +// asynchronously. The record tells it which CPUs activation onlined. The +// readiness path started check_elapsed_us before the helper's clock step, at +// last_step_realtime_ns. static int cmd_restore_finish(int argc, char **argv) { bool new_cpus[MAX_CPUS] = {false}; @@ -3888,6 +4032,7 @@ static int cmd_restore_finish(int argc, char **argv) struct time_state state; struct checks checks; char text[256]; + int64_t start; bool ack = false; int new_count = 0; int signaled; @@ -3921,7 +4066,10 @@ static int cmd_restore_finish(int argc, char **argv) fprintf(stderr, "nvx-time: ioperm: %s\n", strerror(errno)); return 1; } - signaled = signal_restore(daemon_pid(), ack); + start = clock_ns(CLOCK_MONOTONIC) - + (clock_ns(CLOCK_REALTIME) - state.last_step_realtime_ns) - + state.check_elapsed_us * NSEC_PER_USEC; + signaled = signal_restore(daemon_pid(), ack, start); if (signaled == 0) return 0; if (signaled > 0 || !g_report_only) { @@ -3933,7 +4081,8 @@ static int cmd_restore_finish(int argc, char **argv) // Report-only runs may lack the daemon; finish the restore here. if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); - (void)restore_work(new_cpus, new_count, true); + (void)restore_work(new_cpus, new_count, true, + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC); return 0; } @@ -4334,6 +4483,73 @@ static int test_parallel(int argc, char **argv) return 0; } +// A thread that an idle thread starts, so it is idle too; it waits until the +// test ends. +struct test_thread { + pthread_mutex_t lock; + pthread_cond_t changed; + pid_t tid; + bool done; +}; + +static void *test_thread_run(void *argument) +{ + struct test_thread *thread = argument; + + pthread_mutex_lock(&thread->lock); + thread->tid = (pid_t)syscall(SYS_gettid); + pthread_cond_broadcast(&thread->changed); + while (!thread->done) + pthread_cond_wait(&thread->changed, &thread->lock); + pthread_mutex_unlock(&thread->lock); + return NULL; +} + +// Prints the policies of an idle caller and of the idle thread it started +// once a promoter fires 50 ms later, then the caller's policy after a +// promoter stopped before its 10 s bound, and how long stopping took. +static int test_promote(void) +{ + struct test_thread thread = {.lock = PTHREAD_MUTEX_INITIALIZER, + .changed = PTHREAD_COND_INITIALIZER}; + struct promoter promoter; + pthread_t handle; + long caller; + long started; + int64_t start; + + if (!promoter_start(&promoter, clock_ns(CLOCK_MONOTONIC) - IDLE_BOUND_NS + + 50 * NSEC_PER_MSEC)) + return 1; + set_idle_priority(0, true); + if (pthread_create(&handle, NULL, test_thread_run, &thread) != 0) + return 1; + pthread_mutex_lock(&thread.lock); + while (thread.tid == 0) + pthread_cond_wait(&thread.changed, &thread.lock); + pthread_mutex_unlock(&thread.lock); + pthread_join(promoter.thread, NULL); + caller = syscall(SYS_sched_getscheduler, 0); + started = syscall(SYS_sched_getscheduler, thread.tid); + pthread_mutex_lock(&thread.lock); + thread.done = true; + pthread_cond_broadcast(&thread.changed); + pthread_mutex_unlock(&thread.lock); + pthread_join(handle, NULL); + printf("promoted caller=%ld thread=%ld\n", caller, started); + + start = clock_ns(CLOCK_MONOTONIC); + if (!promoter_start(&promoter, + start - IDLE_BOUND_NS + 10 * NSEC_PER_SEC)) + return 1; + set_idle_priority(0, true); + promoter_stop(&promoter); + printf("stopped caller=%ld stop_ms=%" PRId64 "\n", + syscall(SYS_sched_getscheduler, 0), + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_MSEC); + return 0; +} + static int cmd_test(int argc, char **argv) { const char *name = argc > 0 ? argv[0] : ""; @@ -4446,6 +4662,8 @@ static int cmd_test(int argc, char **argv) return test_restore_record(argc, argv); if (strcmp(name, "parallel") == 0) return test_parallel(argc, argv); + if (strcmp(name, "promote") == 0 && argc == 0) + return test_promote(); if (strcmp(name, "idle-priority") == 0 && argc == 0) { long idle; diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 5e3a9b048..852714791 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -66,6 +66,23 @@ def encode_sample( return struct.pack(" bool: + """Whether this process may switch from SCHED_IDLE back to SCHED_OTHER. + + That needs CAP_SYS_NICE, which the guest has, or an RLIMIT_NICE of at + least 20; containers and CI runners usually have neither. + """ + status = Path("/proc/self/status").read_text(encoding="utf-8") + capabilities = re.search(r"^CapEff:\s*([0-9a-f]+)$", status, re.MULTILINE) + if capabilities is not None and int(capabilities.group(1), 16) & (1 << 23): + return True + limits = Path("/proc/self/limits").read_text(encoding="utf-8") + nice = re.search(r"^Max nice priority\s+(\S+)", limits, re.MULTILINE) + return nice is not None and ( + nice.group(1) == "unlimited" or int(nice.group(1)) >= 20 + ) + + TIMER_LIST_CPU = """\ Tick Device: mode: 1 Per CPU device: {cpu} @@ -213,18 +230,22 @@ def test_musl_build_matches_the_reference_build(self): self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.stdout, self.run_test(*arguments[1:])) # musl has no sched_setscheduler(); the daemon must still reach - # SCHED_IDLE after a restore. Returning to SCHED_OTHER needs - # CAP_SYS_NICE, which the guest has and a container may not. - result = subprocess.run( - [str(binary), "test", "idle-priority"], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertTrue(result.stdout.startswith("idle=5 normal="), result.stdout) - self.assertTrue(self.run_test("idle-priority").startswith("idle=5 normal=")) + # SCHED_IDLE after a restore and leave it after the bound. + policy = 0 if can_leave_idle_priority() else 5 + for name, expected in ( + ("idle-priority", f"idle=5 normal={policy}\n"), + ("promote", f"promoted caller={policy} thread={policy}\n"), + ): + result = subprocess.run( + [str(binary), "test", name], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(result.stdout.startswith(expected), result.stdout) + self.assertTrue(self.run_test(name).startswith(expected)) def test_packet_memory_ranges_follow_the_header(self): packet = encode_packet( @@ -395,6 +416,8 @@ def test_watcher_rules_match_linux_6_18_messages(self): ), "unchecked MSR access error: RDMSR from 0x1ad at rIP: " "0xffffffff816f11ee (__rdmsr_on_cpu+0x1e/0x30)": "G_UNCHECKED_MSR", + "[Firmware Bug]: CPU 1: APIC ID mismatch. CPUID: 0x0000 " + "APIC: 0x0001": "G_APIC_ID_MISMATCH", "clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fa3": "none", "Hyper-V: LAPIC Timer Frequency: 0x1e8480": "none", "rcu: Hierarchical RCU implementation.": "none", @@ -820,6 +843,17 @@ def test_parallel_checks_merge_every_failure(self): self.assertTrue(output.endswith("failures=8\n"), output) self.assertIn(f"cannot run on cpu {unusable[0]}", output) + def test_idle_work_continues_at_normal_priority_after_the_bound(self): + # The asynchronous checks leave SCHED_IDLE (5) for SCHED_OTHER (0) + # 100 ms after they start, every thread included; checks that end + # first stop the promoter without waiting for the bound. + policy = 0 if can_leave_idle_priority() else 5 + promoted, stopped = self.run_test("promote").splitlines() + self.assertEqual(promoted, f"promoted caller={policy} thread={policy}") + fields = dict(field.split("=") for field in stopped.split()[1:]) + self.assertEqual(fields["caller"], "5") + self.assertLess(int(fields["stop_ms"]), 1000) + if __name__ == "__main__": unittest.main() From d006a819bba667b780e61b0543f696fa99692742 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 18:23:42 -0700 Subject: [PATCH 102/188] guest: report every recorded time ABI check from nvx-time status Following spec a9ef0f5b and c7c2b76d, the state file keeps one record per phase instead of the last check only: boot_status, capture_status, and restore_status, with each check's CPUs, duration, and, for capture and restore, the generation it ran in. A phase's keys are absent until its check first runs. - nvx-time status prints one line per recorded phase, in the order boot, capture, and restore, then the runtime line. It exits with 0 only when every phase line reports ok; a missing state file or boot record counts as a pending boot check. - Step 8 records restore_status=pending and restore_generation in the state-file update that records the restore, so status waits for that restore's deferred work. - The capture record measures the step 1 checks only and travels in the snapshot. Step 1 waits for pending boot checks only. - Report-only mode records a failed check's status as fail, with the boot_failures, capture_failures, and restore_failures keys the spec reserves for it, instead of a separate report file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 6 +- guest/common/nvx-time.c | 371 +++++++++++++++++++++++-------------- scripts/test_guest_time.py | 119 ++++++++---- 3 files changed, 320 insertions(+), 176 deletions(-) diff --git a/doc/build.md b/doc/build.md index 0192bacd5..8edb168d1 100644 --- a/doc/build.md +++ b/doc/build.md @@ -151,9 +151,9 @@ checks, the violation watcher, the restore packet v4 and time-sample parsers, the snapshot agent's time steps, and the wall-clock discipline. Before shell-ready it only steps the clock to host UTC; the other boot checks run in the background, and the production console shows no time ABI line except a -violation event. `nvx-time status` prints the last check's marker and the -runtime state on demand, after waiting for pending checks, for CI and test -harnesses. Its boot check +violation event. `nvx-time status` prints every recorded check (boot, capture, +and restore) and the runtime state on demand, after waiting for pending +checks, for CI and test harnesses. Its boot check also fails when the kernel's boot-time W+X audit reports a writable and executable mapping. `nvx-time exhaustive` runs the CI-only exhaustive check (checks X1 to X6 on every online CPU); it reports and exits, and production diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index 478e6e90a..c627c2ff8 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -20,7 +20,8 @@ // restore-finish [--ack] [--new-cpus LIST] // acknowledgement, then the daemon's restore // checks and release -// status print the last check and the runtime state +// status print every recorded check and the runtime +// state, after waiting for pending checks // sample print one host time sample // generation-id print the VM generation ID // exhaustive the CI-only exhaustive check @@ -139,7 +140,6 @@ #define STATE_PATH TIME_DIR "/state" #define STATE_TEMP_PATH TIME_DIR "/.state" #define STATE_LOCK_PATH TIME_DIR "/state.lock" -#define REPORT_PATH TIME_DIR "/report" #define DAEMON_PID_PATH TIME_DIR "/daemon.pid" #define SUPPRESSION_PATH TIME_DIR "/suppression" #define SUPPRESSION_LOCK_PATH TIME_DIR "/suppression.lock" @@ -652,12 +652,25 @@ static void fail_fatal(int status, const char *code, const char *source, // Published state (/run/nvx/time/state) and private records // --------------------------------------------------------------------------- +// A conformance check's record: STATUS is empty until the phase first runs, +// then "pending" while its check runs, then "ok" (a failed check powers the +// guest off; report-only mode records "fail" instead). GENERATION is g when +// the check ran; the boot check's is 0. FAILURES is the check's failure +// count, kept in report-only mode only, and -1 otherwise. +struct check_state { + char status[16]; + uint64_t cpus; + int64_t elapsed_us; + uint64_t generation; + int64_t failures; +}; + +// The phases with a check record, indexed by enum phase. +#define CHECK_PHASES 3 + struct time_state { uint64_t generation; - char check_phase[16]; - char check_status[16]; - uint64_t check_cpus; - int64_t check_elapsed_us; + struct check_state checks[CHECK_PHASES]; uint64_t tsc_hz; uint64_t lapic_hz; uint64_t discontinuities; @@ -677,12 +690,14 @@ struct time_state { }; // A fresh state is published at the start of the boot checks, which are then -// pending. +// pending; a state without a boot record also counts as pending. static void state_init(struct time_state *state) { memset(state, 0, sizeof(*state)); - snprintf(state->check_phase, sizeof(state->check_phase), "boot"); - snprintf(state->check_status, sizeof(state->check_status), "pending"); + for (int phase = 0; phase < CHECK_PHASES; phase++) + state->checks[phase].failures = -1; + snprintf(state->checks[PHASE_BOOT].status, + sizeof(state->checks[PHASE_BOOT].status), "pending"); snprintf(state->last_discontinuity, sizeof(state->last_discontinuity), "none"); snprintf(state->last_downtime_source, sizeof(state->last_downtime_source), @@ -700,14 +715,26 @@ static void copy_word(char *destination, size_t size, const char *value) // known numeric key fails the parse. static int state_parse(char *text, struct time_state *state) { + struct check_state *boot = &state->checks[PHASE_BOOT]; + struct check_state *capture = &state->checks[PHASE_CAPTURE]; + struct check_state *restore = &state->checks[PHASE_RESTORE]; struct { const char *key; uint64_t *unsigned_value; int64_t *signed_value; } const numbers[] = { {"generation", &state->generation, NULL}, - {"check_cpus", &state->check_cpus, NULL}, - {"check_elapsed_us", NULL, &state->check_elapsed_us}, + {"boot_cpus", &boot->cpus, NULL}, + {"capture_cpus", &capture->cpus, NULL}, + {"restore_cpus", &restore->cpus, NULL}, + {"boot_elapsed_us", NULL, &boot->elapsed_us}, + {"capture_elapsed_us", NULL, &capture->elapsed_us}, + {"restore_elapsed_us", NULL, &restore->elapsed_us}, + {"capture_generation", &capture->generation, NULL}, + {"restore_generation", &restore->generation, NULL}, + {"boot_failures", NULL, &boot->failures}, + {"capture_failures", NULL, &capture->failures}, + {"restore_failures", NULL, &restore->failures}, {"tsc_hz", &state->tsc_hz, NULL}, {"lapic_hz", &state->lapic_hz, NULL}, {"discontinuities", &state->discontinuities, NULL}, @@ -727,8 +754,9 @@ static int state_parse(char *text, struct time_state *state) char *value; size_t size; } const words[] = { - {"check_phase", state->check_phase, sizeof(state->check_phase)}, - {"check_status", state->check_status, sizeof(state->check_status)}, + {"boot_status", boot->status, sizeof(boot->status)}, + {"capture_status", capture->status, sizeof(capture->status)}, + {"restore_status", restore->status, sizeof(restore->status)}, {"last_discontinuity", state->last_discontinuity, sizeof(state->last_discontinuity)}, {"last_downtime_source", state->last_downtime_source, @@ -768,41 +796,85 @@ static int state_parse(char *text, struct time_state *state) return 0; } +// Appends a formatted line to TEXT at *LENGTH, if it fits. +static void append_text(char *text, size_t size, size_t *length, + const char *format, ...) + __attribute__((format(printf, 4, 5))); + +static void append_text(char *text, size_t size, size_t *length, + const char *format, ...) +{ + va_list arguments; + int written; + + if (*length >= size) + return; + va_start(arguments, format); + written = vsnprintf(text + *length, size - *length, format, arguments); + va_end(arguments); + if (written > 0) + *length += (size_t)written; +} + +// Formats the state in the order of the spec's state file table; the keys of +// a phase whose check never ran are absent. static void state_format(const struct time_state *state, char *text, size_t size) { - snprintf(text, size, - "version=1\n" - "generation=%" PRIu64 "\n" - "check_phase=%s\n" - "check_status=%s\n" - "check_cpus=%" PRIu64 "\n" - "check_elapsed_us=%" PRId64 "\n" - "tsc_hz=%" PRIu64 "\n" - "lapic_hz=%" PRIu64 "\n" - "discontinuities=%" PRIu64 "\n" - "last_discontinuity=%s\n" - "last_step_ns=%" PRId64 "\n" - "last_step_realtime_ns=%" PRId64 "\n" - "last_downtime_ns=%" PRIu64 "\n" - "last_downtime_source=%s\n" - "synchronized=%" PRIu64 "\n" - "offset_ns=%" PRId64 "\n" - "uncertainty_ns=%" PRId64 "\n" - "frequency_ppb=%" PRId64 "\n" - "samples=%" PRIu64 "\n" - "rejected_samples=%" PRIu64 "\n" - "last_sample_error=%s\n" - "violations=%" PRIu64 "\n", - state->generation, state->check_phase, state->check_status, - state->check_cpus, state->check_elapsed_us, state->tsc_hz, - state->lapic_hz, state->discontinuities, - state->last_discontinuity, state->last_step_ns, - state->last_step_realtime_ns, state->last_downtime_ns, - state->last_downtime_source, state->synchronized, - state->offset_ns, state->uncertainty_ns, state->frequency_ppb, - state->samples, state->rejected_samples, - state->last_sample_error, state->violations); + size_t length = 0; + + append_text(text, size, &length, "version=1\ngeneration=%" PRIu64 "\n", + state->generation); + for (int phase = 0; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0') + append_text(text, size, &length, "%s_status=%s\n", + k_phase_names[phase], state->checks[phase].status); + } + for (int phase = 0; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0') + append_text(text, size, &length, "%s_cpus=%" PRIu64 "\n", + k_phase_names[phase], state->checks[phase].cpus); + } + for (int phase = 0; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0') + append_text(text, size, &length, "%s_elapsed_us=%" PRId64 "\n", + k_phase_names[phase], state->checks[phase].elapsed_us); + } + for (int phase = PHASE_CAPTURE; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0') + append_text(text, size, &length, "%s_generation=%" PRIu64 "\n", + k_phase_names[phase], state->checks[phase].generation); + } + for (int phase = 0; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0' && + state->checks[phase].failures >= 0) + append_text(text, size, &length, "%s_failures=%" PRId64 "\n", + k_phase_names[phase], state->checks[phase].failures); + } + append_text(text, size, &length, + "tsc_hz=%" PRIu64 "\n" + "lapic_hz=%" PRIu64 "\n" + "discontinuities=%" PRIu64 "\n" + "last_discontinuity=%s\n" + "last_step_ns=%" PRId64 "\n" + "last_step_realtime_ns=%" PRId64 "\n" + "last_downtime_ns=%" PRIu64 "\n" + "last_downtime_source=%s\n" + "synchronized=%" PRIu64 "\n" + "offset_ns=%" PRId64 "\n" + "uncertainty_ns=%" PRId64 "\n" + "frequency_ppb=%" PRId64 "\n" + "samples=%" PRIu64 "\n" + "rejected_samples=%" PRIu64 "\n" + "last_sample_error=%s\n" + "violations=%" PRIu64 "\n", + state->tsc_hz, state->lapic_hz, state->discontinuities, + state->last_discontinuity, state->last_step_ns, + state->last_step_realtime_ns, state->last_downtime_ns, + state->last_downtime_source, state->synchronized, + state->offset_ns, state->uncertainty_ns, state->frequency_ppb, + state->samples, state->rejected_samples, + state->last_sample_error, state->violations); } static int state_load(struct time_state *state) @@ -880,41 +952,43 @@ struct check_record { int64_t elapsed_us; uint64_t tsc_hz; uint64_t lapic_hz; + int failures; }; static void apply_check(struct time_state *state, const void *context) { const struct check_record *record = context; - - snprintf(state->check_phase, sizeof(state->check_phase), "%s", - k_phase_names[record->phase]); - snprintf(state->check_status, sizeof(state->check_status), "%s", - record->status); - state->check_cpus = (uint64_t)record->cpus; - state->check_elapsed_us = record->elapsed_us; + struct check_state *check = &state->checks[record->phase]; + + snprintf(check->status, sizeof(check->status), "%s", record->status); + check->cpus = (uint64_t)record->cpus; + check->elapsed_us = record->elapsed_us; + check->generation = record->phase == PHASE_BOOT ? 0 : state->generation; + check->failures = -1; + // Report-only mode keeps the failures in the state file, as the spec's + // keys reserved for it. + if (g_report_only) { + check->failures = record->failures; + if (record->failures > 0 && strcmp(record->status, "ok") == 0) + snprintf(check->status, sizeof(check->status), "fail"); + } if (record->tsc_hz != 0) state->tsc_hz = record->tsc_hz; if (record->lapic_hz != 0) state->lapic_hz = record->lapic_hz; } -// Records a conformance check in the state file: "pending" while it runs, -// then "ok" (a failed check powers the guest off instead). Rates of 0 keep -// the published ones. In report-only mode the failure count goes beside the -// state, for nvx-time status. +// Records a phase's conformance check in the state file: "pending" while it +// runs, then "ok" (a failed check powers the guest off instead), with the +// generation it ran in. Rates of 0 keep the published ones. Report-only mode +// records "fail" and the failure count of a check that failed. static int record_check(enum phase phase, const char *status, int cpus, int64_t elapsed_us, uint64_t tsc_hz, uint64_t lapic_hz, int failures) { - struct check_record record = {phase, status, cpus, - elapsed_us, tsc_hz, lapic_hz}; - - if (g_report_only) { - char text[32]; + struct check_record record = {phase, status, cpus, elapsed_us, + tsc_hz, lapic_hz, failures}; - snprintf(text, sizeof(text), "failures=%d\n", failures); - (void)replace_file(REPORT_PATH, REPORT_PATH ".tmp", text); - } return state_apply(apply_check, &record); } @@ -3717,17 +3791,30 @@ static int cmd_check(int argc, char **argv) return checks.failures == 0 ? 0 : 1; } -// Waits until no conformance check is pending, for at most STATUS_WAIT_NS, -// and loads the state into STATE. Returns 0, 1 when a check is still -// pending, or -1 when the state is unreadable. -static int wait_for_checks(struct time_state *state) +// Whether a recorded check of STATE is still pending: the boot check only +// with BOOT_ONLY, or any. A state without a boot record parses as a pending +// boot check. +static bool checks_pending(const struct time_state *state, bool boot_only) +{ + for (int phase = 0; phase < (boot_only ? 1 : CHECK_PHASES); phase++) { + if (strcmp(state->checks[phase].status, "pending") == 0) + return true; + } + return false; +} + +// Waits until no recorded check (with BOOT_ONLY, no boot check) is pending, +// for at most STATUS_WAIT_NS, and loads the state into STATE; a missing +// state file counts as a pending boot check. Returns 0, or 1 when a check is +// still pending. +static int wait_for_checks(struct time_state *state, bool boot_only) { int64_t deadline = clock_ns(CLOCK_MONOTONIC) + STATUS_WAIT_NS; for (;;) { if (state_load(state) != 0) - return -1; - if (strcmp(state->check_status, "pending") != 0) + state_init(state); + else if (!checks_pending(state, boot_only)) return 0; if (clock_ns(CLOCK_MONOTONIC) >= deadline) return 1; @@ -3739,17 +3826,15 @@ static int cmd_pre_capture(void) { struct time_state state; struct checks checks; + int64_t elapsed; int64_t start; - int waited; - // The asynchronous boot checks start the daemon that C10 requires. - waited = wait_for_checks(&state); - if (waited != 0) { + // Step 1: the asynchronous boot checks start the daemon that C10 + // requires. + if (wait_for_checks(&state, true) != 0) { memset(&checks, 0, sizeof(checks)); checks.phase = PHASE_CAPTURE; - check_failed(&checks, "C10", "%s", - waited > 0 ? "the boot checks are still pending" - : "the time state is unreadable"); + check_failed(&checks, "C10", "the boot checks are still pending"); return 1; } start = clock_ns(CLOCK_MONOTONIC); @@ -3759,6 +3844,7 @@ static int cmd_pre_capture(void) return 1; } run_capture_checks(&checks); + elapsed = clock_ns(CLOCK_MONOTONIC) - start; if (suppression_save() != 0) { char detail[DETAIL_MAX]; int error = errno; @@ -3770,9 +3856,9 @@ static int cmd_pre_capture(void) : strerror(error)); return 1; } + // Step 1's record travels in the snapshot. if (record_check(PHASE_CAPTURE, "ok", checks.online_count, - (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, 0, 0, - checks.failures) != 0) + elapsed / NSEC_PER_USEC, 0, 0, checks.failures) != 0) check_failed(&checks, "C10", "cannot record the capture check: %s", strerror(errno)); return 0; @@ -3820,13 +3906,15 @@ static void apply_repair(struct time_state *state, const void *context) state->samples++; snprintf(state->last_sample_error, sizeof(state->last_sample_error), "none"); - // The daemon's step 13 runs the restore checks after the acknowledgement. - // Until then, check_elapsed_us is the readiness path up to the clock - // step, at last_step_realtime_ns. - snprintf(state->check_phase, sizeof(state->check_phase), "restore"); - snprintf(state->check_status, sizeof(state->check_status), "pending"); - state->check_cpus = 0; - state->check_elapsed_us = repair->elapsed_us; + // Step 8 marks this restore's check pending, so nvx-time status waits for + // the daemon's step 13. Until then, restore_elapsed_us is the readiness + // path up to the clock step, at last_step_realtime_ns. + snprintf(state->checks[PHASE_RESTORE].status, + sizeof(state->checks[PHASE_RESTORE].status), "pending"); + state->checks[PHASE_RESTORE].cpus = 0; + state->checks[PHASE_RESTORE].elapsed_us = repair->elapsed_us; + state->checks[PHASE_RESTORE].generation = state->generation; + state->checks[PHASE_RESTORE].failures = -1; } static int repair_failed(const char *code, const char *detail) @@ -4023,8 +4111,8 @@ static int cmd_capture(int argc, char **argv) // Restore steps 12 and 13 after the shell's activation and identity work: the // acknowledgement when the packet asked for one, then the daemon runs step 13 // asynchronously. The record tells it which CPUs activation onlined. The -// readiness path started check_elapsed_us before the helper's clock step, at -// last_step_realtime_ns. +// readiness path started restore_elapsed_us before the helper's clock step, +// at last_step_realtime_ns. static int cmd_restore_finish(int argc, char **argv) { bool new_cpus[MAX_CPUS] = {false}; @@ -4068,7 +4156,7 @@ static int cmd_restore_finish(int argc, char **argv) } start = clock_ns(CLOCK_MONOTONIC) - (clock_ns(CLOCK_REALTIME) - state.last_step_realtime_ns) - - state.check_elapsed_us * NSEC_PER_USEC; + state.checks[PHASE_RESTORE].elapsed_us * NSEC_PER_USEC; signaled = signal_restore(daemon_pid(), ack, start); if (signaled == 0) return 0; @@ -4086,62 +4174,67 @@ static int cmd_restore_finish(int argc, char **argv) return 0; } -// Formats nvx-time status's lines from STATE: the marker of the last -// conformance check, with status=pending and no elapsed_us while it runs, -// and the runtime line. Report-only mode adds the check's FAILURES. -static void format_status(const struct time_state *state, int failures, - char *text, size_t size) -{ - bool pending = strcmp(state->check_status, "pending") == 0; - char elapsed[40] = ""; - char suffix[32] = ""; - int length; - - if (!pending) - snprintf(elapsed, sizeof(elapsed), " elapsed_us=%" PRId64, - state->check_elapsed_us); - if (g_report_only) - snprintf(suffix, sizeof(suffix), " failures=%d", failures); - length = snprintf(text, size, - "%s: v=1 phase=%s status=%s cpus=%" PRIu64 - " tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 - " generation=%" PRIu64 "%s%s\n", - abi_prefix(), state->check_phase, state->check_status, - state->check_cpus, state->tsc_hz, state->lapic_hz, - state->generation, elapsed, suffix); - if (length < 0 || (size_t)length >= size) - return; - snprintf(text + length, size - (size_t)length, - "%s: v=1 phase=runtime status=%s generation=%" PRIu64 - " discontinuities=%" PRIu64 " offset_ns=%" PRId64 - " uncertainty_ns=%" PRId64 " rejected_samples=%" PRIu64 - " last_sample_error=%s\n", - abi_prefix(), - state->synchronized != 0 ? "synchronized" : "unsynchronized", - state->generation, state->discontinuities, state->offset_ns, - state->uncertainty_ns, state->rejected_samples, - state->last_sample_error); +// Formats nvx-time status's lines from STATE: one per recorded phase, in the +// order boot, capture, and restore, with status=pending and no elapsed_us +// while its check runs, then the runtime line. Report-only mode appends each +// check's failure count, and a failed check's status is fail. Returns whether +// every phase line reports ok. +static bool format_status(const struct time_state *state, char *text, + size_t size) +{ + size_t length = 0; + bool ok = true; + + text[0] = '\0'; + for (int phase = 0; phase < CHECK_PHASES; phase++) { + const struct check_state *check = &state->checks[phase]; + const char *status = check->status; + char elapsed[40] = ""; + char suffix[40] = ""; + + if (status[0] == '\0') + continue; + if (strcmp(status, "pending") != 0) + snprintf(elapsed, sizeof(elapsed), " elapsed_us=%" PRId64, + check->elapsed_us); + if (g_report_only) + snprintf(suffix, sizeof(suffix), " failures=%" PRId64, + check->failures > 0 ? check->failures : 0); + ok = ok && strcmp(status, "ok") == 0; + append_text(text, size, &length, + "%s: v=1 phase=%s status=%s cpus=%" PRIu64 + " tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 + " generation=%" PRIu64 "%s%s\n", + abi_prefix(), k_phase_names[phase], status, check->cpus, + state->tsc_hz, state->lapic_hz, check->generation, elapsed, + suffix); + } + append_text(text, size, &length, + "%s: v=1 phase=runtime status=%s generation=%" PRIu64 + " discontinuities=%" PRIu64 " offset_ns=%" PRId64 + " uncertainty_ns=%" PRId64 " rejected_samples=%" PRIu64 + " last_sample_error=%s\n", + abi_prefix(), + state->synchronized != 0 ? "synchronized" : "unsynchronized", + state->generation, state->discontinuities, state->offset_ns, + state->uncertainty_ns, state->rejected_samples, + state->last_sample_error); + return ok; } // Prints the time state on demand, after waiting for pending checks: CI and // the matrix driver run it over the console; production paths never do. +// Exits with 0 only when every phase line reports ok. static int cmd_status(void) { struct time_state state; - char text[768]; - int failures = 0; - int waited = wait_for_checks(&state); + char text[1024]; + int waited = wait_for_checks(&state, false); + bool ok = format_status(&state, text, sizeof(text)); - if (waited < 0) { - fprintf(stderr, "nvx-time: the time state is unavailable\n"); - return 1; - } - if (g_report_only && read_text(REPORT_PATH, text, sizeof(text)) >= 0) - (void)sscanf(text, "failures=%d", &failures); - format_status(&state, failures, text, sizeof(text)); fputs(text, stdout); fflush(stdout); - return waited == 0 ? 0 : 1; + return waited == 0 && ok ? 0 : 1; } static int cmd_sample(void) @@ -4645,17 +4738,19 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "state") == 0) return test_state(argc, argv); - if (strcmp(name, "status") == 0 && argc == 3) { + if (strcmp(name, "status") == 0 && argc == 2) { static char text[4096]; struct time_state state; - char out[768]; + char out[1024]; + bool ok; if (test_text(argv[0], text, sizeof(text)) != 0 || state_parse(text, &state) != 0) return 2; g_report_only = strcmp(argv[1], "report-only") == 0; - format_status(&state, atoi(argv[2]), out, sizeof(out)); + ok = format_status(&state, out, sizeof(out)); fputs(out, stdout); + printf("ok=%d\n", ok ? 1 : 0); return 0; } if (strcmp(name, "restore-record") == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index 852714791..e90e718b3 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -553,9 +553,9 @@ def syslog(text_lines: list[str]) -> list[str]: self.assertEqual(syslog([*lines, own_event]), ["ok"]) def test_daemon_lines_start_on_a_fresh_console_line(self): - # The daemon prints the restore marker and runtime events while a - # shell may be in the middle of a line (its prompt, for example). - line = "NVX-TIME-ABI: v=1 phase=restore status=ok\n" + # The daemon prints violation events while a shell may be in the + # middle of a line (its prompt, for example). + line = "NVX-TIME-ABI-VIOLATION: v=1 code=G_RCU_STALL source=watcher\n" self.assertEqual(self.run_test("console", line, "sync"), line) self.assertEqual(self.run_test("console", line, "async"), "\n" + line) @@ -734,8 +734,10 @@ def test_violation_event_format_escaping_and_limit(self): def test_state_file_round_trip(self): # The keys and their order follow the spec's state file table. text = ( - "version=1\ngeneration=2\ncheck_phase=restore\ncheck_status=ok\n" - "check_cpus=4\ncheck_elapsed_us=1830\ntsc_hz=2194843000\n" + "version=1\ngeneration=2\nboot_status=ok\ncapture_status=ok\n" + "restore_status=pending\nboot_cpus=8\ncapture_cpus=8\nrestore_cpus=0\n" + "boot_elapsed_us=2400\ncapture_elapsed_us=310\nrestore_elapsed_us=1830\n" + "capture_generation=1\nrestore_generation=2\ntsc_hz=2194843000\n" "lapic_hz=200000000\ndiscontinuities=3\nlast_discontinuity=step\n" "last_step_ns=-200000000\nlast_step_realtime_ns=1790841600000000000\n" "last_downtime_ns=31000000000\nlast_downtime_source=utc\n" @@ -744,11 +746,22 @@ def test_state_file_round_trip(self): "last_sample_error=none\nviolations=0\n" ) self.assertEqual(self.run_test("state", self.fixture("state.txt", text)), text) - # A fresh state is the pending boot check. + # Report-only mode adds the failure counts, reserved for it, after the + # generations, and a failed check's status is fail. + report = text.replace("restore_status=pending", "restore_status=fail").replace( + "restore_generation=2\n", + "restore_generation=2\nboot_failures=0\ncapture_failures=0\n" + "restore_failures=3\n", + ) + self.assertEqual( + self.run_test("state", self.fixture("state.txt", report)), report + ) + # A fresh state is the pending boot check; the capture and restore keys + # are absent until those checks first run. self.assertTrue( self.run_test("state", self.fixture("state.txt", "version=1\n")).startswith( - "version=1\ngeneration=0\ncheck_phase=boot\ncheck_status=pending\n" - "check_cpus=0\ncheck_elapsed_us=0\ntsc_hz=0\nlapic_hz=0\n" + "version=1\ngeneration=0\nboot_status=pending\nboot_cpus=0\n" + "boot_elapsed_us=0\ntsc_hz=0\nlapic_hz=0\n" ) ) self.assertEqual( @@ -758,45 +771,81 @@ def test_state_file_round_trip(self): "error", ) - def status(self, text: str, mode: str = "enforcing", failures: int = 0) -> str: + def status(self, text: str, mode: str = "enforcing") -> list[str]: return self.run_test( - "status", self.fixture("state.txt", text), mode, str(failures) - ) + "status", self.fixture("state.txt", text), mode + ).splitlines() - def test_status_prints_the_last_check_and_the_runtime_line(self): + def test_status_prints_every_recorded_check_and_the_runtime_line(self): state = ( - "version=1\ngeneration=1\ncheck_phase=restore\ncheck_status=ok\n" - "check_cpus=4\ncheck_elapsed_us=1830\ntsc_hz=2194843000\n" - "lapic_hz=200000000\ndiscontinuities=1\nsynchronized=1\n" + "version=1\ngeneration=2\nboot_status=ok\ncapture_status=ok\n" + "restore_status=ok\nboot_cpus=8\ncapture_cpus=8\nrestore_cpus=4\n" + "boot_elapsed_us=2400\ncapture_elapsed_us=310\nrestore_elapsed_us=1830\n" + "capture_generation=1\nrestore_generation=2\ntsc_hz=2194843000\n" + "lapic_hz=200000000\ndiscontinuities=2\nsynchronized=1\n" "offset_ns=-42\nuncertainty_ns=1800\nrejected_samples=2\n" "last_sample_error=G_SAMPLE_UNCERTAIN\n" ) + rates = "tsc_hz=2194843000 lapic_hz=200000000" + # One line per recorded phase, each with the generation its check ran + # in, then the runtime line; the hook adds whether status exits 0. self.assertEqual( self.status(state), - "NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 tsc_hz=2194843000 " - "lapic_hz=200000000 generation=1 elapsed_us=1830\n" - "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=1 " - "discontinuities=1 offset_ns=-42 uncertainty_ns=1800 " - "rejected_samples=2 last_sample_error=G_SAMPLE_UNCERTAIN\n", - ) - # A check still pending after the wait has no elapsed_us. - pending = state.replace("check_status=ok", "check_status=pending") - pending = pending.replace("synchronized=1", "synchronized=0") - self.assertEqual( - self.status(pending).splitlines(), [ - "NVX-TIME-ABI: v=1 phase=restore status=pending cpus=4 " - "tsc_hz=2194843000 lapic_hz=200000000 generation=1", - "NVX-TIME-ABI: v=1 phase=runtime status=unsynchronized " - "generation=1 discontinuities=1 offset_ns=-42 uncertainty_ns=1800 " + f"NVX-TIME-ABI: v=1 phase=boot status=ok cpus=8 {rates} " + "generation=0 elapsed_us=2400", + f"NVX-TIME-ABI: v=1 phase=capture status=ok cpus=8 {rates} " + "generation=1 elapsed_us=310", + f"NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 {rates} " + "generation=2 elapsed_us=1830", + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=2 " + "discontinuities=2 offset_ns=-42 uncertainty_ns=1800 " "rejected_samples=2 last_sample_error=G_SAMPLE_UNCERTAIN", + "ok=1", + ], + ) + # A check still pending after the wait has no elapsed_us and fails the + # exit status. A cold boot has a boot line only, and a state without a + # boot record counts as a pending boot check. + pending = state.replace("restore_status=ok", "restore_status=pending") + self.assertEqual( + self.status(pending)[2::2], + [ + f"NVX-TIME-ABI: v=1 phase=restore status=pending cpus=4 {rates} " + "generation=2", + "ok=0", ], ) - # Report-only mode marks both lines and counts the check's failures. - report = self.status(state, "report-only", 3).splitlines() - self.assertTrue(report[0].startswith("NVX-TIME-REPORT: v=1 phase=restore ")) - self.assertTrue(report[0].endswith(" elapsed_us=1830 failures=3")) - self.assertTrue(report[1].startswith("NVX-TIME-REPORT: v=1 phase=runtime ")) + boot_pending = ( + "NVX-TIME-ABI: v=1 phase=boot status=pending cpus=0 tsc_hz=0 " + "lapic_hz=0 generation=0" + ) + for text in ("version=1\ngeneration=0\nboot_status=pending\n", "version=1\n"): + lines = self.status(text) + self.assertEqual(lines[0], boot_pending, text) + self.assertTrue(lines[1].startswith("NVX-TIME-ABI: v=1 phase=runtime ")) + self.assertEqual(lines[2:], ["ok=0"], text) + # Report-only mode marks every line and appends each check's failure + # count from the state file, where a failed check's status is fail. + report = self.status( + state.replace("restore_status=ok", "restore_status=fail").replace( + "restore_generation=2\n", + "restore_generation=2\nboot_failures=0\nrestore_failures=3\n", + ), + "report-only", + ) + self.assertEqual( + report[0], + f"NVX-TIME-REPORT: v=1 phase=boot status=ok cpus=8 {rates} " + "generation=0 elapsed_us=2400 failures=0", + ) + self.assertEqual( + report[2], + f"NVX-TIME-REPORT: v=1 phase=restore status=fail cpus=4 {rates} " + "generation=2 elapsed_us=1830 failures=3", + ) + self.assertTrue(report[3].startswith("NVX-TIME-REPORT: v=1 phase=runtime ")) + self.assertEqual(report[4], "ok=0") def restore_record(self, text: str) -> str: return self.run_test( From 8167dcb899fe294ecbf6c286ce8d2ff2935d5b6c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 23:03:19 -0700 Subject: [PATCH 103/188] guest: defer the time ABI checks by 100 ms and report their CPU time Implements spec 9c73a76f, eb00ea34, and 6f44e2ca. Restore step 13 (the restore checks, the RCU release, and C7) starts 100 ms after the acknowledgement at SCHED_IDLE, and continues at normal priority 100 ms after it starts. The daemon keeps the restore pending and polls until it is due. The asynchronous boot checks likewise start 100 ms after the time ABI boot step at SCHED_IDLE, with the promoter firing 100 ms later. The boot step is the one point that every guest mode shares, and init runs it shortly before shell-ready. Neither elapsed_us counts the delay: the restore's runs from when step 13 was due, so a wait for a CPU at SCHED_IDLE still counts. Every phase now records cpu_us, the CLOCK_PROCESS_CPUTIME_ID time of the work that elapsed_us covers, summed over the processes that run it: - boot: C12 and the checker, which becomes the daemon; - capture: the step 1 checks; - restore: the readiness path (the helper, plus restore-finish on a tiered restore), the daemon's preparation, and the forked worker's step 11 checks. The restored TSC advances by the downtime, and the first scheduler update after the restore charges that to the helper, which ran the capture. So the helper reads its CPU baseline right after the restore. The signal to the daemon carries both readiness times in one 64-bit sigqueue() value. The state file gains boot_cpu_us, capture_cpu_us, and restore_cpu_us, and nvx-time status prints cpu_us after elapsed_us on every phase line that is not pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-time.c | 349 +++++++++++++++++++++++++++---------- scripts/test_guest_time.py | 37 ++-- 2 files changed, 282 insertions(+), 104 deletions(-) diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index c627c2ff8..a22bf0653 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -124,9 +124,13 @@ #define SLOW_TIME_CONSTANT 6 #define TIMER_LIST_WAIT_NS (200 * NSEC_PER_MSEC) #define DEFAULT_STALL_TIMEOUT_S 21 -// The asynchronous boot checks and restore step 13 run at SCHED_IDLE for -// 100 ms, then at normal priority, which bounds the fail-fast window under a -// CPU-bound workload. +// The asynchronous boot checks start 100 ms after the time ABI boot step, +// which init runs shortly before shell-ready, and restore step 13 100 ms +// after the acknowledgement: earlier, they contend with the readiness path +// while the VMM still demand-faults guest memory. Both start at SCHED_IDLE +// and continue at normal priority 100 ms after they start, which bounds the +// fail-fast window under a CPU-bound workload. +#define DEFERRED_START_NS (100 * NSEC_PER_MSEC) #define IDLE_BOUND_NS (100 * NSEC_PER_MSEC) // nvx-time status and pre-capture wait at most 30 s for pending checks. #define STATUS_WAIT_NS (30 * NSEC_PER_SEC) @@ -180,6 +184,13 @@ static int64_t clock_ns(clockid_t clock) return (int64_t)now.tv_sec * NSEC_PER_SEC + now.tv_nsec; } +// The CPU time of this process, every thread's, exited threads' included: +// what the spec's cpu_us sums. +static int64_t cpu_time_ns(void) +{ + return clock_ns(CLOCK_PROCESS_CPUTIME_ID); +} + static int write_all(int fd, const void *data, size_t size) { const char *bytes = data; @@ -654,13 +665,15 @@ static void fail_fatal(int status, const char *code, const char *source, // A conformance check's record: STATUS is empty until the phase first runs, // then "pending" while its check runs, then "ok" (a failed check powers the -// guest off; report-only mode records "fail" instead). GENERATION is g when -// the check ran; the boot check's is 0. FAILURES is the check's failure -// count, kept in report-only mode only, and -1 otherwise. +// guest off; report-only mode records "fail" instead). ELAPSED_US is the +// check's wall time and CPU_US its CPU time over every process that ran it. +// GENERATION is g when the check ran; the boot check's is 0. FAILURES is the +// check's failure count, kept in report-only mode only, and -1 otherwise. struct check_state { char status[16]; uint64_t cpus; int64_t elapsed_us; + int64_t cpu_us; uint64_t generation; int64_t failures; }; @@ -730,6 +743,9 @@ static int state_parse(char *text, struct time_state *state) {"boot_elapsed_us", NULL, &boot->elapsed_us}, {"capture_elapsed_us", NULL, &capture->elapsed_us}, {"restore_elapsed_us", NULL, &restore->elapsed_us}, + {"boot_cpu_us", NULL, &boot->cpu_us}, + {"capture_cpu_us", NULL, &capture->cpu_us}, + {"restore_cpu_us", NULL, &restore->cpu_us}, {"capture_generation", &capture->generation, NULL}, {"restore_generation", &restore->generation, NULL}, {"boot_failures", NULL, &boot->failures}, @@ -840,6 +856,11 @@ static void state_format(const struct time_state *state, char *text, append_text(text, size, &length, "%s_elapsed_us=%" PRId64 "\n", k_phase_names[phase], state->checks[phase].elapsed_us); } + for (int phase = 0; phase < CHECK_PHASES; phase++) { + if (state->checks[phase].status[0] != '\0') + append_text(text, size, &length, "%s_cpu_us=%" PRId64 "\n", + k_phase_names[phase], state->checks[phase].cpu_us); + } for (int phase = PHASE_CAPTURE; phase < CHECK_PHASES; phase++) { if (state->checks[phase].status[0] != '\0') append_text(text, size, &length, "%s_generation=%" PRIu64 "\n", @@ -950,6 +971,7 @@ struct check_record { const char *status; int cpus; int64_t elapsed_us; + int64_t cpu_us; uint64_t tsc_hz; uint64_t lapic_hz; int failures; @@ -963,6 +985,7 @@ static void apply_check(struct time_state *state, const void *context) snprintf(check->status, sizeof(check->status), "%s", record->status); check->cpus = (uint64_t)record->cpus; check->elapsed_us = record->elapsed_us; + check->cpu_us = record->cpu_us; check->generation = record->phase == PHASE_BOOT ? 0 : state->generation; check->failures = -1; // Report-only mode keeps the failures in the state file, as the spec's @@ -980,14 +1003,15 @@ static void apply_check(struct time_state *state, const void *context) // Records a phase's conformance check in the state file: "pending" while it // runs, then "ok" (a failed check powers the guest off instead), with the -// generation it ran in. Rates of 0 keep the published ones. Report-only mode -// records "fail" and the failure count of a check that failed. +// generation it ran in and its wall and CPU times. Rates of 0 keep the +// published ones. Report-only mode records "fail" and the failure count of a +// check that failed. static int record_check(enum phase phase, const char *status, int cpus, - int64_t elapsed_us, uint64_t tsc_hz, uint64_t lapic_hz, - int failures) + int64_t elapsed_us, int64_t cpu_us, uint64_t tsc_hz, + uint64_t lapic_hz, int failures) { - struct check_record record = {phase, status, cpus, elapsed_us, - tsc_hz, lapic_hz, failures}; + struct check_record record = {phase, status, cpus, elapsed_us, + cpu_us, tsc_hz, lapic_hz, failures}; return state_apply(apply_check, &record); } @@ -2641,25 +2665,50 @@ static void promoter_stop(struct promoter *promoter) pthread_join(promoter->thread, NULL); } +// The readiness path's wall and CPU times travel to the daemon in one +// sigqueue() value, 64 bits on x86-64: each in microseconds, clamped to 32 +// bits, the wall time in the low half. +_Static_assert(sizeof(void *) == sizeof(uint64_t), "sigval holds 64 bits"); + +static uint64_t clamp_u32(int64_t value) +{ + return value < 0 ? 0 : value > UINT32_MAX ? UINT32_MAX : (uint64_t)value; +} + +static uint64_t pack_readiness(int64_t elapsed_us, int64_t cpu_us) +{ + return clamp_u32(cpu_us) << 32 | clamp_u32(elapsed_us); +} + +static void unpack_readiness(uint64_t value, int64_t *elapsed_us, + int64_t *cpu_us) +{ + *elapsed_us = (int64_t)(value & UINT32_MAX); + *cpu_us = (int64_t)(value >> 32); +} + // Restore steps 12 and 13 on the readiness path: the acknowledgement when the -// host gated its input, then hand step 13 to DAEMON, with the readiness -// path's duration since START_NS (CLOCK_MONOTONIC), in microseconds, in the -// signal. Returns -1, without acknowledging, when no daemon runs; 1 when the -// daemon vanished after the acknowledgement. The caller has enabled the ports -// when ACK is set. -static int signal_restore(pid_t daemon, bool ack, int64_t start_ns) +// host gated its input, then hand step 13 to DAEMON. The signal carries the +// readiness path's wall time since START_NS (CLOCK_MONOTONIC) and its CPU +// time since CPU_START_NS (this process's CPU clock, less any earlier +// process's share), in microseconds, both read after the acknowledgement so +// that neither read delays it. Returns -1, without acknowledging, when no +// daemon runs; 1 when the daemon vanished after the acknowledgement. The +// caller has enabled the ports when ACK is set. +static int signal_restore(pid_t daemon, bool ack, int64_t start_ns, + int64_t cpu_start_ns) { union sigval value; int64_t elapsed_us; + int64_t cpu_us; if (daemon <= 0 || kill(daemon, 0) != 0) return -1; if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; - value.sival_int = (int)(elapsed_us < 0 ? 0 - : elapsed_us > INT_MAX ? INT_MAX - : elapsed_us); + cpu_us = (cpu_time_ns() - cpu_start_ns) / NSEC_PER_USEC; + value.sival_ptr = (void *)(uintptr_t)pack_readiness(elapsed_us, cpu_us); return sigqueue(daemon, SIGUSR1, value) == 0 ? 0 : 1; } @@ -3032,7 +3081,9 @@ struct daemon { int64_t worker_deadline_ns; bool suppression_released; bool restore_pending; + int64_t restore_due_ns; int64_t readiness_us; + int64_t readiness_cpu_us; }; static void watcher_consume(void *context, const char *message) @@ -3244,30 +3295,41 @@ static int parse_restore_record(const char *text, unsigned long long *generation return parse_new_cpus(cpus, new_cpus); } +// The times that restore_work records. The wall time is READINESS_US, the +// readiness path's, plus the checks' since START_NS, when they were due. The +// CPU time is PRIOR_CPU_NS, the readiness path's and the daemon's share, plus +// this process's since CPU_BASE_NS. +struct restore_timing { + int64_t start_ns; + int64_t readiness_us; + int64_t prior_cpu_ns; + int64_t cpu_base_ns; +}; + // Restore step 13, fail-fast: the step 11 restore checks, the grace-period // release, the saved watchdog settings, the deferred C7, and the restore -// check's record. A child of the daemon runs it at SCHED_IDLE, and the daemon -// raises it to normal priority after 100 ms and releases the suppression -// itself at the stall timeout; restore-finish runs it inline, with its own -// deadline (WITH_DEADLINE), in report-only mode when no daemon runs. The -// recorded duration is READINESS_US, the readiness path's, plus the checks', -// without the grace period wait or C7. Returns 0, or 1 when a report-only -// release failed. +// check's record. A child of the daemon runs it at SCHED_IDLE from 100 ms +// after the acknowledgement; the daemon raises it to normal priority 100 ms +// later and releases the suppression itself at the stall timeout. +// restore-finish runs it inline, with its own deadline (WITH_DEADLINE), in +// report-only mode when no daemon runs. Neither recorded time counts the +// grace period wait or C7. Returns 0, or 1 when a report-only release failed. static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, - int64_t readiness_us) + const struct restore_timing *timing) { - int64_t start = clock_ns(CLOCK_MONOTONIC); int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; char detail[DETAIL_MAX]; struct checks checks; int64_t elapsed; + int64_t cpu; bool released; if (checks_init(&checks, PHASE_RESTORE) != 0) check_failed(&checks, "C6", "cannot read the CPU sets"); else run_restore_checks(&checks, new_cpus, new_count); - elapsed = clock_ns(CLOCK_MONOTONIC) - start; + elapsed = clock_ns(CLOCK_MONOTONIC) - timing->start_ns; + cpu = timing->prior_cpu_ns + cpu_time_ns() - timing->cpu_base_ns; if (with_deadline) { (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); released = release_with_deadline(timeout_s); @@ -3282,19 +3344,22 @@ static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, PHASE_RESTORE, "%s", detail); check_timer_list(&checks, TIMER_LIST_WAIT_NS); if (record_check(PHASE_RESTORE, "ok", checks.online_count, - readiness_us + elapsed / NSEC_PER_USEC, 0, 0, - checks.failures) != 0) + timing->readiness_us + elapsed / NSEC_PER_USEC, + cpu / NSEC_PER_USEC, 0, 0, checks.failures) != 0) check_failed(&checks, "C10", "cannot record the restore check: %s", strerror(errno)); return released ? 0 : 1; } -// Starts restore step 13 after the acknowledgement, in a child at -// SCHED_IDLE, so the watcher and the discipline keep normal priority; the -// fork itself also runs at SCHED_IDLE. READINESS_US comes with the signal. -static void start_restore(struct daemon *daemon, int64_t readiness_us) +// Starts restore step 13 when it is due, 100 ms after the acknowledgement, in +// a child at SCHED_IDLE, so the watcher and the discipline keep normal +// priority; the fork itself also runs at SCHED_IDLE. The readiness path's +// times came with the signal. +static void start_restore(struct daemon *daemon) { + int64_t cpu_start = cpu_time_ns(); bool new_cpus[MAX_CPUS] = {false}; + struct restore_timing timing; struct time_state state; unsigned long long generation; int64_t timeout_s = DEFAULT_STALL_TIMEOUT_S; @@ -3323,12 +3388,17 @@ static void start_restore(struct daemon *daemon, int64_t readiness_us) } } (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); - // The repair paired the clock with the packet's UTC. - daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); + // The worker's wall time runs from when step 13 was due. Its CPU time + // runs from the fork, and the daemon's share is this preparation. + timing.start_ns = daemon->restore_due_ns; + timing.readiness_us = daemon->readiness_us; + timing.cpu_base_ns = 0; set_idle_priority(0, true); + timing.prior_cpu_ns = daemon->readiness_cpu_us * NSEC_PER_USEC + + cpu_time_ns() - cpu_start; pid = fork(); if (pid == 0) - _exit(restore_work(new_cpus, new_count, false, readiness_us)); + _exit(restore_work(new_cpus, new_count, false, &timing)); set_idle_priority(0, false); if (pid < 0) { fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", @@ -3337,7 +3407,7 @@ static void start_restore(struct daemon *daemon, int64_t readiness_us) return; } daemon->worker_pid = pid; - daemon->worker_promote_ns = clock_ns(CLOCK_MONOTONIC) + IDLE_BOUND_NS; + daemon->worker_promote_ns = daemon->restore_due_ns + IDLE_BOUND_NS; daemon->worker_deadline_ns = clock_ns(CLOCK_MONOTONIC) + timeout_s * NSEC_PER_SEC; daemon->suppression_released = false; @@ -3387,20 +3457,24 @@ static void handle_signals(struct daemon *daemon) while (read(daemon->signals, &info, sizeof(info)) == sizeof(info)) { if (info.ssi_signo == SIGTERM) _exit(0); - // sigqueue() carries the readiness path's duration in microseconds. + // sigqueue() carries the readiness path's wall and CPU times. Step 13 + // is due 100 ms later; the repair paired the clock with the packet's + // UTC now. if (info.ssi_signo == SIGUSR1) { daemon->restore_pending = true; - daemon->readiness_us = - info.ssi_code == SI_QUEUE ? info.ssi_int : 0; + daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); + daemon->restore_due_ns = + daemon->last_accepted_ns + DEFERRED_START_NS; + daemon->readiness_us = 0; + daemon->readiness_cpu_us = 0; + if (info.ssi_code == SI_QUEUE) + unpack_readiness(info.ssi_ptr, &daemon->readiness_us, + &daemon->readiness_cpu_us); } } - // The previous restore's worker can end just before a capture, and the - // signalfd returns SIGUSR1 before SIGCHLD: reap before starting. + // A worker that ended, if any: the daemon loop starts a due restore only + // once the previous worker is reaped. reap_children(daemon); - if (daemon->restore_pending && daemon->worker_pid == 0) { - daemon->restore_pending = false; - start_restore(daemon, daemon->readiness_us); - } } // Sets up the daemon's signal and timer descriptors and its OOM exemption. @@ -3443,6 +3517,16 @@ static int poll_timeout_until(int64_t deadline_ns) : (int)((remaining + NSEC_PER_MSEC - 1) / NSEC_PER_MSEC); } +// Returns the earlier of two poll timeouts, where -1 is none. +static int earlier_timeout(int first, int second) +{ + if (first < 0) + return second; + if (second < 0) + return first; + return first < second ? first : second; +} + static void daemon_loop(struct daemon *daemon) { for (;;) { @@ -3451,13 +3535,14 @@ static void daemon_loop(struct daemon *daemon) {daemon->timer, POLLIN, 0}}; bool waiting = daemon->worker_pid > 0 && !daemon->suppression_released; bool idle = daemon->worker_pid > 0 && daemon->worker_promote_ns != 0; - int timeout = poll_timeout_until(idle ? daemon->worker_promote_ns : 0); - int deadline = - poll_timeout_until(waiting ? daemon->worker_deadline_ns : 0); + bool due = daemon->restore_pending && daemon->worker_pid == 0; + int timeout = earlier_timeout( + earlier_timeout( + poll_timeout_until(idle ? daemon->worker_promote_ns : 0), + poll_timeout_until(waiting ? daemon->worker_deadline_ns : 0)), + poll_timeout_until(due ? daemon->restore_due_ns : 0)); int64_t now; - if (timeout < 0 || (deadline >= 0 && deadline < timeout)) - timeout = deadline; if (poll(fds, ARRAY_SIZE(fds), timeout) < 0 && errno != EINTR) { sleep(1); continue; @@ -3476,7 +3561,14 @@ static void daemon_loop(struct daemon *daemon) arm_timer(daemon); } now = clock_ns(CLOCK_MONOTONIC); - // Step 13 continues at normal priority 100 ms after the ack. + // Step 13 starts 100 ms after the ack, and continues at normal + // priority 100 ms after it starts. + if (daemon->restore_pending && daemon->worker_pid == 0 && + now >= daemon->restore_due_ns) { + daemon->restore_pending = false; + start_restore(daemon); + now = clock_ns(CLOCK_MONOTONIC); + } if (daemon->worker_pid > 0 && daemon->worker_promote_ns != 0 && now >= daemon->worker_promote_ns) { daemon->worker_promote_ns = 0; @@ -3588,14 +3680,31 @@ static void boot_foreground(struct checks *checks, void *context) check_debug_watchdogs(checks); } -// The asynchronous boot checks: every boot check but C12, after shell-ready, -// fail-fast, at SCHED_IDLE for at most IDLE_BOUND_NS. When they pass, the -// checker becomes the daemon and records the boot check, whose duration adds -// C12's (C12_NS) and the daemon's start. It never returns. +// Sleeps until DEADLINE_NS on CLOCK_MONOTONIC. +static void sleep_until(int64_t deadline_ns) +{ + struct timespec when = {(time_t)(deadline_ns / NSEC_PER_SEC), + (long)(deadline_ns % NSEC_PER_SEC)}; + + while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &when, NULL) == + EINTR) + ; +} + +// The asynchronous boot checks: every boot check but C12, fail-fast, from +// 100 ms after the boot step at SCHED_IDLE, and at normal priority from +// IDLE_BOUND_NS after they start. When they pass, the checker becomes the +// daemon and records the boot check, whose wall and CPU times add C12's +// (C12_NS and C12_CPU_NS) and the daemon's start, but not the delay. It never +// returns. static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, - int c12_failures) + int64_t c12_cpu_ns, int c12_failures) { - int64_t start = clock_ns(CLOCK_MONOTONIC); + // The 100 ms count from this fork in the time ABI boot step, the one + // point that every guest mode shares: init runs it before the rest of its + // setup, so the checks start slightly less than 100 ms after shell-ready. + int64_t start = clock_ns(CLOCK_MONOTONIC) + DEFERRED_START_NS; + int64_t cpu_start = cpu_time_ns(); int null = open("/dev/null", O_RDWR | O_CLOEXEC); struct boot_log_job log_job; struct promoter promoter; @@ -3619,6 +3728,7 @@ static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, _exit(1); promoting = promoter_start(&promoter, start); set_idle_priority(0, true); + sleep_until(start); if (checks_init(&checks, PHASE_BOOT) != 0) { check_failed(&checks, "C1", "cannot read the CPU sets"); _exit(1); @@ -3668,6 +3778,7 @@ static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, if (record_check(PHASE_BOOT, "ok", checks.online_count, (c12_ns + clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, + (c12_cpu_ns + cpu_time_ns() - cpu_start) / NSEC_PER_USEC, checks.tsc_hz, checks.lapic_hz, checks.failures) != 0) check_failed(&checks, "C10", "cannot publish the time state: %s", strerror(errno)); @@ -3683,9 +3794,11 @@ static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, static int cmd_boot(void) { int64_t start = clock_ns(CLOCK_MONOTONIC); + int64_t cpu_start = cpu_time_ns(); struct boot_sample sample = {0, 0, 0}; struct time_state state; struct checks checks; + int64_t c12_cpu; pid_t pid; g_generation = 0; @@ -3706,9 +3819,11 @@ static int cmd_boot(void) return 1; } fflush(NULL); + // The child cannot read this process's CPU clock. + c12_cpu = cpu_time_ns() - cpu_start; pid = fork(); if (pid == 0) - boot_checker(&sample, clock_ns(CLOCK_MONOTONIC) - start, + boot_checker(&sample, clock_ns(CLOCK_MONOTONIC) - start, c12_cpu, checks.failures); if (pid < 0) { check_failed(&checks, "C10", "cannot start the boot checks: %s", @@ -3753,6 +3868,7 @@ static void run_restore_checks(struct checks *checks, const bool *new_cpus, static int cmd_check(int argc, char **argv) { int64_t start = clock_ns(CLOCK_MONOTONIC); + int64_t cpu_start = cpu_time_ns(); bool new_cpus[MAX_CPUS] = {false}; int new_count = 0; enum phase phase = PHASE_RUNTIME; @@ -3786,8 +3902,9 @@ static int cmd_check(int argc, char **argv) else run_restore_checks(&checks, new_cpus, new_count); (void)record_check(phase, "ok", checks.online_count, - (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, 0, - 0, checks.failures); + (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC, + (cpu_time_ns() - cpu_start) / NSEC_PER_USEC, 0, 0, + checks.failures); return checks.failures == 0 ? 0 : 1; } @@ -3826,8 +3943,10 @@ static int cmd_pre_capture(void) { struct time_state state; struct checks checks; + int64_t cpu_start; int64_t elapsed; int64_t start; + int64_t cpu; // Step 1: the asynchronous boot checks start the daemon that C10 // requires. @@ -3838,6 +3957,7 @@ static int cmd_pre_capture(void) return 1; } start = clock_ns(CLOCK_MONOTONIC); + cpu_start = cpu_time_ns(); g_generation = (uint32_t)state.generation; if (checks_init(&checks, PHASE_CAPTURE) != 0) { check_failed(&checks, "C6", "cannot read the CPU sets"); @@ -3845,6 +3965,7 @@ static int cmd_pre_capture(void) } run_capture_checks(&checks); elapsed = clock_ns(CLOCK_MONOTONIC) - start; + cpu = cpu_time_ns() - cpu_start; if (suppression_save() != 0) { char detail[DETAIL_MAX]; int error = errno; @@ -3858,7 +3979,8 @@ static int cmd_pre_capture(void) } // Step 1's record travels in the snapshot. if (record_check(PHASE_CAPTURE, "ok", checks.online_count, - elapsed / NSEC_PER_USEC, 0, 0, checks.failures) != 0) + elapsed / NSEC_PER_USEC, cpu / NSEC_PER_USEC, 0, 0, + checks.failures) != 0) check_failed(&checks, "C10", "cannot record the capture check: %s", strerror(errno)); return 0; @@ -3882,6 +4004,7 @@ struct repair_record { int64_t step_realtime_ns; int64_t frequency; int64_t elapsed_us; + int64_t cpu_us; }; static void apply_repair(struct time_state *state, const void *context) @@ -3908,11 +4031,14 @@ static void apply_repair(struct time_state *state, const void *context) "none"); // Step 8 marks this restore's check pending, so nvx-time status waits for // the daemon's step 13. Until then, restore_elapsed_us is the readiness - // path up to the clock step, at last_step_realtime_ns. + // path up to the clock step, at last_step_realtime_ns, and + // restore_cpu_us the helper's CPU time, which restore-finish adds to its + // own (0 when the helper finishes the restore itself). snprintf(state->checks[PHASE_RESTORE].status, sizeof(state->checks[PHASE_RESTORE].status), "pending"); state->checks[PHASE_RESTORE].cpus = 0; state->checks[PHASE_RESTORE].elapsed_us = repair->elapsed_us; + state->checks[PHASE_RESTORE].cpu_us = repair->cpu_us; state->checks[PHASE_RESTORE].generation = state->generation; state->checks[PHASE_RESTORE].failures = -1; } @@ -3926,14 +4052,14 @@ static int repair_failed(const char *code, const char *detail) // Restore steps 6 to 8: read the packet with four-byte reads, validate it, // and set the wall clock from its bracketed UTC, then write the entropy to // ENTROPY_FD, the caller's file at ENTROPY_PATH. START_NS is when the capture -// request returned. The caller holds the portb lock. With FINISH (untiered) -// and a DAEMON, a restore with no processors or memory to activate needs no -// shell work before the acknowledgement, so steps 12 and 13 start here too -// and no second helper process starts; the metadata then begins with 2 -// instead of 1. +// request returned, and CPU_START_NS this process's CPU clock right after the +// restore. The caller holds the portb lock. With FINISH (untiered) and a +// DAEMON, a restore with no processors or memory to activate needs no shell +// work before the acknowledgement, so steps 12 and 13 start here too and no +// second helper process starts; the metadata then begins with 2 instead of 1. static int repair_restore(const uint8_t *previous_id, const char *entropy_path, - int entropy_fd, int64_t start_ns, bool finish, - pid_t daemon) + int entropy_fd, int64_t start_ns, + int64_t cpu_start_ns, bool finish, pid_t daemon) { static uint8_t body[PACKET_MAX_SIZE]; static struct restore_packet packet; @@ -3946,6 +4072,7 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, int64_t t1; int64_t sample_theta; int64_t sample_epsilon; + bool finishing; int finished = -1; t0 = clock_ns(CLOCK_REALTIME); @@ -4001,6 +4128,15 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, repair.step_realtime_ns = clock_ns(CLOCK_REALTIME); repair.elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; repair.packet = &packet; + // A plain untiered restore needs no shell work: this helper also + // acknowledges and hands step 13 to the daemon, and reads its CPU clock + // again only after the acknowledgement. Otherwise restore-finish adds + // this helper's CPU time to its own. + finishing = finish && daemon > 0 && packet.online_vp_count == 0 && + packet.range_count == 0 && + (packet.flags & PACKET_MEMORY_TARGET) == 0; + repair.cpu_us = + finishing ? 0 : (cpu_time_ns() - cpu_start_ns) / NSEC_PER_USEC; // The entropy goes to the caller in every case, an untiered restore's // too; the file already exists, empty, so only its bytes are written. if (pwrite(entropy_fd, packet.entropy, PACKET_ENTROPY_SIZE, 0) != @@ -4016,12 +4152,10 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, return repair_failed("G_REPAIR_CLOCK", detail); } format_hex(packet.entropy, GENERATION_ID_SIZE, id); - // A plain untiered restore needs no shell work: this helper also - // acknowledges and hands step 13 to the daemon. - if (finish && daemon > 0 && packet.online_vp_count == 0 && - packet.range_count == 0 && (packet.flags & PACKET_MEMORY_TARGET) == 0) { - finished = signal_restore( - daemon, (packet.flags & PACKET_ACK_REQUIRED) != 0, start_ns); + if (finishing) { + finished = signal_restore(daemon, + (packet.flags & PACKET_ACK_REQUIRED) != 0, + start_ns, cpu_start_ns); if (finished > 0) fail_fatal(STATUS_CONFORMANCE, "G_CONFORMANCE_C10", "conformance", PHASE_RESTORE, "the time daemon is not running"); @@ -4048,6 +4182,7 @@ static int cmd_capture(int argc, char **argv) char *end; long request; int64_t start; + int64_t cpu_start; unsigned status; pid_t daemon = -1; int entropy_fd; @@ -4102,8 +4237,13 @@ static int cmd_capture(int argc, char **argv) puts("0"); return 0; } - result = repair_restore(previous_id, argv[2], entropy_fd, start, argc == 4, - daemon); + // The restored TSC advanced by the downtime, and the first scheduler + // update after the restore charges that to this process, which ran the + // capture. Reading the CPU clock makes that update, so the baseline + // includes it. + cpu_start = cpu_time_ns(); + result = repair_restore(previous_id, argv[2], entropy_fd, start, cpu_start, + argc == 4, daemon); close(lock); return result; } @@ -4112,14 +4252,16 @@ static int cmd_capture(int argc, char **argv) // acknowledgement when the packet asked for one, then the daemon runs step 13 // asynchronously. The record tells it which CPUs activation onlined. The // readiness path started restore_elapsed_us before the helper's clock step, -// at last_step_realtime_ns. +// at last_step_realtime_ns, and the helper spent restore_cpu_us of CPU time. static int cmd_restore_finish(int argc, char **argv) { bool new_cpus[MAX_CPUS] = {false}; const char *new_list = "none"; + struct restore_timing timing; struct time_state state; struct checks checks; char text[256]; + int64_t cpu_start; int64_t start; bool ack = false; int new_count = 0; @@ -4157,7 +4299,10 @@ static int cmd_restore_finish(int argc, char **argv) start = clock_ns(CLOCK_MONOTONIC) - (clock_ns(CLOCK_REALTIME) - state.last_step_realtime_ns) - state.checks[PHASE_RESTORE].elapsed_us * NSEC_PER_USEC; - signaled = signal_restore(daemon_pid(), ack, start); + // This process runs only for the restore, so all of its CPU time counts, + // plus the helper's. + cpu_start = -state.checks[PHASE_RESTORE].cpu_us * NSEC_PER_USEC; + signaled = signal_restore(daemon_pid(), ack, start, cpu_start); if (signaled == 0) return 0; if (signaled > 0 || !g_report_only) { @@ -4169,16 +4314,19 @@ static int cmd_restore_finish(int argc, char **argv) // Report-only runs may lack the daemon; finish the restore here. if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); - (void)restore_work(new_cpus, new_count, true, - (clock_ns(CLOCK_MONOTONIC) - start) / NSEC_PER_USEC); + timing.start_ns = clock_ns(CLOCK_MONOTONIC); + timing.readiness_us = (timing.start_ns - start) / NSEC_PER_USEC; + timing.cpu_base_ns = cpu_time_ns(); + timing.prior_cpu_ns = timing.cpu_base_ns - cpu_start; + (void)restore_work(new_cpus, new_count, true, &timing); return 0; } // Formats nvx-time status's lines from STATE: one per recorded phase, in the -// order boot, capture, and restore, with status=pending and no elapsed_us -// while its check runs, then the runtime line. Report-only mode appends each -// check's failure count, and a failed check's status is fail. Returns whether -// every phase line reports ok. +// order boot, capture, and restore, with status=pending and neither +// elapsed_us nor cpu_us while its check runs, then the runtime line. +// Report-only mode appends each check's failure count, and a failed check's +// status is fail. Returns whether every phase line reports ok. static bool format_status(const struct time_state *state, char *text, size_t size) { @@ -4189,14 +4337,15 @@ static bool format_status(const struct time_state *state, char *text, for (int phase = 0; phase < CHECK_PHASES; phase++) { const struct check_state *check = &state->checks[phase]; const char *status = check->status; - char elapsed[40] = ""; + char times[64] = ""; char suffix[40] = ""; if (status[0] == '\0') continue; if (strcmp(status, "pending") != 0) - snprintf(elapsed, sizeof(elapsed), " elapsed_us=%" PRId64, - check->elapsed_us); + snprintf(times, sizeof(times), + " elapsed_us=%" PRId64 " cpu_us=%" PRId64, + check->elapsed_us, check->cpu_us); if (g_report_only) snprintf(suffix, sizeof(suffix), " failures=%" PRId64, check->failures > 0 ? check->failures : 0); @@ -4206,7 +4355,7 @@ static bool format_status(const struct time_state *state, char *text, " tsc_hz=%" PRIu64 " lapic_hz=%" PRIu64 " generation=%" PRIu64 "%s%s\n", abi_prefix(), k_phase_names[phase], status, check->cpus, - state->tsc_hz, state->lapic_hz, check->generation, elapsed, + state->tsc_hz, state->lapic_hz, check->generation, times, suffix); } append_text(text, size, &length, @@ -4755,6 +4904,18 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "restore-record") == 0) return test_restore_record(argc, argv); + if (strcmp(name, "readiness") == 0 && argc == 2) { + uint64_t value = pack_readiness(strtoll(argv[0], NULL, 10), + strtoll(argv[1], NULL, 10)); + int64_t elapsed_us; + int64_t cpu_us; + + unpack_readiness(value, &elapsed_us, &cpu_us); + printf("value=0x%016" PRIx64 " elapsed_us=%" PRId64 " cpu_us=%" PRId64 + "\n", + value, elapsed_us, cpu_us); + return 0; + } if (strcmp(name, "parallel") == 0) return test_parallel(argc, argv); if (strcmp(name, "promote") == 0 && argc == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index e90e718b3..bdb84c9b0 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -737,6 +737,7 @@ def test_state_file_round_trip(self): "version=1\ngeneration=2\nboot_status=ok\ncapture_status=ok\n" "restore_status=pending\nboot_cpus=8\ncapture_cpus=8\nrestore_cpus=0\n" "boot_elapsed_us=2400\ncapture_elapsed_us=310\nrestore_elapsed_us=1830\n" + "boot_cpu_us=1900\ncapture_cpu_us=290\nrestore_cpu_us=640\n" "capture_generation=1\nrestore_generation=2\ntsc_hz=2194843000\n" "lapic_hz=200000000\ndiscontinuities=3\nlast_discontinuity=step\n" "last_step_ns=-200000000\nlast_step_realtime_ns=1790841600000000000\n" @@ -761,7 +762,7 @@ def test_state_file_round_trip(self): self.assertTrue( self.run_test("state", self.fixture("state.txt", "version=1\n")).startswith( "version=1\ngeneration=0\nboot_status=pending\nboot_cpus=0\n" - "boot_elapsed_us=0\ntsc_hz=0\nlapic_hz=0\n" + "boot_elapsed_us=0\nboot_cpu_us=0\ntsc_hz=0\nlapic_hz=0\n" ) ) self.assertEqual( @@ -781,6 +782,7 @@ def test_status_prints_every_recorded_check_and_the_runtime_line(self): "version=1\ngeneration=2\nboot_status=ok\ncapture_status=ok\n" "restore_status=ok\nboot_cpus=8\ncapture_cpus=8\nrestore_cpus=4\n" "boot_elapsed_us=2400\ncapture_elapsed_us=310\nrestore_elapsed_us=1830\n" + "boot_cpu_us=1900\ncapture_cpu_us=290\nrestore_cpu_us=640\n" "capture_generation=1\nrestore_generation=2\ntsc_hz=2194843000\n" "lapic_hz=200000000\ndiscontinuities=2\nsynchronized=1\n" "offset_ns=-42\nuncertainty_ns=1800\nrejected_samples=2\n" @@ -788,25 +790,26 @@ def test_status_prints_every_recorded_check_and_the_runtime_line(self): ) rates = "tsc_hz=2194843000 lapic_hz=200000000" # One line per recorded phase, each with the generation its check ran - # in, then the runtime line; the hook adds whether status exits 0. + # in and its wall and CPU times, then the runtime line; the hook adds + # whether status exits 0. self.assertEqual( self.status(state), [ f"NVX-TIME-ABI: v=1 phase=boot status=ok cpus=8 {rates} " - "generation=0 elapsed_us=2400", + "generation=0 elapsed_us=2400 cpu_us=1900", f"NVX-TIME-ABI: v=1 phase=capture status=ok cpus=8 {rates} " - "generation=1 elapsed_us=310", + "generation=1 elapsed_us=310 cpu_us=290", f"NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 {rates} " - "generation=2 elapsed_us=1830", + "generation=2 elapsed_us=1830 cpu_us=640", "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=2 " "discontinuities=2 offset_ns=-42 uncertainty_ns=1800 " "rejected_samples=2 last_sample_error=G_SAMPLE_UNCERTAIN", "ok=1", ], ) - # A check still pending after the wait has no elapsed_us and fails the - # exit status. A cold boot has a boot line only, and a state without a - # boot record counts as a pending boot check. + # A check still pending after the wait has neither elapsed_us nor + # cpu_us and fails the exit status. A cold boot has a boot line only, + # and a state without a boot record counts as a pending boot check. pending = state.replace("restore_status=ok", "restore_status=pending") self.assertEqual( self.status(pending)[2::2], @@ -837,16 +840,30 @@ def test_status_prints_every_recorded_check_and_the_runtime_line(self): self.assertEqual( report[0], f"NVX-TIME-REPORT: v=1 phase=boot status=ok cpus=8 {rates} " - "generation=0 elapsed_us=2400 failures=0", + "generation=0 elapsed_us=2400 cpu_us=1900 failures=0", ) self.assertEqual( report[2], f"NVX-TIME-REPORT: v=1 phase=restore status=fail cpus=4 {rates} " - "generation=2 elapsed_us=1830 failures=3", + "generation=2 elapsed_us=1830 cpu_us=640 failures=3", ) self.assertTrue(report[3].startswith("NVX-TIME-REPORT: v=1 phase=runtime ")) self.assertEqual(report[4], "ok=0") + def test_readiness_times_travel_in_one_signal_value(self): + # The helper hands the readiness path's wall and CPU times to the + # daemon in one 64-bit sigqueue() value, each clamped to 32 bits. + for elapsed, cpu, value in ( + (2400, 1200, "0x000004b000000960"), + (-5, 2**40, "0xffffffff00000000"), + (2**32 - 1, 0, "0x00000000ffffffff"), + ): + self.assertEqual( + self.run_test("readiness", str(elapsed), str(cpu)), + f"value={value} elapsed_us={min(max(elapsed, 0), 2**32 - 1)} " + f"cpu_us={min(max(cpu, 0), 2**32 - 1)}\n", + ) + def restore_record(self, text: str) -> str: return self.run_test( "restore-record", self.fixture("restore.txt", text) From 10b331d897484514c88616445aa55832685638b3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 02:14:17 -0700 Subject: [PATCH 104/188] guest: start the deferred time ABI checks at 150 ms and count step 13's CPU The spec (aeb7e04b) moves restore step 13 and the asynchronous boot checks from 100 ms to 150 ms after the acknowledgement or the boot step, so that a workload's own activity right after readiness does not meet them. The promotion to normal priority 100 ms after they start is unchanged. The spec (01d17012, Q24 and Q26) also limits the restore check's cpu_us to step 13: the daemon's preparation and the step 11 checks, in the nvx-time processes only. The readiness path, steps 6 to 12, is the restore latency gate's to measure. The helper therefore no longer reads its CPU clock after the restore, and the hand-off signal carries only the readiness path's wall time, which elapsed_us still covers. The daemon's preparation now includes its fork of the step 13 worker. The fork's cost to the daemon arises after the worker's copy of the timing was taken, so the daemon sends it on a pipe that the worker reads before it records the check. The daemon returns to normal priority as soon as the fork does, before that write, so it can still promote the worker under a CPU-bound workload. The readiness packing test is replaced by tests of the two delays and of step 13's CPU time, which fails without the fork share. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- guest/common/nvx-time.c | 295 ++++++++++++++++++++++--------------- scripts/test_guest_time.py | 33 +++-- 2 files changed, 195 insertions(+), 133 deletions(-) diff --git a/guest/common/nvx-time.c b/guest/common/nvx-time.c index a22bf0653..222016716 100644 --- a/guest/common/nvx-time.c +++ b/guest/common/nvx-time.c @@ -124,13 +124,14 @@ #define SLOW_TIME_CONSTANT 6 #define TIMER_LIST_WAIT_NS (200 * NSEC_PER_MSEC) #define DEFAULT_STALL_TIMEOUT_S 21 -// The asynchronous boot checks start 100 ms after the time ABI boot step, -// which init runs shortly before shell-ready, and restore step 13 100 ms +// The asynchronous boot checks start 150 ms after the time ABI boot step, +// which init runs shortly before shell-ready, and restore step 13 150 ms // after the acknowledgement: earlier, they contend with the readiness path -// while the VMM still demand-faults guest memory. Both start at SCHED_IDLE -// and continue at normal priority 100 ms after they start, which bounds the -// fail-fast window under a CPU-bound workload. -#define DEFERRED_START_NS (100 * NSEC_PER_MSEC) +// while the VMM still demand-faults guest memory, and with the workload's own +// first activity after readiness. Both start at SCHED_IDLE and continue at +// normal priority 100 ms after they start, which bounds the fail-fast window +// under a CPU-bound workload. +#define DEFERRED_START_NS (150 * NSEC_PER_MSEC) #define IDLE_BOUND_NS (100 * NSEC_PER_MSEC) // nvx-time status and pre-capture wait at most 30 s for pending checks. #define STATUS_WAIT_NS (30 * NSEC_PER_SEC) @@ -2665,50 +2666,27 @@ static void promoter_stop(struct promoter *promoter) pthread_join(promoter->thread, NULL); } -// The readiness path's wall and CPU times travel to the daemon in one -// sigqueue() value, 64 bits on x86-64: each in microseconds, clamped to 32 -// bits, the wall time in the low half. -_Static_assert(sizeof(void *) == sizeof(uint64_t), "sigval holds 64 bits"); - -static uint64_t clamp_u32(int64_t value) -{ - return value < 0 ? 0 : value > UINT32_MAX ? UINT32_MAX : (uint64_t)value; -} - -static uint64_t pack_readiness(int64_t elapsed_us, int64_t cpu_us) -{ - return clamp_u32(cpu_us) << 32 | clamp_u32(elapsed_us); -} - -static void unpack_readiness(uint64_t value, int64_t *elapsed_us, - int64_t *cpu_us) -{ - *elapsed_us = (int64_t)(value & UINT32_MAX); - *cpu_us = (int64_t)(value >> 32); -} - // Restore steps 12 and 13 on the readiness path: the acknowledgement when the -// host gated its input, then hand step 13 to DAEMON. The signal carries the -// readiness path's wall time since START_NS (CLOCK_MONOTONIC) and its CPU -// time since CPU_START_NS (this process's CPU clock, less any earlier -// process's share), in microseconds, both read after the acknowledgement so -// that neither read delays it. Returns -1, without acknowledging, when no -// daemon runs; 1 when the daemon vanished after the acknowledgement. The -// caller has enabled the ports when ACK is set. -static int signal_restore(pid_t daemon, bool ack, int64_t start_ns, - int64_t cpu_start_ns) +// host gated its input, then hand step 13 to DAEMON. The sigqueue() value is +// the readiness path's wall time since START_NS (CLOCK_MONOTONIC), in +// microseconds clamped to 32 bits, read after the acknowledgement so that the +// read does not delay it. The readiness path's CPU time is not reported: the +// restore latency gate measures that path. Returns -1, without acknowledging, +// when no daemon runs; 1 when the daemon vanished after the acknowledgement. +// The caller has enabled the ports when ACK is set. +static int signal_restore(pid_t daemon, bool ack, int64_t start_ns) { union sigval value; int64_t elapsed_us; - int64_t cpu_us; if (daemon <= 0 || kill(daemon, 0) != 0) return -1; if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; - cpu_us = (cpu_time_ns() - cpu_start_ns) / NSEC_PER_USEC; - value.sival_ptr = (void *)(uintptr_t)pack_readiness(elapsed_us, cpu_us); + if (elapsed_us > UINT32_MAX) + elapsed_us = UINT32_MAX; + value.sival_ptr = (void *)(uintptr_t)(elapsed_us < 0 ? 0 : elapsed_us); return sigqueue(daemon, SIGUSR1, value) == 0 ? 0 : 1; } @@ -3083,7 +3061,6 @@ struct daemon { bool restore_pending; int64_t restore_due_ns; int64_t readiness_us; - int64_t readiness_cpu_us; }; static void watcher_consume(void *context, const char *message) @@ -3297,23 +3274,79 @@ static int parse_restore_record(const char *text, unsigned long long *generation // The times that restore_work records. The wall time is READINESS_US, the // readiness path's, plus the checks' since START_NS, when they were due. The -// CPU time is PRIOR_CPU_NS, the readiness path's and the daemon's share, plus -// this process's since CPU_BASE_NS. +// CPU time is step 13's alone: this process's since CPU_BASE_NS, plus the +// daemon's preparation, PRIOR_CPU_NS up to the fork and the fork's own share, +// which the daemon writes to FORK_FD (-1 when there is none) once the fork +// returns. struct restore_timing { int64_t start_ns; int64_t readiness_us; int64_t prior_cpu_ns; int64_t cpu_base_ns; + int fork_fd; }; +// Step 13's CPU time so far, in a process that TIMING describes. The read +// waits for the daemon's fork share if the daemon has not yet written it. +static int64_t step13_cpu_ns(const struct restore_timing *timing) +{ + int64_t cpu = timing->prior_cpu_ns + cpu_time_ns() - timing->cpu_base_ns; + int64_t fork_share = 0; + + if (timing->fork_fd >= 0 && + read(timing->fork_fd, &fork_share, sizeof(fork_share)) == + sizeof(fork_share) && + fork_share > 0) + cpu += fork_share; + return cpu; +} + +// Forks the step 13 worker at SCHED_IDLE for a daemon whose preparation +// started at CPU_START_NS, its CPU clock, and completes TIMING's CPU fields +// for the child. The parent returns to normal priority as soon as the fork +// does. Its fork costs CPU time after the child's copy of TIMING was taken, so +// it writes that share to a pipe that the child reads. Returns fork()'s +// result. +static pid_t fork_step13(int64_t cpu_start_ns, struct restore_timing *timing) +{ + int fds[2] = {-1, -1}; + int64_t fork_share; + pid_t pid; + + if (pipe2(fds, O_CLOEXEC) != 0) + fds[0] = fds[1] = -1; + timing->fork_fd = fds[0]; + timing->cpu_base_ns = 0; + set_idle_priority(0, true); + timing->prior_cpu_ns = cpu_time_ns() - cpu_start_ns; + pid = fork(); + if (pid == 0) { + if (fds[1] >= 0) + close(fds[1]); + return 0; + } + set_idle_priority(0, false); + if (fds[0] >= 0) + close(fds[0]); + timing->fork_fd = -1; + if (fds[1] >= 0) { + fork_share = cpu_time_ns() - cpu_start_ns - timing->prior_cpu_ns; + if (pid > 0) + (void)write_all(fds[1], &fork_share, sizeof(fork_share)); + close(fds[1]); + } + return pid; +} + // Restore step 13, fail-fast: the step 11 restore checks, the grace-period // release, the saved watchdog settings, the deferred C7, and the restore -// check's record. A child of the daemon runs it at SCHED_IDLE from 100 ms -// after the acknowledgement; the daemon raises it to normal priority 100 ms -// later and releases the suppression itself at the stall timeout. -// restore-finish runs it inline, with its own deadline (WITH_DEADLINE), in -// report-only mode when no daemon runs. Neither recorded time counts the -// grace period wait or C7. Returns 0, or 1 when a report-only release failed. +// check's record. A child of the daemon runs it at SCHED_IDLE from +// DEFERRED_START_NS after the acknowledgement; the daemon raises it to normal +// priority IDLE_BOUND_NS later and releases the suppression itself at the +// stall timeout. restore-finish runs it inline, with its own deadline +// (WITH_DEADLINE), in report-only mode when no daemon runs. Neither recorded +// time counts the grace period wait or C7. Returns 0, or 1 when a report-only +// release failed. static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, const struct restore_timing *timing) { @@ -3329,7 +3362,7 @@ static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, else run_restore_checks(&checks, new_cpus, new_count); elapsed = clock_ns(CLOCK_MONOTONIC) - timing->start_ns; - cpu = timing->prior_cpu_ns + cpu_time_ns() - timing->cpu_base_ns; + cpu = step13_cpu_ns(timing); if (with_deadline) { (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); released = release_with_deadline(timeout_s); @@ -3351,10 +3384,10 @@ static int restore_work(const bool *new_cpus, int new_count, bool with_deadline, return released ? 0 : 1; } -// Starts restore step 13 when it is due, 100 ms after the acknowledgement, in -// a child at SCHED_IDLE, so the watcher and the discipline keep normal -// priority; the fork itself also runs at SCHED_IDLE. The readiness path's -// times came with the signal. +// Starts restore step 13 when it is due, DEFERRED_START_NS after the +// acknowledgement, in a child at SCHED_IDLE, so the watcher and the +// discipline keep normal priority; the fork itself also runs at SCHED_IDLE. +// The readiness path's wall time came with the signal. static void start_restore(struct daemon *daemon) { int64_t cpu_start = cpu_time_ns(); @@ -3388,18 +3421,13 @@ static void start_restore(struct daemon *daemon) } } (void)read_int64(RCU_STALL_TIMEOUT, &timeout_s); - // The worker's wall time runs from when step 13 was due. Its CPU time - // runs from the fork, and the daemon's share is this preparation. + // The worker's wall time runs from when step 13 was due. Its CPU time is + // step 13's: this preparation, the fork, and the worker's own. timing.start_ns = daemon->restore_due_ns; timing.readiness_us = daemon->readiness_us; - timing.cpu_base_ns = 0; - set_idle_priority(0, true); - timing.prior_cpu_ns = daemon->readiness_cpu_us * NSEC_PER_USEC + - cpu_time_ns() - cpu_start; - pid = fork(); + pid = fork_step13(cpu_start, &timing); if (pid == 0) _exit(restore_work(new_cpus, new_count, false, &timing)); - set_idle_priority(0, false); if (pid < 0) { fail_fatal(STATUS_REPAIR, "G_REPAIR_SUPPRESSION", "repair", PHASE_RESTORE, "cannot start the restore work: %s", @@ -3457,19 +3485,17 @@ static void handle_signals(struct daemon *daemon) while (read(daemon->signals, &info, sizeof(info)) == sizeof(info)) { if (info.ssi_signo == SIGTERM) _exit(0); - // sigqueue() carries the readiness path's wall and CPU times. Step 13 - // is due 100 ms later; the repair paired the clock with the packet's - // UTC now. + // sigqueue() carries the readiness path's wall time. Step 13 is due + // DEFERRED_START_NS later; the repair paired the clock with the + // packet's UTC now. if (info.ssi_signo == SIGUSR1) { daemon->restore_pending = true; daemon->last_accepted_ns = clock_ns(CLOCK_MONOTONIC); daemon->restore_due_ns = daemon->last_accepted_ns + DEFERRED_START_NS; - daemon->readiness_us = 0; - daemon->readiness_cpu_us = 0; - if (info.ssi_code == SI_QUEUE) - unpack_readiness(info.ssi_ptr, &daemon->readiness_us, - &daemon->readiness_cpu_us); + daemon->readiness_us = info.ssi_code == SI_QUEUE + ? (int64_t)(info.ssi_ptr & UINT32_MAX) + : 0; } } // A worker that ended, if any: the daemon loop starts a due restore only @@ -3561,8 +3587,8 @@ static void daemon_loop(struct daemon *daemon) arm_timer(daemon); } now = clock_ns(CLOCK_MONOTONIC); - // Step 13 starts 100 ms after the ack, and continues at normal - // priority 100 ms after it starts. + // Step 13 starts DEFERRED_START_NS after the ack, and continues at + // normal priority IDLE_BOUND_NS after it starts. if (daemon->restore_pending && daemon->worker_pid == 0 && now >= daemon->restore_due_ns) { daemon->restore_pending = false; @@ -3692,17 +3718,18 @@ static void sleep_until(int64_t deadline_ns) } // The asynchronous boot checks: every boot check but C12, fail-fast, from -// 100 ms after the boot step at SCHED_IDLE, and at normal priority from -// IDLE_BOUND_NS after they start. When they pass, the checker becomes the +// DEFERRED_START_NS after the boot step at SCHED_IDLE, and at normal priority +// from IDLE_BOUND_NS after they start. When they pass, the checker becomes the // daemon and records the boot check, whose wall and CPU times add C12's // (C12_NS and C12_CPU_NS) and the daemon's start, but not the delay. It never // returns. static void boot_checker(const struct boot_sample *sample, int64_t c12_ns, int64_t c12_cpu_ns, int c12_failures) { - // The 100 ms count from this fork in the time ABI boot step, the one + // The delay counts from this fork in the time ABI boot step, the one // point that every guest mode shares: init runs it before the rest of its - // setup, so the checks start slightly less than 100 ms after shell-ready. + // setup, so the checks start somewhat less than DEFERRED_START_NS after + // shell-ready. int64_t start = clock_ns(CLOCK_MONOTONIC) + DEFERRED_START_NS; int64_t cpu_start = cpu_time_ns(); int null = open("/dev/null", O_RDWR | O_CLOEXEC); @@ -4004,7 +4031,6 @@ struct repair_record { int64_t step_realtime_ns; int64_t frequency; int64_t elapsed_us; - int64_t cpu_us; }; static void apply_repair(struct time_state *state, const void *context) @@ -4031,14 +4057,13 @@ static void apply_repair(struct time_state *state, const void *context) "none"); // Step 8 marks this restore's check pending, so nvx-time status waits for // the daemon's step 13. Until then, restore_elapsed_us is the readiness - // path up to the clock step, at last_step_realtime_ns, and - // restore_cpu_us the helper's CPU time, which restore-finish adds to its - // own (0 when the helper finishes the restore itself). + // path up to the clock step, at last_step_realtime_ns; restore_cpu_us + // counts only step 13, which records it. snprintf(state->checks[PHASE_RESTORE].status, sizeof(state->checks[PHASE_RESTORE].status), "pending"); state->checks[PHASE_RESTORE].cpus = 0; state->checks[PHASE_RESTORE].elapsed_us = repair->elapsed_us; - state->checks[PHASE_RESTORE].cpu_us = repair->cpu_us; + state->checks[PHASE_RESTORE].cpu_us = 0; state->checks[PHASE_RESTORE].generation = state->generation; state->checks[PHASE_RESTORE].failures = -1; } @@ -4052,14 +4077,14 @@ static int repair_failed(const char *code, const char *detail) // Restore steps 6 to 8: read the packet with four-byte reads, validate it, // and set the wall clock from its bracketed UTC, then write the entropy to // ENTROPY_FD, the caller's file at ENTROPY_PATH. START_NS is when the capture -// request returned, and CPU_START_NS this process's CPU clock right after the -// restore. The caller holds the portb lock. With FINISH (untiered) and a -// DAEMON, a restore with no processors or memory to activate needs no shell -// work before the acknowledgement, so steps 12 and 13 start here too and no -// second helper process starts; the metadata then begins with 2 instead of 1. +// request returned. The caller holds the portb lock. With FINISH (untiered) +// and a DAEMON, a restore with no processors or memory to activate needs no +// shell work before the acknowledgement, so steps 12 and 13 start here too +// and no second helper process starts; the metadata then begins with 2 +// instead of 1. static int repair_restore(const uint8_t *previous_id, const char *entropy_path, - int entropy_fd, int64_t start_ns, - int64_t cpu_start_ns, bool finish, pid_t daemon) + int entropy_fd, int64_t start_ns, bool finish, + pid_t daemon) { static uint8_t body[PACKET_MAX_SIZE]; static struct restore_packet packet; @@ -4129,14 +4154,10 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, repair.elapsed_us = (clock_ns(CLOCK_MONOTONIC) - start_ns) / NSEC_PER_USEC; repair.packet = &packet; // A plain untiered restore needs no shell work: this helper also - // acknowledges and hands step 13 to the daemon, and reads its CPU clock - // again only after the acknowledgement. Otherwise restore-finish adds - // this helper's CPU time to its own. + // acknowledges and hands step 13 to the daemon. finishing = finish && daemon > 0 && packet.online_vp_count == 0 && packet.range_count == 0 && (packet.flags & PACKET_MEMORY_TARGET) == 0; - repair.cpu_us = - finishing ? 0 : (cpu_time_ns() - cpu_start_ns) / NSEC_PER_USEC; // The entropy goes to the caller in every case, an untiered restore's // too; the file already exists, empty, so only its bytes are written. if (pwrite(entropy_fd, packet.entropy, PACKET_ENTROPY_SIZE, 0) != @@ -4155,7 +4176,7 @@ static int repair_restore(const uint8_t *previous_id, const char *entropy_path, if (finishing) { finished = signal_restore(daemon, (packet.flags & PACKET_ACK_REQUIRED) != 0, - start_ns, cpu_start_ns); + start_ns); if (finished > 0) fail_fatal(STATUS_CONFORMANCE, "G_CONFORMANCE_C10", "conformance", PHASE_RESTORE, "the time daemon is not running"); @@ -4182,7 +4203,6 @@ static int cmd_capture(int argc, char **argv) char *end; long request; int64_t start; - int64_t cpu_start; unsigned status; pid_t daemon = -1; int entropy_fd; @@ -4237,13 +4257,8 @@ static int cmd_capture(int argc, char **argv) puts("0"); return 0; } - // The restored TSC advanced by the downtime, and the first scheduler - // update after the restore charges that to this process, which ran the - // capture. Reading the CPU clock makes that update, so the baseline - // includes it. - cpu_start = cpu_time_ns(); - result = repair_restore(previous_id, argv[2], entropy_fd, start, cpu_start, - argc == 4, daemon); + result = repair_restore(previous_id, argv[2], entropy_fd, start, argc == 4, + daemon); close(lock); return result; } @@ -4252,7 +4267,7 @@ static int cmd_capture(int argc, char **argv) // acknowledgement when the packet asked for one, then the daemon runs step 13 // asynchronously. The record tells it which CPUs activation onlined. The // readiness path started restore_elapsed_us before the helper's clock step, -// at last_step_realtime_ns, and the helper spent restore_cpu_us of CPU time. +// at last_step_realtime_ns. static int cmd_restore_finish(int argc, char **argv) { bool new_cpus[MAX_CPUS] = {false}; @@ -4261,7 +4276,6 @@ static int cmd_restore_finish(int argc, char **argv) struct time_state state; struct checks checks; char text[256]; - int64_t cpu_start; int64_t start; bool ack = false; int new_count = 0; @@ -4299,10 +4313,7 @@ static int cmd_restore_finish(int argc, char **argv) start = clock_ns(CLOCK_MONOTONIC) - (clock_ns(CLOCK_REALTIME) - state.last_step_realtime_ns) - state.checks[PHASE_RESTORE].elapsed_us * NSEC_PER_USEC; - // This process runs only for the restore, so all of its CPU time counts, - // plus the helper's. - cpu_start = -state.checks[PHASE_RESTORE].cpu_us * NSEC_PER_USEC; - signaled = signal_restore(daemon_pid(), ack, start, cpu_start); + signaled = signal_restore(daemon_pid(), ack, start); if (signaled == 0) return 0; if (signaled > 0 || !g_report_only) { @@ -4311,13 +4322,15 @@ static int cmd_restore_finish(int argc, char **argv) check_failed(&checks, "C10", "the time daemon is not running"); return 1; } - // Report-only runs may lack the daemon; finish the restore here. + // Report-only runs may lack the daemon; finish the restore here, where + // step 13's CPU time is restore_work's own. if (ack) outb(SNAPSHOT_ACKNOWLEDGE, PORT_SNAPSHOT); timing.start_ns = clock_ns(CLOCK_MONOTONIC); timing.readiness_us = (timing.start_ns - start) / NSEC_PER_USEC; + timing.prior_cpu_ns = 0; timing.cpu_base_ns = cpu_time_ns(); - timing.prior_cpu_ns = timing.cpu_base_ns - cpu_start; + timing.fork_fd = -1; (void)restore_work(new_cpus, new_count, true, &timing); return 0; } @@ -4792,6 +4805,52 @@ static int test_promote(void) return 0; } +static int64_t g_test_fork_burn_ms; + +// Spends MS milliseconds of this process's CPU time. +static void burn_cpu(int64_t ms) +{ + int64_t until = cpu_time_ns() + ms * NSEC_PER_MSEC; + + while (cpu_time_ns() < until) + ; +} + +static void test_fork_burn(void) +{ + burn_cpu(g_test_fork_burn_ms); +} + +// Forks a step 13 worker as start_restore does, after PRE_MS of preparation, +// with a fork that costs the parent FORK_MS; the worker spends CHILD_MS and +// prints its preparation share and step 13's CPU time. +static int test_step13_cpu(int argc, char **argv) +{ + struct restore_timing timing = {0}; + int64_t start = cpu_time_ns(); + int status; + pid_t pid; + + if (argc != 3) + return 2; + g_test_fork_burn_ms = strtoll(argv[1], NULL, 10); + if (pthread_atfork(NULL, test_fork_burn, NULL) != 0) + return 1; + burn_cpu(strtoll(argv[0], NULL, 10)); + pid = fork_step13(start, &timing); + if (pid == 0) { + burn_cpu(strtoll(argv[2], NULL, 10)); + printf("prior_ms=%" PRId64 " total_ms=%" PRId64 "\n", + timing.prior_cpu_ns / NSEC_PER_MSEC, + step13_cpu_ns(&timing) / NSEC_PER_MSEC); + fflush(stdout); + _exit(0); + } + if (pid < 0 || waitpid(pid, &status, 0) != pid) + return 1; + return WIFEXITED(status) ? WEXITSTATUS(status) : 1; +} + static int cmd_test(int argc, char **argv) { const char *name = argc > 0 ? argv[0] : ""; @@ -4904,16 +4963,12 @@ static int cmd_test(int argc, char **argv) } if (strcmp(name, "restore-record") == 0) return test_restore_record(argc, argv); - if (strcmp(name, "readiness") == 0 && argc == 2) { - uint64_t value = pack_readiness(strtoll(argv[0], NULL, 10), - strtoll(argv[1], NULL, 10)); - int64_t elapsed_us; - int64_t cpu_us; - - unpack_readiness(value, &elapsed_us, &cpu_us); - printf("value=0x%016" PRIx64 " elapsed_us=%" PRId64 " cpu_us=%" PRId64 - "\n", - value, elapsed_us, cpu_us); + if (strcmp(name, "step13-cpu") == 0) + return test_step13_cpu(argc, argv); + if (strcmp(name, "deferral") == 0 && argc == 0) { + printf("deferred_start_ms=%" PRId64 " idle_bound_ms=%" PRId64 "\n", + (int64_t)(DEFERRED_START_NS / NSEC_PER_MSEC), + (int64_t)(IDLE_BOUND_NS / NSEC_PER_MSEC)); return 0; } if (strcmp(name, "parallel") == 0) diff --git a/scripts/test_guest_time.py b/scripts/test_guest_time.py index bdb84c9b0..7de7daf3b 100644 --- a/scripts/test_guest_time.py +++ b/scripts/test_guest_time.py @@ -850,19 +850,26 @@ def test_status_prints_every_recorded_check_and_the_runtime_line(self): self.assertTrue(report[3].startswith("NVX-TIME-REPORT: v=1 phase=runtime ")) self.assertEqual(report[4], "ok=0") - def test_readiness_times_travel_in_one_signal_value(self): - # The helper hands the readiness path's wall and CPU times to the - # daemon in one 64-bit sigqueue() value, each clamped to 32 bits. - for elapsed, cpu, value in ( - (2400, 1200, "0x000004b000000960"), - (-5, 2**40, "0xffffffff00000000"), - (2**32 - 1, 0, "0x00000000ffffffff"), - ): - self.assertEqual( - self.run_test("readiness", str(elapsed), str(cpu)), - f"value={value} elapsed_us={min(max(elapsed, 0), 2**32 - 1)} " - f"cpu_us={min(max(cpu, 0), 2**32 - 1)}\n", - ) + def test_deferred_work_starts_150_ms_late_and_is_promoted_100_ms_later(self): + # Step 13 and the asynchronous boot checks start 150 ms after the + # acknowledgement or the boot step and leave SCHED_IDLE 100 ms after + # they start. + self.assertEqual( + self.run_test("deferral"), "deferred_start_ms=150 idle_bound_ms=100\n" + ) + + def test_restore_cpu_time_is_step_13_with_the_daemons_fork(self): + # The worker's step 13 CPU time adds the daemon's preparation before + # the fork, the fork's cost to the daemon, which arrives on a pipe + # after the worker's copy of the timing was taken, and its own. + fields = dict( + field.split("=") + for field in self.run_test("step13-cpu", "20", "30", "10").split() + ) + self.assertGreaterEqual(int(fields["prior_ms"]), 20) + self.assertLess(int(fields["prior_ms"]), 30) + self.assertGreaterEqual(int(fields["total_ms"]), 60) + self.assertLess(int(fields["total_ms"]), 90) def restore_record(self, text: str) -> str: return self.run_test( From 96a0ed0ca78611e013d0ab9fb28ec0e9427a3ec4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:05:07 -0700 Subject: [PATCH 105/188] openvmm: pin the time ABI v1 integration head Pin nanvix/openvmm nvx/time-abi-v1 at cae746f6f: the time ABI v1 stack through the flip, which makes the time ABI the only microVM time path, and the legacy clock deletions, rebased onto microvm/v2.1 2728f33ea, the base that dev pins. It lands in lockstep with the guest's kernel and time ABI wiring commits, which need the flipped OpenVMM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 4355c010e..cae746f6f 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 4355c010e726128c751138e7f74ad969e76a4c19 +Subproject commit cae746f6f20e956ec0decc7d35971c08e3f12234 From a277f72f55913cda2f193b9b189b45dd6c1236af Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:40:53 -0700 Subject: [PATCH 106/188] ci: drop clock tuning from the benchmark kernel command lines The benchmark harness worked around each backend's guest clock with kernel command-line tokens: BASE_TUNING passed tsc=reliable and no_timer_check to every lifecycle boot, KVM lifecycle boots forced clocksource=kvm-clock, and the cold-start clocksource variant used kvm-clock on KVM and tsc elsewhere. Under time ABI v1 the Hyper-V identity gives Linux the TSC and LAPIC rates, marks the TSC reliable, and sets no_timer_check on every backend, there is no kvmclock, and the guest's boot check rejects a clocksource= token other than tsc. Remove tsc=reliable and no_timer_check from BASE_TUNING, remove clocksource=kvm-clock from the KVM worker and native lifecycle commands, and make cold_start_clocksource pass clocksource=tsc on every backend, as the spec accepts without an exemption. The isolated tsc=reliable and no_timer_check cold-start variants stay: they are gated metrics. This lands with the OpenVMM switch to the time ABI: today's guests still rely on these tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 6 ++--- scripts/nvx_tools/benchmark.py | 17 +++++------- scripts/test_nvx_tools.py | 47 +++++++++++++++++++++++++++++++--- 3 files changed, 54 insertions(+), 16 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index e5fcab91b..0ad10be58 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -267,10 +267,10 @@ command line tunes the clock or clocksource, except the `cold_start_clocksource` | Benchmark | Phase | Kernel command line | | --- | --- | --- | -| Shell lifecycle | cold/capture | `BASE` plus the combined `BASE_TUNING` parameters; CI uses 128 MiB. | +| Shell lifecycle | cold/capture | `BASE` plus `BASE_TUNING`: `random.trust_cpu=on rcupdate.rcu_expedited=1 nokaslr mitigations=off cryptomgr.notests quiet loglevel=0`; CI uses 128 MiB. | | Shell lifecycle | restore | restore (from the measured shell-ready snapshot) | | Cold start | baseline | `QUIET` | -| Cold start | tuning variant | `QUIET` plus one of `clocksource=`, `tsc=reliable`, `no_timer_check`, `random.trust_cpu=on`, `rcupdate.rcu_expedited=1`, `nokaslr`, `mitigations=off`, or `cryptomgr.notests` | +| Cold start | tuning variant | `QUIET` plus one of `clocksource=tsc`, `tsc=reliable`, `no_timer_check`, `random.trust_cpu=on`, `rcupdate.rcu_expedited=1`, `nokaslr`, `mitigations=off`, or `cryptomgr.notests` | | Virtual file system | guest runs | `QUIET` | | Device operation rates | guest runs | `QUIET`; virtio-net uses the portable profile at `10.0.0.2/24` | | Shell snapshot | cold | `QUIET` | @@ -346,7 +346,7 @@ exactly one kernel parameter to the quiet baseline. | Metric | Description | | --- | --- | | `cold_start_base` | Quiet baseline with no additional tuning parameter. | -| `cold_start_clocksource` | Baseline plus `clocksource=kvm-clock` on KVM or `clocksource=tsc` on MSHV/WHP. | +| `cold_start_clocksource` | Baseline plus `clocksource=tsc` on every backend. Before the time ABI, KVM measured `clocksource=kvm-clock`, so KVM history before that change measures a different clocksource. | | `cold_start_tsc_reliable` | Baseline plus `tsc=reliable`. | | `cold_start_no_timer_check` | Baseline plus `no_timer_check`. | | `cold_start_random_trust_cpu` | Baseline plus `random.trust_cpu=on`. | diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index 6f1152c0b..7a0081511 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -52,10 +52,12 @@ OUTPUT_BUFFER_LIMIT_BYTES = 1024 * 1024 PEAK_RSS_SAMPLE_ATTEMPTS = 3 BASE_TUNING = ( - "tsc=reliable no_timer_check random.trust_cpu=on " - "rcupdate.rcu_expedited=1 nokaslr mitigations=off " + "random.trust_cpu=on rcupdate.rcu_expedited=1 nokaslr mitigations=off " "cryptomgr.notests quiet loglevel=0" ) +# The cold-start clocksource variant selects the only clocksource the time ABI +# allows, on every backend. +COLD_START_CLOCKSOURCE = "clocksource=tsc" CLOCKSOURCE_CURRENT_PATH = ( "/sys/devices/system/clocksource/clocksource0/current_clocksource" ) @@ -2085,10 +2087,6 @@ def format_sample_summary(samples: Sequence[float], *, unit: str = "ms") -> str: ) -def clocksource_parameter(backend: str) -> str: - return "clocksource=kvm-clock" if backend == "kvm" else "clocksource=tsc" - - def stable_clocksource_wait_script() -> str: """Wait up to five seconds for Linux to replace its tsc-early clocksource.""" return "\n".join( @@ -3016,10 +3014,9 @@ def benchmark_cold_start_workload( command_prefix: Sequence[str] = (), windows_cpus: set[int] | None = None, ) -> None: - clocksource = clocksource_parameter(backend) scenarios = ( ("base", None), - (clocksource, clocksource), + (COLD_START_CLOCKSOURCE, COLD_START_CLOCKSOURCE), ("tsc=reliable", "tsc=reliable"), ("no_timer_check", "no_timer_check"), ("random.trust_cpu=on", "random.trust_cpu=on"), @@ -5687,7 +5684,7 @@ def run_kvm_worker(args: argparse.Namespace) -> int: "--initrd", str(stage / AlpineBuildConstants.INITRAMFS_NAME), "--cmdline", - f"clocksource=kvm-clock {BASE_TUNING}", + BASE_TUNING, ] if args.net is not None: append_network_arguments(boot_command, args.net, args.network_profile) @@ -5970,7 +5967,7 @@ def make_boot_command(memory_mib: int) -> list[str]: "--initrd", str(initrd), "--cmdline", - f"{'clocksource=kvm-clock ' if backend == 'kvm' else ''}{BASE_TUNING}", + BASE_TUNING, ] if args.net is not None: append_network_arguments(command, args.net, args.network_profile) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 502795757..ef9be7270 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3,6 +3,7 @@ import argparse import ast +import contextlib import errno import hashlib import http.client @@ -10192,10 +10193,50 @@ def test_parses_dd_rates_and_network_gateway(self): self.assertEqual(benchmark.parse_dd_rate(output, 1), 268.4) self.assertEqual(benchmark.parse_dd_rate(output, 2), 1600.0) self.assertEqual(benchmark.network_gateway("10.0.0.2/24"), "10.0.0.1") - self.assertEqual( - benchmark.clocksource_parameter("kvm"), "clocksource=kvm-clock" + + def test_benchmark_command_lines_carry_no_clock_tuning(self): + # Under time ABI v1 the hypervisor identity gives Linux its TSC and + # LAPIC rates and trust, so the lifecycle line tunes no clock and the + # cold-start clocksource variant selects tsc on every backend. + for token in ( + "tsc=", + "no_timer_check", + "clocksource=", + "tsc_early_khz=", + "lapic_timer_hz=", + ): + self.assertNotIn(token, benchmark.BASE_TUNING) + args = argparse.Namespace( + runs=1, + warmups=0, + memory_mib=128, + processors=1, + timeout=10.0, + teardown_mode="guest-exit", ) - self.assertEqual(benchmark.clocksource_parameter("mshv"), "clocksource=tsc") + for backend in ("kvm", "mshv", "whp"): + commands: list[list[str]] = [] + + def fake_benchmark( + command: list[str], + commands: list[list[str]] = commands, + **_kwargs: object, + ): + commands.append(command) + return {"samples_ms": [100.0]} + + with ( + self.subTest(backend=backend), + patch.object(benchmark, "benchmark", side_effect=fake_benchmark), + contextlib.redirect_stdout(io.StringIO()) as stdout, + ): + benchmark.benchmark_cold_start_workload( + args, Path("openvmm"), Path("vmlinux"), Path("initrd"), backend + ) + cmdlines = [command[command.index("--cmdline") + 1] for command in commands] + self.assertIn("quiet loglevel=0 clocksource=tsc", cmdlines) + self.assertFalse(any("kvm-clock" in cmdline for cmdline in cmdlines)) + self.assertIn(" clocksource=tsc :", stdout.getvalue()) def test_smp_probe_uses_explicit_topology_and_worker_rendezvous(self): script = benchmark.smp_probe_script(4) From ba7e9f3620359717457b34722d657a30745be11d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 04:54:31 -0700 Subject: [PATCH 107/188] ci: stop waiting for a stable clocksource before snapshot capture Linux registered the TSC clocksource as tsc-early and replaced it about a second into boot. A snapshot taken in that window could fail after restore without any cross-CPU skew, because the clocksource watchdog compared tsc-early with jiffies across the downtime (#253). The harness therefore made MSHV and WHP captures wait up to five seconds for tsc-early to disappear (0.5 to 0.9 s per capture), WHP snapshot-core waited the same way, and KVM snapshot-core waited for kvm-clock, selected it, and checked that it stayed selected. Under time ABI v1 the TSC rate is known, so Linux registers tsc at device_initcall on every backend and no capture can see tsc-early, and there is no kvm-clock. Remove stable_clocksource_wait_script, the capture-time wait and its SMP-CLOCKSOURCE-FAIL check, and the KVM and WHP snapshot-core branches; snapshot-core.sh.in had no other placeholder, so it becomes snapshot-core.sh. capture_snapshot, prepare_snapshot_capture_script, and benchmark_snapshot_capture took the backend only to select the wait, so they no longer take it. The guest's capture conformance check, which never waits, now asserts the clocksource instead. Refs #253 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 8 +- doc/ci.md | 9 +-- scripts/nvx_tools/benchmark.py | 46 +---------- .../{snapshot-core.sh.in => snapshot-core.sh} | 2 - scripts/nvx_tools/microvm_tests.py | 47 +---------- scripts/test_microvm_tests.py | 77 +++---------------- scripts/test_nvx_tools.py | 46 ++++------- scripts/test_time_abi.py | 6 +- 8 files changed, 37 insertions(+), 204 deletions(-) rename scripts/nvx_tools/microvm_test_scripts/{snapshot-core.sh.in => snapshot-core.sh} (98%) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 0ad10be58..db15c72c8 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -319,11 +319,9 @@ CI runs `test-microvm --scenario smp-lapic --processors 1 2 4 8` before acceptan This repeats the normal SMP probe with `lapic=notscdeadline`, covering the counting LAPIC even on hosts that normally use TSC-deadline timers. The ordinary `smp` scenario retains the default timer selection. -MSHV and WHP captures wait for Linux to replace the transitional `tsc-early` clocksource with -its stable selected clocksource before starting this SMP validation. Until that switch, Linux -uses a periodic tick that doesn't recover the jiffies skipped by a restore's downtime. The -clocksource watchdog can then compare `tsc-early` with jiffies across the restore and mark the -TSC unstable. KVM guests leave `tsc-early` almost immediately after boot. +Captures do not wait for a clocksource: under the time ABI, Linux registers the `tsc` +clocksource at `device_initcall`, before any guest work, so no capture can observe the +transitional `tsc-early` window. The coordinator stages the probe and a capture controller in guest memory. The controller runs the first probe, blocks in `read`, and invokes `nvx-snapshot` when the host sends the trigger. The controller always emits `NVX-SNAPSHOT-DISPATCHED` immediately before diff --git a/doc/ci.md b/doc/ci.md index 03d5f13a8..8f0a73423 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -37,11 +37,10 @@ runs with OpenVMM lifecycle profiling and must report exactly one `startup.vp_thread_bind` record. Its `startup.vp_bind_*` records must show that an explicit MSHV target binds exactly VPs `0..N-1`, while untargeted MSHV restores and all KVM and WHP restores bind the full capacity. -On MSHV and WHP, the capture waits until Linux replaces its transitional -`tsc-early` clocksource. A snapshot taken earlier can fail after restore -without any cross-CPU skew, because the clocksource watchdog compares -`tsc-early` with jiffies across the restore downtime, as described in -[the benchmark guide](benchmarks.md). +Captures do not wait for a clocksource: the time ABI registers `tsc` at +`device_initcall` on every backend, so the transitional `tsc-early` window that +once let the clocksource watchdog compare `tsc-early` with jiffies across a +restore (#253) never reaches the guest's userspace. A restore fails as soon as its guest prints `NVX-RESTORE-PROCESSORS-FAIL`, rather than waiting for the phase timeout. Restore logs also record OpenVMM's `adjusted restored vCPU TSC` event for each VP, which includes the applied diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index 7a0081511..c022e39d9 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -58,9 +58,6 @@ # The cold-start clocksource variant selects the only clocksource the time ABI # allows, on every backend. COLD_START_CLOCKSOURCE = "clocksource=tsc" -CLOCKSOURCE_CURRENT_PATH = ( - "/sys/devices/system/clocksource/clocksource0/current_clocksource" -) KVM_RESULT_PREFIX = "OPENVMM_KVM_RESULT=" KVM_E2E_RESULT_PREFIX = "OPENVMM_KVM_E2E_RESULT=" KVM_RESTORE_RESULT_PREFIX = "OPENVMM_KVM_RESTORE_RESULT=" @@ -2087,21 +2084,6 @@ def format_sample_summary(samples: Sequence[float], *, unit: str = "ms") -> str: ) -def stable_clocksource_wait_script() -> str: - """Wait up to five seconds for Linux to replace its tsc-early clocksource.""" - return "\n".join( - ( - f"clock_path={CLOCKSOURCE_CURRENT_PATH}", - "clock_tries=0", - 'while [ "$(cat "$clock_path")" = tsc-early ] ' - '&& [ "$clock_tries" -lt 100 ]; do', - " sleep 0.05", - " clock_tries=$((clock_tries + 1))", - "done", - ) - ) - - def network_gateway(spec: str) -> str: try: interface = ipaddress.IPv4Interface(spec) @@ -3549,33 +3531,17 @@ def smp_probe_script( def prepare_snapshot_capture_script( processors: int, *, - backend: str, teardown_mode: str, network_gateway: str | None = None, ioapic_irq: int | None = None, post_restore_script: str | None = None, ) -> str: - clocksource_ready = "" - # Until Linux replaces tsc-early, its periodic tick doesn't recover the - # jiffies that a restore's downtime skips. The clocksource watchdog can then - # compare tsc-early with jiffies across the restore and mark the TSC - # unstable. KVM guests leave tsc-early almost immediately after boot. - if backend in ("mshv", "whp"): - clocksource_ready = ( - stable_clocksource_wait_script() - + "\n" - + 'current_clocksource="$(cat "$clock_path")"\n' - + '[ "$current_clocksource" != tsc-early ] || { ' - + 'echo "SMP-CLOCKSOURCE-FAIL expected=stable ' - + 'actual=$current_clocksource"; exit 80; }\n' - ) probe = smp_probe_script( processors, exit_guest=False, network_gateway=network_gateway, ioapic_irq=ioapic_irq, ) - probe = clocksource_ready + probe capture = _render_benchmark_script( "snapshot-capture-controller.sh.in", SMP_PROBE_PATH=SMP_PROBE_PATH, @@ -3970,7 +3936,6 @@ def _benchmark_shell_snapshot_restore( str(snapshot_path), ], snapshot_path, - backend=backend, timeout=args.timeout, windows_cpus=windows_cpus, processors=args.processors, @@ -4694,7 +4659,6 @@ def capture_snapshot( command: Sequence[str], snapshot_path: Path, *, - backend: str, timeout: float, windows_cpus: set[int] | None = None, processors: int | None = None, @@ -4806,7 +4770,6 @@ def request_snapshot() -> None: interaction.write_input( prepare_snapshot_capture_script( processors, - backend=backend, teardown_mode=teardown_mode, network_gateway=smp_network_gateway, ioapic_irq=smp_ioapic_irq, @@ -4916,7 +4879,6 @@ def summarize_snapshot_samples( def benchmark_snapshot_capture( args: argparse.Namespace, - backend: str, boot_command: Sequence[str], *, windows_cpus: set[int] | None = None, @@ -4948,7 +4910,6 @@ def benchmark_snapshot_capture( capture_snapshot( [*boot_command, "--snapshot-destination", str(snapshot_path)], snapshot_path, - backend=backend, timeout=args.timeout, windows_cpus=windows_cpus, processors=args.processors, @@ -5072,7 +5033,6 @@ def benchmark_snapshot_restore( str(generated_snapshot_path), ], generated_snapshot_path, - backend=hypervisor, timeout=args.timeout, windows_cpus=windows_cpus, processors=args.processors, @@ -5234,7 +5194,6 @@ def benchmark_snapshot_profile_matrix( boot_command = make_boot_command(memory_mib) capture = benchmark_snapshot_capture( profile_args, - backend, boot_command, windows_cpus=windows_cpus, retained_snapshot_path=snapshot_path, @@ -5702,7 +5661,6 @@ def run_kvm_worker(args: argparse.Namespace) -> int: snapshot_path = Path(temp_dir) / "snapshot" snapshot_capture = benchmark_snapshot_capture( args, - "kvm", boot_command, retained_snapshot_path=snapshot_path, ) @@ -5728,7 +5686,7 @@ def run_kvm_worker(args: argparse.Namespace) -> int: return 0 if args.suite == "snapshot": print("Benchmarking OpenVMM/KVM snapshot capture", flush=True) - result = benchmark_snapshot_capture(args, "kvm", boot_command) + result = benchmark_snapshot_capture(args, boot_command) print_snapshot_summary("kvm", result) print( KVM_SNAPSHOT_RESULT_PREFIX + json.dumps(result, separators=(",", ":")), @@ -6016,7 +5974,6 @@ def make_boot_command(memory_mib: int) -> list[str]: if run_snapshot: result = benchmark_snapshot_capture( args, - backend, boot_command, retained_snapshot_path=retained_snapshot_path, ) @@ -6519,7 +6476,6 @@ def make_whp_profile_command(memory_mib: int) -> list[str]: print("Benchmarking OpenVMM/WHP snapshot capture", flush=True) whp_snapshot = benchmark_snapshot_capture( args, - "whp", whp_command( boot_binaries["whp"], kernel, diff --git a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh.in b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh similarity index 98% rename from scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh.in rename to scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh index 0806e221b..5e7dab64c 100644 --- a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh.in +++ b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh @@ -6,8 +6,6 @@ fail() { exit "$code" } -@SELECT_CLOCKSOURCE@ -@VALIDATE_CLOCKSOURCE@ generation_id_before="$(/sbin/nvx-port-io read-generation-id 233 234)" || fail 48 [ "${#generation_id_before}" -eq 32 ] || fail 49 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 5f9478dcb..fc365042e 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -33,7 +33,6 @@ record_adversarial_openvmm_pid, smp_probe_script, snapshot_restore_command, - stable_clocksource_wait_script, workload_boot_command, ) from .benchmark import ( @@ -268,7 +267,6 @@ def capture_snapshot( command: Sequence[str], snapshot_path: Path, *, - backend: str, timeout: float, windows_cpus: set[int] | None = None, processors: int | None = None, @@ -283,7 +281,6 @@ def capture_snapshot( return _capture_snapshot( command, snapshot_path, - backend=backend, timeout=timeout, windows_cpus=windows_cpus, processors=processors, @@ -325,44 +322,6 @@ def _stage_script( ) -def _snapshot_core_script(backend: str) -> str: - if backend == "kvm": - select_clocksource = ( - "clock_tries=0\n" - "while ! grep -qw kvm-clock " - "/sys/devices/system/clocksource/clocksource0/available_clocksource " - "&& [ $clock_tries -lt 100 ]; do\n" - " sleep 0.05\n" - " clock_tries=$((clock_tries + 1))\n" - "done\n" - "grep -qw kvm-clock " - "/sys/devices/system/clocksource/clocksource0/available_clocksource " - "|| fail 46\n" - "echo kvm-clock >" - "/sys/devices/system/clocksource/clocksource0/current_clocksource" - ) - validate_clocksource = ( - '[ "$(cat /sys/devices/system/clocksource/clocksource0/' - 'current_clocksource)" = kvm-clock ] || fail 46' - ) - elif backend == "whp": - select_clocksource = stable_clocksource_wait_script() - validate_clocksource = ( - '[ "$(cat /sys/devices/system/clocksource/clocksource0/' - 'current_clocksource)" != tsc-early ] || fail 46' - ) - elif backend == "mshv": - select_clocksource = ":" - validate_clocksource = ":" - else: - raise ValueError(f"unsupported snapshot-core backend {backend!r}") - return ( - _read_script("snapshot-core.sh.in") - .replace("@SELECT_CLOCKSOURCE@", select_clocksource) - .replace("@VALIDATE_CLOCKSOURCE@", validate_clocksource) - ) - - def _output_lines(output: bytes) -> list[bytes]: return [line.removesuffix(b"\r") for line in output.splitlines()] @@ -1194,7 +1153,6 @@ def run_console_exit( capture_snapshot( [*boot_command, "--snapshot-destination", str(snapshot_path)], snapshot_path, - backend=backend, timeout=timeout, processors=processors, post_restore_script=_render_script( @@ -2400,7 +2358,6 @@ def run_smp_snapshot( capture_snapshot( [*boot_command, "--snapshot-destination", str(snapshot_path)], snapshot_path, - backend=backend, timeout=timeout, processors=processors, post_restore_script=smp_probe_script(processors, exit_guest=False), @@ -2602,7 +2559,6 @@ def run_restore_processors( capture_snapshot( [*boot_command, "--snapshot-destination", str(snapshot_path)], snapshot_path, - backend=backend, timeout=timeout, processors=boot_online, post_restore_script=script, @@ -2693,7 +2649,6 @@ def run_restore_memory( str(snapshot_path), ], snapshot_path, - backend=backend, timeout=timeout, processors=1, post_restore_script=_read_script("restore-memory.sh"), @@ -2792,7 +2747,7 @@ def run_snapshot_core( process, "/tmp/nvx-snapshot-core", "NVX_SNAPSHOT_CORE", - _snapshot_core_script(backend), + _read_script("snapshot-core.sh"), ) source = process.wait(timeout) if source.returncode != 0: diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index c3afb19bd..04bf9799a 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -1854,19 +1854,18 @@ def test_tcp_console_connect_closes_failed_connection(self): console.close() connected.close.assert_called_once_with() - def test_snapshot_core_script_selects_backend_clocksource(self): - kvm = microvm_tests._snapshot_core_script("kvm") - whp = microvm_tests._snapshot_core_script("whp") - mshv = microvm_tests._snapshot_core_script("mshv") - - self.assertIn("echo kvm-clock", kvm) - self.assertIn('current_clocksource)" != tsc-early', whp) - self.assertNotIn("@SELECT_CLOCKSOURCE@", mshv) - self.assertIn("/sbin/nvx-reseed", mshv) - self.assertIn("/sbin/nvx-reseed --sample", mshv) - self.assertIn("NVX-SNAPSHOT-GENERATION-ID-", mshv) - self.assertIn("NVX-SNAPSHOT-UUID-", mshv) - self.assertIn("NVX-SNAPSHOT-TEMP-ID-", mshv) + def test_snapshot_core_script_handles_no_clocksource(self): + # The time ABI fixes the clocksource on every backend, so snapshot-core + # neither selects nor waits for one. + script = microvm_tests._read_script("snapshot-core.sh") + + for text in ("clocksource", "kvm-clock", "tsc-early", "@SELECT", "@VALIDATE"): + self.assertNotIn(text, script) + self.assertIn("/sbin/nvx-reseed", script) + self.assertIn("/sbin/nvx-reseed --sample", script) + self.assertIn("NVX-SNAPSHOT-GENERATION-ID-", script) + self.assertIn("NVX-SNAPSHOT-UUID-", script) + self.assertIn("NVX-SNAPSHOT-TEMP-ID-", script) def test_smp_worker_requires_bounded_local_timer_progress(self): shell = _posix_shell() @@ -1930,58 +1929,6 @@ def test_smp_worker_requires_bounded_local_timer_progress(self): ["1", "101", "1"], ) - def test_snapshot_core_whp_waits_for_stable_clocksource(self): - shell = _posix_shell() - if shell is None: - self.skipTest("POSIX shell is unavailable") - clocksource_script = microvm_tests._snapshot_core_script("whp").split( - "generation_id_before=", 1 - )[0] - for ready_after, stable_source, expected_returncode, expected_waits in ( - (0, "tsc", 0, 0), - (2, "refined-jiffies", 0, 2), - (101, "tsc", 46, 100), - ): - with self.subTest( - ready_after=ready_after, - stable_source=stable_source, - ): - result = subprocess.run( - [shell, "-s"], - input=( - "clock_waits=0\n" - "cat() {\n" - f' if [ "$clock_waits" -ge {ready_after} ]; then\n' - f" echo {stable_source}\n" - " else\n" - " echo tsc-early\n" - " fi\n" - "}\n" - "sleep() { clock_waits=$((clock_waits + 1)); }\n" - "nvx_exit() {\n" - ' echo "NVX-CLOCKSOURCE-WAITS-$clock_waits"\n' - ' exit "$1"\n' - "}\n" - + clocksource_script.replace("nvx-exit", "nvx_exit") - + 'echo "NVX-CLOCKSOURCE-WAITS-$clock_waits"\n' - ), - text=True, - capture_output=True, - timeout=5, - check=False, - ) - - self.assertEqual( - result.returncode, - expected_returncode, - result.stdout + result.stderr, - ) - self.assertIn( - f"NVX-CLOCKSOURCE-WAITS-{expected_waits}\n", result.stdout - ) - if expected_returncode: - self.assertIn("NVX-SNAPSHOT-CORE-FAIL code=46", result.stdout) - def test_snapshot_core_waits_for_no_destination_marker_line_before_exit(self): events: list[tuple[str, bytes | str | None]] = [] marker = b"NVX-SNAPSHOT-NO-DESTINATION-OK" diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index ef9be7270..6f82bfe10 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -8896,7 +8896,6 @@ def read_chunk(*, timeout: float) -> tuple[str, bytes | None]: result = benchmark.capture_snapshot( ["openvmm"], snapshot, - backend="whp", processors=1, timeout=5, snapshot_profile=profiled, @@ -9043,7 +9042,6 @@ def close(self) -> None: result = benchmark.capture_snapshot( ["openvmm"], snapshot, - backend="mshv", processors=1, timeout=10, snapshot_profile=True, @@ -10298,7 +10296,6 @@ def test_snapshot_request_triggers_waiting_capture_controller(self): def test_prepare_snapshot_capture_stages_waiting_controller(self): script = benchmark.prepare_snapshot_capture_script( 4, - backend="whp", teardown_mode="guest-exit", network_gateway="10.0.0.1", ioapic_irq=10, @@ -10317,19 +10314,13 @@ def test_prepare_snapshot_capture_stages_waiting_controller(self): ) self.assertIn("IFS= read -r trigger\n", script) self.assertIn("echo NVX-SNAPSHOT-DISPATCHED\n", script) - self.assertIn('while [ "$(cat "$clock_path")" = tsc-early ]', script) - self.assertIn( - "SMP-CLOCKSOURCE-FAIL expected=stable actual=$current_clocksource", - script, - ) self.assertIn( "/sbin/nvx-snapshot\necho OPENVMM-SNAPSHOT-RESTORE-OK\nnvx-exit 0\n", script, ) host_terminated = benchmark.prepare_snapshot_capture_script( - 4, backend="kvm", teardown_mode="host-terminate" + 4, teardown_mode="host-terminate" ) - self.assertNotIn("clock_tries", host_terminated) self.assertNotIn("nvx-exit 0", host_terminated) self.assertTrue( script.endswith( @@ -10340,24 +10331,18 @@ def test_prepare_snapshot_capture_stages_waiting_controller(self): ) ) - def test_mshv_and_whp_capture_after_linux_leaves_tsc_early(self): - wait = benchmark.stable_clocksource_wait_script() - for backend, waits in (("mshv", True), ("whp", True), ("kvm", False)): - with self.subTest(backend=backend): - script = benchmark.prepare_snapshot_capture_script( - 1, backend=backend, teardown_mode="guest-exit" - ) - probe = script.split("<<'NVX_SMP_PROBE_SCRIPT'\n", 1)[1] - probe = probe.split("NVX_SMP_PROBE_SCRIPT\n", 1)[0] - self.assertEqual(probe.startswith(wait), waits) - self.assertEqual("SMP-CLOCKSOURCE-FAIL expected=stable" in probe, waits) - if waits: - # The wait and its check run before the probe completes, so - # the host requests the snapshot only after tsc-early is gone. - self.assertLess( - probe.index("SMP-CLOCKSOURCE-FAIL"), - probe.index("NVX-SMP-PROBE-OK"), - ) + def test_capture_does_not_wait_for_a_clocksource(self): + # The time ABI registers the tsc clocksource at device_initcall on + # every backend, so no capture waits for Linux to leave tsc-early. + script = benchmark.prepare_snapshot_capture_script( + 1, teardown_mode="guest-exit" + ) + probe = script.split("<<'NVX_SMP_PROBE_SCRIPT'\n", 1)[1] + probe = probe.split("NVX_SMP_PROBE_SCRIPT\n", 1)[0] + self.assertEqual(probe, benchmark.smp_probe_script(1, exit_guest=False)) + for text in ("clocksource", "tsc-early", "clock_tries"): + self.assertNotIn(text, script) + self.assertFalse(hasattr(benchmark, "stable_clocksource_wait_script")) def test_output_marker_must_be_a_complete_line(self): marker = benchmark.RESTORE_MARKER @@ -10766,9 +10751,9 @@ def test_snapshot_capture_runs_smp_probe_for_selected_count(self): "capture_snapshot", return_value=(1.0, 1.0, 1.0, 1024), ) as capture: - benchmark.benchmark_snapshot_capture(args, "whp", ["openvmm"]) + benchmark.benchmark_snapshot_capture(args, ["openvmm"]) - self.assertEqual(capture.call_args.kwargs["backend"], "whp") + self.assertNotIn("backend", capture.call_args.kwargs) self.assertEqual(capture.call_args.kwargs["processors"], 8) self.assertEqual(capture.call_args.kwargs["teardown_mode"], "guest-exit") @@ -11087,7 +11072,6 @@ def capture( with patch.object(benchmark, "capture_snapshot", side_effect=capture): result = benchmark.benchmark_snapshot_capture( args, - "kvm", ["openvmm"], retained_snapshot_path=retained, ) diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index d82437fca..458ffec3e 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -529,10 +529,7 @@ def test_exit_errors_lead_with_openvmm_fatal_time_abi_code(self): RuntimeError, f"^snapshot source {expected}" ): benchmark.capture_snapshot( - KVM_BOOT, - Path(temporary) / "snapshot", - backend="kvm", - timeout=5, + KVM_BOOT, Path(temporary) / "snapshot", timeout=5 ) log_path = Path(temporary) / "process.log" with patch.object( @@ -598,7 +595,6 @@ def test_capture_snapshot_classifies_a_failed_capture_check(self): benchmark.capture_snapshot( KVM_BOOT, Path(temporary) / "snapshot", - backend="kvm", timeout=5, ) From 263f9a4818a782b7c6695669a6db5935c8dbfd4d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 05:01:16 -0700 Subject: [PATCH 108/188] ci: remove the smp-lapic scenario and its benchmark precheck smp-lapic repeated the SMP probe with lapic=notscdeadline so that hosts whose guests used TSC-deadline timers still exercised the counting LAPIC. CI ran it twice: in the microVM correctness jobs and again as a 1/2/4/8-vCPU precheck in the benchmark action before acceptance (#286). The time ABI hides TSC-deadline on every backend, so every guest uses the one-shot counting LAPIC and smp-lapic is the smp scenario with an inert token. Remove the scenario, run_smp's force_lapic_timer, the benchmark action's precheck steps and their lapic-correctness output, and point the adversarial guest-isolation interrupt cases and the broker's scenario allowlist at smp. The spec removes the scenario outright, so test-microvm --scenario smp-lapic is gone too. The microVM correctness jobs already gate the required status, the development release, and performance persistence, so dropping the precheck weakens no gate; a test now pins that wiring, and the CI guide documents which jobs consume, gate, publish, and persist benchmark results. Platform benchmarks keep running in parallel with the correctness jobs. Refs #286 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/run-benchmark/action.yml | 25 ---------------- doc/benchmarks.md | 8 ++--- doc/ci.md | 10 ++++++- scripts/nvx_tools/adversarial_broker.py | 2 +- .../adversarial_cases/guest-isolation.json | 4 +-- scripts/nvx_tools/microvm_tests.py | 13 +++------ scripts/test_microvm_tests.py | 23 ++++++--------- scripts/test_nvx_tools.py | 29 +++++++++++++++++++ 8 files changed, 58 insertions(+), 56 deletions(-) diff --git a/.github/actions/run-benchmark/action.yml b/.github/actions/run-benchmark/action.yml index 712017137..4920a2831 100644 --- a/.github/actions/run-benchmark/action.yml +++ b/.github/actions/run-benchmark/action.yml @@ -41,31 +41,6 @@ runs: name: guest-artifacts path: build - - name: Check Linux counting-LAPIC timers - if: runner.os != 'Windows' - shell: bash - run: | - set -euo pipefail - python3 scripts/nvx.py test-microvm \ - --backend "${{ inputs.backend }}" \ - --scenario smp-lapic \ - --processors 1 2 4 8 \ - --timeout "${{ inputs.marker-timeout }}" \ - --output-dir "data/runs/${{ inputs.platform }}/lapic-correctness" - - - name: Check Windows counting-LAPIC timers - if: runner.os == 'Windows' - shell: powershell - run: | - $ErrorActionPreference = "Stop" - python scripts\nvx.py test-microvm ` - --backend "${{ inputs.backend }}" ` - --scenario smp-lapic ` - --processors 1 2 4 8 ` - --timeout "${{ inputs.marker-timeout }}" ` - --output-dir "data/runs/${{ inputs.platform }}/lapic-correctness" - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - - name: Run Linux acceptance test if: runner.os != 'Windows' shell: bash diff --git a/doc/benchmarks.md b/doc/benchmarks.md index db15c72c8..acacf5772 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -315,10 +315,10 @@ to advance and remain at least as large as the worker's observed value. The check remains strict even on a one-vCPU guest: falling back to the PIT after a failed LAPIC calibration is not success. A counter frozen at 12 can indicate Linux's `APIC timer disabled due to verification failure`; increasing the poll budget cannot repair it. -CI runs `test-microvm --scenario smp-lapic --processors 1 2 4 8` before acceptance. -This repeats the normal SMP probe with `lapic=notscdeadline`, covering the counting -LAPIC even on hosts that normally use TSC-deadline timers. The ordinary `smp` scenario -retains the default timer selection. +The time ABI hides the TSC-deadline timer on every backend, so every guest uses the one-shot +counting LAPIC, and the `smp` scenario that the microVM correctness jobs run at 1, 2, 4, and 8 +vCPUs covers it. There is no separate `lapic=notscdeadline` scenario or pre-benchmark LAPIC +gate. Captures do not wait for a clocksource: under the time ABI, Linux registers the `tsc` clocksource at `device_initcall`, before any guest work, so no capture can observe the transitional `tsc-early` window. diff --git a/doc/ci.md b/doc/ci.md index 8f0a73423..b5eb82eef 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -150,7 +150,15 @@ ready, benchmarks run in parallel with the NVX test layer and use any available runner in the matching backend pool. All three use virtual-machine performance series and the constrained eight-CPU affinity policy. Development releases and performance baseline updates still require every applicable test and benchmark -lane to pass. The workflow uses the read-only OpenVMM deploy key stored in the +lane to pass. +The microVM correctness jobs gate every use of benchmark results, so the +benchmark action runs no correctness scenario of its own (#286). `Required +status check` requires each `nvx-microvm-tests-*` job that the change schedules +to succeed. `Publish development release` and `Persist performance +baseline` run only on `dev` pushes in which every microVM test job succeeded or +was skipped. The pull-request `Performance regression gate` reads only the +platform jobs' results and publishes nothing. The counting LAPIC that the +benchmarks depend on is covered by the `smp` scenario of those jobs. The workflow uses the read-only OpenVMM deploy key stored in the `OPENVMM_DEPLOY_KEY` Actions secret to fetch the private submodule at its pinned commit. Shared guest binaries and development release packages move through short-lived workflow artifacts alongside the OpenVMM handoff and benchmark diff --git a/scripts/nvx_tools/adversarial_broker.py b/scripts/nvx_tools/adversarial_broker.py index cccd151af..beb678bac 100644 --- a/scripts/nvx_tools/adversarial_broker.py +++ b/scripts/nvx_tools/adversarial_broker.py @@ -45,7 +45,7 @@ "restore-processors", "sandbox-blocks", "scratch-snapshot", - "smp-lapic", + "smp", "smp-snapshot", "snapshot-core", "snapshot-tiers", diff --git a/scripts/nvx_tools/adversarial_cases/guest-isolation.json b/scripts/nvx_tools/adversarial_cases/guest-isolation.json index 90a57a3a6..ebfead2aa 100644 --- a/scripts/nvx_tools/adversarial_cases/guest-isolation.json +++ b/scripts/nvx_tools/adversarial_cases/guest-isolation.json @@ -43,7 +43,7 @@ }, { "id": "interrupt-topology-boundary", - "scenario": "smp-lapic", + "scenario": "smp", "description": "Exercise repeated interrupt and timer progress across the supported virtual processor topology.", "coverage": [ "lapic", @@ -56,7 +56,7 @@ }, { "id": "processor-interrupt-churn", - "scenario": "smp-lapic", + "scenario": "smp", "description": "Repeat bounded multi-processor scheduling, timer, and interrupt progress to expose CPU-side hangs or leaks.", "coverage": [ "cpu-churn", diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index fc365042e..f6cc7e171 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -81,7 +81,6 @@ "sandbox-blocks", "scratch-snapshot", "smp", - "smp-lapic", "smp-snapshot", "snapshot-core", "snapshot-tiers", @@ -1193,7 +1192,6 @@ def run_smp( memory_mib: int, timeout: float, log_path: Path, - force_lapic_timer: bool = False, ) -> None: command = workload_boot_command( executable, @@ -1201,7 +1199,7 @@ def run_smp( kernel, initrd, memory_mib, - "quiet loglevel=0" + (" lapic=notscdeadline" if force_lapic_timer else ""), + "quiet loglevel=0", processors=processors, ) run_guest_script( @@ -4361,12 +4359,10 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, output_dir=output_dir, ) - for scenario, force_lapic_timer in (("smp", False), ("smp-lapic", True)): - if scenario not in scenarios: - continue + if "smp" in scenarios: for processors in dict.fromkeys(args.processors): print( - f"Running microVM {scenario} correctness ({processors} vCPU) " + f"Running microVM smp correctness ({processors} vCPU) " f"on OpenVMM/{args.backend}" ) run_smp( @@ -4377,8 +4373,7 @@ def run(args: argparse.Namespace) -> int: processors, memory_mib=args.memory_mib, timeout=args.timeout, - log_path=output_dir / f"{scenario}-{processors}.log", - force_lapic_timer=force_lapic_timer, + log_path=output_dir / f"smp-{processors}.log", ) if "sandbox-blocks" in scenarios: print(f"Running microVM sandbox-block correctness on OpenVMM/{args.backend}") diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 04bf9799a..0c86ccd96 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2263,7 +2263,10 @@ def test_smp_uses_requested_processor_count(self): log_path=Path("smp-4.log"), ) - def test_smp_lapic_exercises_counting_timer_without_weakening_probe(self): + def test_smp_exercises_the_counting_lapic_without_a_duplicate_scenario(self): + # The time ABI hides TSC-deadline on every backend, so the ordinary + # smp scenario already exercises the one-shot counting LAPIC (#286). + self.assertNotIn("smp-lapic", microvm_tests.MICROVM_TEST_SCENARIOS) with patch.object(microvm_tests, "run_guest_script") as run: microvm_tests.run_smp( Path("openvmm"), @@ -2273,14 +2276,10 @@ def test_smp_lapic_exercises_counting_timer_without_weakening_probe(self): 4, memory_mib=128, timeout=60, - log_path=Path("smp-lapic-4.log"), - force_lapic_timer=True, + log_path=Path("smp-4.log"), ) command, script, marker = run.call_args.args - self.assertEqual( - command[command.index("--cmdline") + 1], - "quiet loglevel=0 lapic=notscdeadline", - ) + self.assertEqual(command[command.index("--cmdline") + 1], "quiet loglevel=0") self.assertEqual(script, benchmark.smp_probe_script(4)) self.assertEqual(marker, benchmark.SMP_PROBE_COMPLETION_MARKER) @@ -3513,7 +3512,7 @@ def test_runner_dispatches_selected_scenarios_once(self): args = argparse.Namespace( backend="whp", guest="alpine", - scenario=["smp", "smp", "smp-lapic", "smp-lapic"], + scenario=["smp", "smp"], processors=[2, 2, 8], memory_mib=128, timeout=60.0, @@ -3538,15 +3537,11 @@ def require(path: Path, _description: str) -> Path: run_lifecycle.assert_not_called() self.assertEqual( [entry.args[4] for entry in run_smp.call_args_list], - [2, 8, 2, 8], + [2, 8], ) self.assertEqual( [entry.kwargs["log_path"].name for entry in run_smp.call_args_list], - ["smp-2.log", "smp-8.log", "smp-lapic-2.log", "smp-lapic-8.log"], - ) - self.assertEqual( - [entry.kwargs["force_lapic_timer"] for entry in run_smp.call_args_list], - [False, False, True, True], + ["smp-2.log", "smp-8.log"], ) def test_runner_dispatches_public_managed_exec_configuration(self): diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 6f82bfe10..948f72a1f 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -2962,6 +2962,35 @@ def test_guest_artifact_restore_only_leaves_restore_and_scratch_steps_active(sel continue self.assertNotIn("restore-only", condition) + def test_benchmarks_rely_on_the_microvm_correctness_jobs(self): + # #286: the benchmark action ran a second smp-lapic gate before + # acceptance. The microVM correctness jobs already gate the required + # status, the development release, and performance persistence. + workflow = ( + BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" + ).read_text(encoding="utf-8") + action = ( + BuildConstants.REPO_ROOT + / ".github" + / "actions" + / "run-benchmark" + / "action.yml" + ).read_text(encoding="utf-8") + + self.assertNotIn("test-microvm", action) + self.assertNotIn("lapic-correctness", action) + for job_name in ci.REQUIRED_CI_MICROVM_TEST_JOBS: + with self.subTest(job=job_name): + self.assertIn(job_name, ci.REQUIRED_CI_RESULT_ENVIRONMENTS) + for consumer in ("release", "performance-persist"): + job = _workflow_job(workflow, consumer) + self.assertIn(f" - {job_name}\n", job) + self.assertIn( + f"needs.{job_name}.result == 'success' ||\n" + f" needs.{job_name}.result == 'skipped'", + job, + ) + def test_ci_runs_openvmm_tests_and_unit_tests_on_each_backend(self): workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" From 49e493c30ab7a1e7979fa065eb5050e87409def7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 05:54:49 -0700 Subject: [PATCH 109/188] ci: replace the restore TSC guards with the guest warp probe Under the time ABI the guest TSC is tsc_reliable, so Linux skips its CPU-online warp check and never logs the messages that the restore-processors guard grepped for. The spec bounds cross-vCPU skew at 1 us and enforces it with a CI warp probe instead, and host qualification gates on that guest-level measurement. Run the idle-inducing warp schedule, two rounds of /sbin/nvx-time-probe warp over every online CPU pair with all vCPUs halted for 1 s between them, after each smp boot, after every smp-snapshot restore, and after every restore-processors target. The idle gap lets a host without an invariant TSC correct guest TSC timing as idle host CPUs wake (#265). A probe failure prints NVX-WARP-PROBE-FAIL with its round and powers the guest off with status 97; the harness also checks the round count, the pair count, both 1 us bounds, stalls, and conclusiveness. smp-snapshot now captures one snapshot per requested processor count (1, 2, 4, and 8 by default) instead of only 2 vCPUs, and restores the first snapshot twice to keep the reuse check. Removed, all replaced by the probe: - the restore-tsc-sync scenario and its clearcpuid=tsc_adjust option (restore-tsc-sync.sh); - the fresh-boot TSC control (tsc-sync-control.sh.in, run_fresh_boot_tsc_control, _tsc_control_verdict) and the host invariant-TSC note, which host qualification records as evidence; - the dmesg TSC greps and the clocksource echo in restore-processors.sh, which now powers the guest off when a check fails; - the vp_set and virt_mshv TSC log filters, replaced by OpenVMM's time ABI rate report. Lands with the OpenVMM time ABI. Refs #211, #265 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 60 +- .../restore-processors.sh | 17 +- .../microvm_test_scripts/restore-tsc-sync.sh | 9 - .../tsc-sync-control.sh.in | 45 -- scripts/nvx_tools/microvm_tests.py | 339 ++++---- scripts/test_microvm_tests.py | 736 +++++++----------- 6 files changed, 457 insertions(+), 749 deletions(-) delete mode 100644 scripts/nvx_tools/microvm_test_scripts/restore-tsc-sync.sh delete mode 100644 scripts/nvx_tools/microvm_test_scripts/tsc-sync-control.sh.in diff --git a/doc/ci.md b/doc/ci.md index b5eb82eef..070a5733b 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -29,11 +29,21 @@ fails the scenario at once with the guest's code and detail. An OpenVMM exit status of 193, 194, or 195 is reported as the guest's time ABI conformance, runtime-violation, or restore-repair power-off, together with the event that preceded it, instead of as a generic exit status. -The restore-processor scenario also rejects Linux TSC instability diagnostics, -even if the requested CPUs came online, so clock skew cannot silently pass by -falling back to a different clocksource. After the 1/2/4/8-CPU restores, it -restores the same snapshot once without `--restore-processors`. Every restore -runs with OpenVMM lifecycle profiling and must report exactly one +The `smp`, `smp-snapshot`, and `restore-processors` scenarios run the guest +warp probe, `/sbin/nvx-time-probe warp --bound-ns 1000`, over every pair of +online CPUs after each boot or restore, in two rounds with all vCPUs halted +for 1 s between them, so that a host without an invariant TSC corrects the +guest TSC as idle host CPUs wake (#265). Every pair in every round must stay +within the time ABI's 1 µs +[cross-vCPU skew bound](design/time-abi.md#cross-vcpu-skew-bound) for both the +backward TSC step and the ping-pong offset; a stalled pair or an inconclusive +measurement also fails. `smp` boots each requested processor +count. `smp-snapshot` captures one snapshot per requested count and restores +it once, and the first snapshot a second time to prove that a restore leaves it +reusable. `restore-processors` captures one boot-online CPU with capacity 8 and +probes the CPUs that each restore target activated. After the 1/2/4/8-CPU +restores, it restores the same snapshot once without `--restore-processors`. +Every restore runs with OpenVMM lifecycle profiling and must report exactly one `startup.vp_thread_bind` record. Its `startup.vp_bind_*` records must show that an explicit MSHV target binds exactly VPs `0..N-1`, while untargeted MSHV restores and all KVM and WHP restores bind the full capacity. @@ -41,18 +51,12 @@ Captures do not wait for a clocksource: the time ABI registers `tsc` at `device_initcall` on every backend, so the transitional `tsc-early` window that once let the clocksource watchdog compare `tsc-early` with jiffies across a restore (#253) never reaches the guest's userspace. -A restore fails as soon as its guest prints `NVX-RESTORE-PROCESSORS-FAIL`, -rather than waiting for the phase timeout. Restore logs also record OpenVMM's -`adjusted restored vCPU TSC` event for each VP, which includes the applied -snapshot downtime, and its `aligning restored AP TSCs to the BSP` event, which -reports how many created MSHV APs were aligned. When the guest reports -`unstable-tsc`, the harness boots a never-restored eight-vCPU guest with the -same forced warp check and reactivates each AP 20 times. The error then states -whether this control also found TSC instability, which points to host or -hypervisor clock skew rather than restore alignment, and whether the host CPU -exposes an invariant TSC. The control log is kept as -`restore-processors-tsc-control.log`. The control only classifies the -failure; the restore still fails. +A restore fails as soon as its guest prints `NVX-RESTORE-PROCESSORS-FAIL` or +`NVX-WARP-PROBE-FAIL`, rather than waiting for the phase timeout; a failed +probe also powers the guest off with status 97. Restore-processor logs record +OpenVMM's `time ABI rates declared` event, which reports the identity MSR +route, the TSC synchronization method, the native and declared TSC rates, and +the rate deviation from the snapshot. Every job that uses the `validate-runner` action first qualifies its runner for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm @@ -73,21 +77,13 @@ is why the probe schedule includes idle gaps. Runner labels do not encode the generation; per-PR CI captures and restores on one runner, so generations never mix. -The `restore-tsc-sync` scenario repeats the restore-processor sequence with -the test-only kernel option `clearcpuid=tsc_adjust`. Linux normally skips its -cross-CPU TSC warp test when `IA32_TSC_ADJUST` is available and consistent -within a package. This scenario verifies that the feature is masked, forcing -the live CPU-online check even on those hosts, while retaining the existing -TSC-instability guard. It does not force a fallback clocksource or retry failed -restores. Its logs are kept in a separate `restore-tsc-sync` subdirectory. -Run it alone on Windows with: - -```powershell -python scripts\nvx.py test-microvm --backend whp --scenario restore-tsc-sync -``` - -This regression targets the WHP clock instability tracked in #19; a passing -frozen-counter check is not sufficient to validate a fix. +The warp probe replaced the `restore-tsc-sync` scenario, its test-only +`clearcpuid=tsc_adjust` kernel option, the guest's scan of the kernel log for +TSC warp and instability messages, and the fresh-boot TSC control that +classified those failures (#211, #265). Under the time ABI the guest's TSC is +`tsc_reliable`, so Linux skips its CPU-online warp check and never logs the +messages that guard looked for; the probe measures the 1 µs bound on every CPU +pair instead. The `console-exit` scenario delays host console reads for two seconds after snapshot restore to exercise output backpressure. For each requested processor diff --git a/scripts/nvx_tools/microvm_test_scripts/restore-processors.sh b/scripts/nvx_tools/microvm_test_scripts/restore-processors.sh index 2001ad3a2..54aa517ef 100644 --- a/scripts/nvx_tools/microvm_test_scripts/restore-processors.sh +++ b/scripts/nvx_tools/microvm_test_scripts/restore-processors.sh @@ -1,4 +1,11 @@ set -eu +on_exit() { + status=$? + if [ "$status" -ne 0 ]; then + nvx-exit "$status" + fi +} +trap on_exit EXIT online_count="$(getconf _NPROCESSORS_ONLN)" case "$online_count" in 1 | 2 | 4 | 8) ;; @@ -24,14 +31,4 @@ while [ "$cpu" -lt "$online_count" ]; do echo "NVX-RESTORE-PROCESSOR-OK count=$online_count cpu=$cpu" cpu=$((cpu + 1)) done -kernel_log="$(dmesg)" -case "$kernel_log" in - *"TSC warp"* | *"Marking TSC unstable"* | *"TSC found unstable"*) - printf '%s\n' "$kernel_log" - echo "NVX-RESTORE-PROCESSORS-FAIL unstable-tsc" - exit 95 - ;; -esac -clocksource="$(cat /sys/devices/system/clocksource/clocksource0/current_clocksource)" -echo "NVX-RESTORE-CLOCKSOURCE-OK source=$clocksource" echo "NVX-RESTORE-PROCESSORS-OK count=$online_count" diff --git a/scripts/nvx_tools/microvm_test_scripts/restore-tsc-sync.sh b/scripts/nvx_tools/microvm_test_scripts/restore-tsc-sync.sh deleted file mode 100644 index d4ae950b0..000000000 --- a/scripts/nvx_tools/microvm_test_scripts/restore-tsc-sync.sh +++ /dev/null @@ -1,9 +0,0 @@ -set -eu -cpuinfo="$(cat /proc/cpuinfo)" -case "$cpuinfo" in - *tsc_adjust*) - echo "NVX-RESTORE-PROCESSORS-FAIL tsc-adjust-still-enabled" - exit 96 - ;; -esac -echo "NVX-RESTORE-TSC-SYNC-CHECK-ENABLED" diff --git a/scripts/nvx_tools/microvm_test_scripts/tsc-sync-control.sh.in b/scripts/nvx_tools/microvm_test_scripts/tsc-sync-control.sh.in deleted file mode 100644 index e7dacb6b6..000000000 --- a/scripts/nvx_tools/microvm_test_scripts/tsc-sync-control.sh.in +++ /dev/null @@ -1,45 +0,0 @@ -set -eu -fail() { - code="$1" - trap - EXIT - echo "NVX-TSC-CONTROL-FAIL code=$code" - nvx-exit "$code" - exit "$code" -} -trap 'status=$?; if [ "$status" -ne 0 ]; then fail "$status"; fi' EXIT - -tsc_unstable() { - kernel_log="$(dmesg)" || fail "$?" - case "$kernel_log" in - *"TSC warp"* | *"Marking TSC unstable"* | *"TSC found unstable"*) return 0 ;; - esac - return 1 -} - -online="$(getconf _NPROCESSORS_ONLN)" -[ "$online" -eq @PROCESSORS@ ] || fail 60 -case "$(cat /proc/cpuinfo)" in - *tsc_adjust*) fail 61 ;; -esac -# Boot activated every AP once; each round reactivates them against CPU 0. -activations=$((online - 1)) -round=0 -while [ "$round" -lt @ROUNDS@ ] && ! tsc_unstable; do - cpu=1 - while [ "$cpu" -lt "$online" ]; do - echo 0 >"/sys/devices/system/cpu/cpu$cpu/online" || fail 62 - echo 1 >"/sys/devices/system/cpu/cpu$cpu/online" || fail 63 - activations=$((activations + 1)) - cpu=$((cpu + 1)) - done - round=$((round + 1)) -done -if tsc_unstable; then - dmesg | grep -E "TSC|tsc:" || true - echo "NVX-TSC-CONTROL-RESULT unstable activations=$activations" -else - echo "NVX-TSC-CONTROL-RESULT stable activations=$activations" -fi -echo NVX-TSC-CONTROL-DONE -trap - EXIT -nvx-exit 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index f6cc7e171..3ef3f2d9b 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -26,6 +26,7 @@ SNAPSHOT_PROFILE_ENV, GuestCommandResult, GuestFailureReported, + contains_output_line, measure_once, parse_snapshot_profile_line, positive_float, @@ -58,7 +59,14 @@ from .guests import GUEST_NAMES, GuestDescriptor, guest_descriptor from .managed_exec_tests import run_managed_exec_configuration from .openvmm_process import OpenvmmProcess, TcpConsole -from .time_abi import describe_exit_status +from .time_abi import ( + WARP_PROBE_COMPLETION_MARKER, + WARP_PROBE_FAILURE_MARKER, + check_warp_probe, + describe_exit_status, + warp_probe_script, + warp_rounds, +) MICROVM_TEST_SCENARIOS = ( "console-exit", @@ -77,7 +85,6 @@ "network-snapshot", "restore-memory", "restore-processors", - "restore-tsc-sync", "sandbox-blocks", "scratch-snapshot", "smp", @@ -158,17 +165,11 @@ GUEST_BOOT_COMPLETION_MARKER = b"NVX-GUEST-BOOT-CHECK-OK" GUEST_IDENTITY_COMPLETION_MARKER = b"NVX-GUEST-IDENTITY-OK" RESTORE_PROCESSORS_FAILURE_MARKER = b"NVX-RESTORE-PROCESSORS-FAIL" -RESTORE_UNSTABLE_TSC_FAILURE = "NVX-RESTORE-PROCESSORS-FAIL unstable-tsc" -# Records each VP's applied restore downtime and which restored MSHV APs were -# aligned to the BSP counter. -RESTORE_TSC_LOG_FILTER = ( - "off,vmm_core::partition_unit::vp_set::tsc=debug,virt_mshv::x86_64::tsc=info" -) -TSC_CONTROL_PROCESSORS = 8 -TSC_CONTROL_ROUNDS = 20 -TSC_CONTROL_COMPLETION_MARKER = b"NVX-TSC-CONTROL-DONE" -TSC_CONTROL_RESULT_PREFIX = "NVX-TSC-CONTROL-RESULT " -HOST_CPUINFO = Path("/proc/cpuinfo") +# Records the time ABI rates and restore clock report of each restore. These +# records are written before the restored VPs run, so they never split a guest +# console line; targets that log while the guest runs, such as virt_kvm's +# hidden-MSR #GPs, could split a marker in the merged console stream. +TIME_ABI_RESTORE_LOG_FILTER = "off,openvmm_core::worker::dispatch::time_abi=info" OUTCOME_TOP_LEVEL_FIELDS = frozenset( { "schema_version", @@ -1202,13 +1203,27 @@ def run_smp( "quiet loglevel=0", processors=processors, ) - run_guest_script( + result = run_guest_script( command, - smp_probe_script(processors), - SMP_PROBE_COMPLETION_MARKER, + smp_probe_script(processors, exit_guest=False) + + warp_probe_script() + + "nvx-exit 0\n", + WARP_PROBE_COMPLETION_MARKER, timeout=timeout, log_path=log_path, ) + output = result["text"].encode("utf-8") + if not contains_output_line(output, SMP_PROBE_COMPLETION_MARKER): + raise RuntimeError( + f"{processors}-vCPU guest finished without the SMP probe marker " + f"{SMP_PROBE_COMPLETION_MARKER.decode()!r}" + ) + check_warp_probe( + result["text"], + cpus=processors, + context=f"{processors}-vCPU boot", + rounds=warp_rounds(processors), + ) def run_virtio_net( @@ -2331,56 +2346,104 @@ def run_sandbox_blocks( ) +def _measure_restore( + command: Sequence[str], + *, + context: str, + environment: dict[str, str], + timeout: float, + log_path: Path, + failure_marker: bytes, +) -> bytes: + """Restore a snapshot whose post-restore script ends with the restore + marker and a queued guest exit, and return the complete output.""" + try: + measure_once( + command, + environment=environment, + timeout=timeout, + marker=RESTORE_MARKER, + marker_must_be_line=True, + guest_exit_prequeued=True, + log_path=log_path, + failure_marker=failure_marker, + ) + except GuestFailureReported as error: + raise RuntimeError( + f"{context}: guest reported {error.line}\n" + f"--- OpenVMM output ---\n{error.output_tail}" + ) from error + return log_path.read_bytes() + + +def _check_restore_warp(output: bytes, *, processors: int, context: str) -> None: + """Validate the warp probe that a post-restore script ran.""" + if not contains_output_line(output, WARP_PROBE_COMPLETION_MARKER): + raise RuntimeError(f"{context}: the guest did not finish its warp probe") + check_warp_probe( + output.decode("utf-8", "replace"), + cpus=processors, + context=context, + rounds=warp_rounds(processors), + ) + + def run_smp_snapshot( executable: Path, kernel: Path, initrd: Path, backend: str, + processor_counts: list[int], *, memory_mib: int, timeout: float, output_dir: Path, ) -> None: - processors = 2 - with tempfile.TemporaryDirectory(prefix="nvx-smp-snapshot-") as temporary: - snapshot_path = Path(temporary) / "snapshot" - boot_command = workload_boot_command( - executable, - backend, - kernel, - initrd, - memory_mib, - "quiet loglevel=0", - processors=processors, - ) - capture_snapshot( - [*boot_command, "--snapshot-destination", str(snapshot_path)], - snapshot_path, - timeout=timeout, - processors=processors, - post_restore_script=smp_probe_script(processors, exit_guest=False), - log_path=output_dir / "smp-snapshot-capture.log", - ) - fingerprint = _snapshot_fingerprint(snapshot_path) - for restore_index in range(2): - measure_once( - snapshot_restore_command( - executable, - backend, - snapshot_path, - processors=processors, - ), - environment=_restore_environment(), + counts = list(dict.fromkeys(processor_counts)) + for processors in counts: + # The first snapshot is restored twice to prove that a restore leaves + # it reusable; the warp probe runs after every restore. + restores = 2 if processors == counts[0] else 1 + with tempfile.TemporaryDirectory(prefix="nvx-smp-snapshot-") as temporary: + snapshot_path = Path(temporary) / "snapshot" + boot_command = workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + processors=processors, + ) + capture_snapshot( + [*boot_command, "--snapshot-destination", str(snapshot_path)], + snapshot_path, timeout=timeout, - marker=RESTORE_MARKER, - marker_must_be_line=True, - guest_exit_prequeued=True, - log_path=output_dir / f"smp-snapshot-restore-{restore_index}.log", + processors=processors, + post_restore_script=smp_probe_script(processors, exit_guest=False) + + warp_probe_script(), + log_path=output_dir / f"smp-snapshot-{processors}-capture.log", ) - if _snapshot_fingerprint(snapshot_path) != fingerprint: - raise RuntimeError( - f"SMP restore {restore_index} modified snapshot artifacts" + fingerprint = _snapshot_fingerprint(snapshot_path) + for restore_index in range(restores): + context = f"{processors}-vCPU SMP restore {restore_index}" + output = _measure_restore( + snapshot_restore_command( + executable, + backend, + snapshot_path, + processors=processors, + ), + context=context, + environment=_restore_environment(), + timeout=timeout, + log_path=output_dir + / f"smp-snapshot-{processors}-restore-{restore_index}.log", + failure_marker=WARP_PROBE_FAILURE_MARKER, ) + _check_restore_warp(output, processors=processors, context=context) + if _snapshot_fingerprint(snapshot_path) != fingerprint: + raise RuntimeError(f"{context} modified snapshot artifacts") def _restore_vp_bindings(output: bytes) -> list[int]: @@ -2431,94 +2494,6 @@ def _check_restore_vp_bindings( ) -def _tsc_control_verdict(text: str) -> str: - """Summarize the fresh-boot TSC control result printed by the guest.""" - for line in text.splitlines(): - line = line.removesuffix("\r") - if line.startswith(TSC_CONTROL_RESULT_PREFIX): - fields = line.removeprefix(TSC_CONTROL_RESULT_PREFIX) - state, _, activations = fields.partition(" activations=") - if state == "stable" and activations.isdecimal(): - return ( - "fresh-boot TSC control: no TSC instability across " - f"{activations} CPU activations without snapshot restore" - ) - if state == "unstable" and activations.isdecimal(): - return ( - "fresh-boot TSC control: Linux also found TSC instability " - f"without snapshot restore after {activations} CPU activations" - ) - break - return "fresh-boot TSC control did not report a result" - - -def _host_invariant_tsc_note(cpuinfo: Path = HOST_CPUINFO) -> str: - """Report whether a Linux host CPU exposes an invariant TSC. - - Guests on a host without one intermittently see cross-vCPU TSC warps - whether or not they were restored (#211). - """ - try: - text = cpuinfo.read_text(encoding="utf-8") - except OSError: - return "" - for line in text.splitlines(): - name, _, flags = line.partition(":") - if name.strip() == "flags": - if "nonstop_tsc" in flags.split(): - return "host CPU exposes an invariant TSC (nonstop_tsc)\n" - return ( - "host CPU does not expose an invariant TSC (nonstop_tsc); guests " - "on this host intermittently see cross-vCPU TSC warps\n" - ) - return "" - - -def run_fresh_boot_tsc_control( - executable: Path, - kernel: Path, - initrd: Path, - backend: str, - *, - memory_mib: int, - timeout: float, - log_path: Path, -) -> str: - """Check whether a never-restored guest reproduces a restore TSC failure. - - The guest boots every processor with Linux's cross-CPU TSC warp check - forced, then repeatedly reactivates each AP against CPU 0. The result only - classifies the failure that triggered it. - """ - command = workload_boot_command( - executable, - backend, - kernel, - initrd, - memory_mib, - "quiet loglevel=0 clearcpuid=tsc_adjust", - processors=TSC_CONTROL_PROCESSORS, - ) - script = _render_script( - "tsc-sync-control.sh.in", - PROCESSORS=str(TSC_CONTROL_PROCESSORS), - ROUNDS=str(TSC_CONTROL_ROUNDS), - ) - try: - result = run_guest_script( - command, - script, - TSC_CONTROL_COMPLETION_MARKER, - timeout=timeout, - log_path=log_path, - ) - except Exception as error: - # The control only annotates the restore failure that triggered it. - summary = str(error).splitlines()[0] if str(error) else type(error).__name__ - return f"fresh-boot TSC control did not complete: {summary}" - return _tsc_control_verdict(result["text"]) - - def run_restore_processors( executable: Path, kernel: Path, @@ -2529,19 +2504,17 @@ def run_restore_processors( memory_mib: int, timeout: float, output_dir: Path, - check_tsc_sync: bool = False, ) -> None: capacity = 8 boot_online = 1 cmdline = f"quiet loglevel=0 maxcpus={boot_online}" - script = _read_script("restore-processors.sh") - if check_tsc_sync: - cmdline += " clearcpuid=tsc_adjust" - script = _read_script("restore-tsc-sync.sh") + script + # The warp probe replaces the guest TSC warp guard: it measures the skew + # between every pair of CPUs, including the ones the restore activated. + script = _read_script("restore-processors.sh") + warp_probe_script() # The VP-binding lifecycle records identify the VPs that each restore # instantiates without changing restore behavior. environment = _restore_environment() - environment["OPENVMM_LOG"] = RESTORE_TSC_LOG_FILTER + environment["OPENVMM_LOG"] = TIME_ABI_RESTORE_LOG_FILTER environment[SNAPSHOT_PROFILE_ENV] = "1" with tempfile.TemporaryDirectory(prefix="nvx-restore-processors-") as temporary: snapshot_path = Path(temporary) / "snapshot" @@ -2568,44 +2541,30 @@ def run_restore_processors( for target in targets: name = "untargeted" if target is None else str(target) online = boot_online if target is None else target + output = _measure_restore( + snapshot_restore_command( + executable, + backend, + snapshot_path, + processors=capacity, + restore_processors=target, + ), + context=_restore_label(target), + environment=environment, + timeout=timeout, + log_path=output_dir / f"restore-processors-{name}.log", + failure_marker=RESTORE_PROCESSORS_FAILURE_MARKER, + ) marker = f"NVX-RESTORE-PROCESSORS-OK count={online}".encode() - log_path = output_dir / f"restore-processors-{name}.log" - try: - measure_once( - snapshot_restore_command( - executable, - backend, - snapshot_path, - processors=capacity, - restore_processors=target, - ), - environment=environment, - timeout=timeout, - marker=marker, - marker_must_be_line=True, - guest_exit_prequeued=True, - log_path=log_path, - failure_marker=RESTORE_PROCESSORS_FAILURE_MARKER, - ) - except GuestFailureReported as error: - verdict = "" - if error.line == RESTORE_UNSTABLE_TSC_FAILURE: - verdict = run_fresh_boot_tsc_control( - executable, - kernel, - initrd, - backend, - memory_mib=memory_mib, - timeout=timeout, - log_path=output_dir / "restore-processors-tsc-control.log", - ) - verdict += "\n" + _host_invariant_tsc_note() + if not contains_output_line(output, marker): raise RuntimeError( - f"{_restore_label(target)}: guest reported {error.line}\n" - f"{verdict}--- OpenVMM output ---\n{error.output_tail}" - ) from error + f"{_restore_label(target)} did not report {marker.decode()!r}" + ) + _check_restore_warp( + output, processors=online, context=_restore_label(target) + ) _check_restore_vp_bindings( - log_path.read_bytes(), + output, backend, target=target, capacity=capacity, @@ -4404,6 +4363,7 @@ def run(args: argparse.Namespace) -> int: kernel, initrd, args.backend, + args.processors, memory_mib=args.memory_mib, timeout=args.timeout, output_dir=output_dir, @@ -4422,21 +4382,6 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, output_dir=output_dir, ) - if "restore-tsc-sync" in scenarios: - print(f"Running microVM restore TSC synchronization on OpenVMM/{args.backend}") - tsc_output_dir = output_dir / "restore-tsc-sync" - tsc_output_dir.mkdir(parents=True, exist_ok=True) - run_restore_processors( - executable, - kernel, - initrd, - args.backend, - args.processors, - memory_mib=args.memory_mib, - timeout=args.timeout, - output_dir=tsc_output_dir, - check_tsc_sync=True, - ) if "restore-memory" in scenarios: print(f"Running microVM restore-memory correctness on OpenVMM/{args.backend}") run_restore_memory( diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 0c86ccd96..1cd284a62 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -24,6 +24,7 @@ managed_exec_tests, microvm_tests, openvmm_process, + time_abi, ) from nvx_tools.build_constants import ( # noqa: E402 BuildConstants, @@ -80,19 +81,41 @@ def _restore_target(command: list[str]) -> int | None: return int(command[command.index("--restore-processors") + 1]) +def _warp_probe_output(cpus: int, *, offset_ns: int = 40) -> bytes: + """Return the console output of the idle-inducing warp schedule.""" + pairs = cpus * (cpus - 1) // 2 + probe_round = ( + f"NVX-TIME-PROBE warp max_backward_cycles=0 max_backward_ns=0 " + f"max_abs_offset_ns={offset_ns} pairs={pairs} verdict=PASS bound_ns=1000\r\n" + "NVX-TIME-PROBE warp-detail max_abs_offset_cycles=1.0 " + "max_uncertainty_ns=30 max_skew_bound_ns=50 total_warps=0 " + f"inconsistent_pairs=0 stalled_pairs=0 cpus=0-{cpus - 1} duration_ms=100 " + "tsc_hz=2793437000 tsc_hz_source=cpuid conclusive=1\r\n" + ) + return (probe_round * time_abi.warp_rounds(cpus) + "NVX-WARP-PROBE-OK\r\n").encode() + + def _restore_processors_measure( backend: str, *, mshv_prefix: bool = True, + warp_offset_ns: int = 40, ) -> MagicMock: def measure(command: list[str], **kwargs: object) -> None: target = _restore_target(command) + online = 1 if target is None else target vp_count = 8 if mshv_prefix and backend == "mshv" and target is not None: vp_count = target log_path = cast(Path, kwargs["log_path"]) log_path.parent.mkdir(parents=True, exist_ok=True) - log_path.write_bytes(_vp_binding_profile(list(range(vp_count)))) + log_path.write_bytes( + f"NVX-RESTORE-PROCESSORS-OK count={online}\r\n".encode() + + _warp_probe_output(online, offset_ns=warp_offset_ns) + + benchmark.RESTORE_MARKER + + b"\r\n" + + _vp_binding_profile(list(range(vp_count))) + ) return MagicMock(side_effect=measure) @@ -2221,6 +2244,9 @@ def test_lifecycle_script_exits_on_unexpected_command_failure(self): ) def test_smp_uses_requested_processor_count(self): + text = ( + b"NVX-SMP-PROBE-OK\r\n" + _warp_probe_output(4) + b"nvx-exit 0\r\n" + ).decode() with ( patch.object( microvm_tests, @@ -2232,7 +2258,9 @@ def test_smp_uses_requested_processor_count(self): "smp_probe_script", return_value="probe\n", ) as smp_probe_script, - patch.object(microvm_tests, "run_guest_script") as run_guest_script, + patch.object( + microvm_tests, "run_guest_script", return_value={"text": text} + ) as run_guest_script, ): microvm_tests.run_smp( Path("openvmm"), @@ -2254,20 +2282,52 @@ def test_smp_uses_requested_processor_count(self): "quiet loglevel=0", processors=4, ) - smp_probe_script.assert_called_once_with(4) + smp_probe_script.assert_called_once_with(4, exit_guest=False) run_guest_script.assert_called_once_with( ["openvmm", "boot"], - "probe\n", - benchmark.SMP_PROBE_COMPLETION_MARKER, + "probe\n" + time_abi.warp_probe_script() + "nvx-exit 0\n", + time_abi.WARP_PROBE_COMPLETION_MARKER, timeout=90, log_path=Path("smp-4.log"), ) + def test_smp_requires_the_probe_marker_and_a_passing_warp_probe(self): + cases = ( + (_warp_probe_output(4).decode(), "without the SMP probe marker"), + ( + "NVX-SMP-PROBE-OK\n" + _warp_probe_output(2).decode(), + "measured 1 CPU pairs instead of 6", + ), + ( + "NVX-SMP-PROBE-OK\n" + _warp_probe_output(4, offset_ns=1001).decode(), + "max_abs_offset_ns=1001 exceeds 1000", + ), + ) + for text, message in cases: + with self.subTest(message=message): + with patch.object( + microvm_tests, "run_guest_script", return_value={"text": text} + ): + with self.assertRaisesRegex(RuntimeError, message): + microvm_tests.run_smp( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + 4, + memory_mib=128, + timeout=60, + log_path=Path("smp-4.log"), + ) + def test_smp_exercises_the_counting_lapic_without_a_duplicate_scenario(self): # The time ABI hides TSC-deadline on every backend, so the ordinary # smp scenario already exercises the one-shot counting LAPIC (#286). self.assertNotIn("smp-lapic", microvm_tests.MICROVM_TEST_SCENARIOS) - with patch.object(microvm_tests, "run_guest_script") as run: + text = (b"NVX-SMP-PROBE-OK\r\n" + _warp_probe_output(4)).decode() + with patch.object( + microvm_tests, "run_guest_script", return_value={"text": text} + ) as run: microvm_tests.run_smp( Path("openvmm"), Path("vmlinux"), @@ -2280,8 +2340,10 @@ def test_smp_exercises_the_counting_lapic_without_a_duplicate_scenario(self): ) command, script, marker = run.call_args.args self.assertEqual(command[command.index("--cmdline") + 1], "quiet loglevel=0") - self.assertEqual(script, benchmark.smp_probe_script(4)) - self.assertEqual(marker, benchmark.SMP_PROBE_COMPLETION_MARKER) + self.assertTrue( + script.startswith(benchmark.smp_probe_script(4, exit_guest=False)) + ) + self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) def test_virtio_net_uses_portable_endpoint_policy(self): with ( @@ -2707,7 +2769,13 @@ def test_sandbox_blocks_use_fixed_roles_and_access(self): microvm_tests.SANDBOX_BLOCKS_COMPLETION_MARKER, ) - def test_smp_snapshot_reruns_probe_and_checks_two_restores(self): + def test_smp_snapshot_runs_the_warp_probe_after_each_restore(self): + def measure(command: list[str], **kwargs: object) -> None: + processors = int(command[command.index("--processors") + 1]) + log_path = cast(Path, kwargs["log_path"]) + log_path.parent.mkdir(parents=True, exist_ok=True) + log_path.write_bytes(_warp_probe_output(processors)) + with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) / "logs" with ( @@ -2720,9 +2788,11 @@ def test_smp_snapshot_reruns_probe_and_checks_two_restores(self): microvm_tests, "smp_probe_script", return_value="probe\n", - ), + ) as smp_probe_script, patch.object(microvm_tests, "capture_snapshot") as capture_snapshot, - patch.object(microvm_tests, "measure_once") as measure_once, + patch.object( + microvm_tests, "measure_once", side_effect=measure + ) as measure_once, patch.object( microvm_tests, "_snapshot_fingerprint", @@ -2734,30 +2804,92 @@ def test_smp_snapshot_reruns_probe_and_checks_two_restores(self): Path("vmlinux"), Path("initrd"), "whp", + [1, 2, 2, 4, 8], memory_mib=128, timeout=60, output_dir=output_dir, ) + logs = sorted(path.name for path in output_dir.iterdir()) - self.assertEqual(capture_snapshot.call_args.kwargs["processors"], 2) self.assertEqual( - capture_snapshot.call_args.kwargs["post_restore_script"], - "probe\n", + [call.kwargs["processors"] for call in capture_snapshot.call_args_list], + [1, 2, 4, 8], + ) + self.assertEqual( + [call.args for call in smp_probe_script.call_args_list], + [(1,), (2,), (4,), (8,)], ) - self.assertEqual(measure_once.call_count, 2) self.assertTrue( all( - entry.kwargs["guest_exit_prequeued"] - for entry in measure_once.call_args_list + call.kwargs["post_restore_script"] + == "probe\n" + time_abi.warp_probe_script() + for call in capture_snapshot.call_args_list ) ) - self.assertTrue( - all( - entry.kwargs["marker"] == benchmark.RESTORE_MARKER - for entry in measure_once.call_args_list + # The first snapshot is restored twice. + self.assertEqual( + [ + int(call.args[0][call.args[0].index("--processors") + 1]) + for call in measure_once.call_args_list + ], + [1, 1, 2, 4, 8], + ) + for entry in measure_once.call_args_list: + self.assertTrue(entry.kwargs["guest_exit_prequeued"]) + self.assertEqual(entry.kwargs["marker"], benchmark.RESTORE_MARKER) + self.assertEqual( + entry.kwargs["failure_marker"], time_abi.WARP_PROBE_FAILURE_MARKER ) + self.assertEqual( + logs, + [ + "smp-snapshot-1-restore-0.log", + "smp-snapshot-1-restore-1.log", + "smp-snapshot-2-restore-0.log", + "smp-snapshot-4-restore-0.log", + "smp-snapshot-8-restore-0.log", + ], ) + def test_smp_snapshot_rejects_a_restore_without_a_passing_warp_probe(self): + def writer(output: bytes): + def measure(_command: list[str], **kwargs: object) -> None: + cast(Path, kwargs["log_path"]).write_bytes(output) + + return measure + + cases = ( + (b"", "did not finish its warp probe"), + (_warp_probe_output(4, offset_ns=5000), "max_abs_offset_ns=5000"), + ) + for output, message in cases: + with self.subTest(message=message): + with tempfile.TemporaryDirectory() as temporary: + with ( + patch.object(microvm_tests, "capture_snapshot"), + patch.object( + microvm_tests, "measure_once", side_effect=writer(output) + ), + patch.object( + microvm_tests, + "_snapshot_fingerprint", + return_value=("manifest", "state", "memory"), + ), + self.assertRaisesRegex( + RuntimeError, rf"4-vCPU SMP restore 0: .*{message}" + ), + ): + microvm_tests.run_smp_snapshot( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + [4], + memory_mib=128, + timeout=60, + output_dir=Path(temporary), + ) + def test_restore_processors_uses_capacity_eight_and_each_target(self): with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) / "logs" @@ -2810,15 +2942,18 @@ def test_restore_processors_uses_capacity_eight_and_each_target(self): [_restore_target(entry.args[0]) for entry in measure_once.call_args_list], [1, 2, 4, 8, None], ) + # Each restore runs to the end of its post-restore script, so the log + # holds the processor check and the warp probe for every CPU. + for entry in measure_once.call_args_list: + self.assertEqual(entry.kwargs["marker"], benchmark.RESTORE_MARKER) + self.assertEqual( + entry.kwargs["failure_marker"], b"NVX-RESTORE-PROCESSORS-FAIL" + ) + script = capture_snapshot.call_args.kwargs["post_restore_script"] self.assertEqual( - [entry.kwargs["marker"] for entry in measure_once.call_args_list], - [ - b"NVX-RESTORE-PROCESSORS-OK count=1", - b"NVX-RESTORE-PROCESSORS-OK count=2", - b"NVX-RESTORE-PROCESSORS-OK count=4", - b"NVX-RESTORE-PROCESSORS-OK count=8", - b"NVX-RESTORE-PROCESSORS-OK count=1", - ], + script, + microvm_tests._read_script("restore-processors.sh") + + time_abi.warp_probe_script(), ) self.assertEqual( logs, @@ -2869,7 +3004,11 @@ def test_restore_processors_requires_full_capacity_off_mshv(self): def measure(command: list[str], **kwargs: object) -> None: log_path = cast(Path, kwargs["log_path"]) - log_path.write_bytes(_vp_binding_profile([0, 1])) + log_path.write_bytes( + b"NVX-RESTORE-PROCESSORS-OK count=2\r\n" + + _warp_probe_output(2) + + _vp_binding_profile([0, 1]) + ) with ( patch.object(microvm_tests, "capture_snapshot"), @@ -2942,107 +3081,24 @@ def test_restore_vp_bindings_require_one_complete_record_set(self): profile.replace(b"exclusive=0", b"exclusive=2") ) - def test_restore_tsc_sync_forces_linux_warp_check_and_keeps_failure_guard(self): - for backend in ("kvm", "mshv", "whp"): - with self.subTest(backend=backend): - measure = _restore_processors_measure(backend) - with tempfile.TemporaryDirectory() as temporary: - with ( - patch.object(microvm_tests, "capture_snapshot") as capture, - patch.object(microvm_tests, "measure_once", measure), - patch.object( - microvm_tests, - "_snapshot_fingerprint", - return_value=("manifest", "state", "memory"), - ), - ): - microvm_tests.run_restore_processors( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - backend, - [1, 2, 4, 8], - memory_mib=128, - timeout=60, - output_dir=Path(temporary), - check_tsc_sync=True, - ) - command = capture.call_args.args[0] - self.assertEqual( - command[command.index("--cmdline") + 1], - "quiet loglevel=0 maxcpus=1 clearcpuid=tsc_adjust", - ) - script = capture.call_args.kwargs["post_restore_script"] - self.assertTrue( - script.startswith(microvm_tests._read_script("restore-tsc-sync.sh")) - ) - self.assertTrue( - script.endswith(microvm_tests._read_script("restore-processors.sh")) - ) - self.assertEqual(capture.call_args.kwargs["processors"], 1) - self.assertEqual(measure.call_count, 5) - self.assertEqual( - [_restore_target(call.args[0]) for call in measure.call_args_list], - [1, 2, 4, 8, None], - ) - - def test_restore_tsc_sync_rejects_an_ineffective_cpu_feature_mask(self): + def test_restore_processors_script_checks_activation_without_the_kernel_log(self): shell = _posix_shell() if shell is None: self.skipTest("POSIX shell is unavailable") - script = microvm_tests._read_script("restore-tsc-sync.sh") - for cpuinfo, status, expected in ( - ("flags : tsc constant_tsc rdtscp", 0, 0), - ("flags : tsc tsc_adjust constant_tsc", 0, 96), - ("flags : tsc tsc_adjust", 0, 96), - ("", 1, 1), - ): - with self.subTest(cpuinfo=cpuinfo, status=status): - result = subprocess.run( - [shell], - input=( - f"cat() {{ printf '%s\\n' '{cpuinfo}'; return {status}; }}\n" - + script - ), - text=True, - capture_output=True, - timeout=5, - check=False, - ) - self.assertEqual(result.returncode, expected, result.stderr) - self.assertEqual( - "NVX-RESTORE-TSC-SYNC-CHECK-ENABLED" in result.stdout, - expected == 0, - ) - def test_restore_processors_rejects_unstable_tsc_after_cpu_activation(self): - shell = _posix_shell() - if shell is None: - self.skipTest("POSIX shell is unavailable") - - script = microvm_tests._read_script("restore-processors.sh") - for kernel_log, dmesg_status, expected_status in ( - ("clocksource: Switched to clocksource tsc", 0, 0), - ("Measured 10992 cycles TSC warp between CPUs", 0, 95), - ("tsc: Marking TSC unstable due to check_tsc_sync_source failed", 0, 95), - ("TSC found unstable after boot", 0, 95), - ("", 1, 1), - ): - with self.subTest(kernel_log=kernel_log, dmesg_status=dmesg_status): + script = microvm_tests._read_script("restore-processors.sh").replace( + "nvx-exit", "nvx_exit" + ) + self.assertNotIn("dmesg", script) + for online, expected_status in (("0-3", 0), ("0-2", 93)): + with self.subTest(online=online): result = subprocess.run( [shell], input=( "getconf() { printf '4\\n'; }\n" - "cat() {\n" - ' case "$1" in\n' - " */cpu/online) printf '0-3\\n' ;;\n" - " */current_clocksource) printf 'tsc\\n' ;;\n" - " *) return 99 ;;\n" - " esac\n" - "}\n" + f"cat() {{ printf '%s\\n' '{online}'; }}\n" 'taskset() { printf "%s\\n" "$2"; }\n' - f"dmesg() {{ printf '%s\\n' '{kernel_log}'; " - f"return {dmesg_status}; }}\n" + script + 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' + script ), text=True, capture_output=True, @@ -3051,157 +3107,107 @@ def test_restore_processors_rejects_unstable_tsc_after_cpu_activation(self): ) self.assertEqual(result.returncode, expected_status, result.stderr) - self.assertIn("NVX-RESTORE-PROCESSOR-OK count=4 cpu=3", result.stdout) if expected_status == 0: self.assertIn( - "NVX-RESTORE-CLOCKSOURCE-OK source=tsc", result.stdout + "NVX-RESTORE-PROCESSOR-OK count=4 cpu=3", result.stdout ) self.assertIn("NVX-RESTORE-PROCESSORS-OK count=4", result.stdout) + self.assertNotIn("NVX-EXIT", result.stdout) else: - self.assertNotIn("NVX-RESTORE-PROCESSORS-OK", result.stdout) - if expected_status == 95: - self.assertIn(kernel_log, result.stdout) self.assertIn( - "NVX-RESTORE-PROCESSORS-FAIL unstable-tsc", result.stdout + "NVX-RESTORE-PROCESSORS-FAIL expected=0-3 actual=0-2", + result.stdout, ) + # A failed check powers the guest off with its status. + self.assertIn("NVX-EXIT 93", result.stdout) + self.assertNotIn("NVX-RESTORE-PROCESSORS-OK", result.stdout) + + def test_restore_processors_fail_fast_and_record_time_abi_logs(self): + measure = _restore_processors_measure("mshv") + with tempfile.TemporaryDirectory() as temporary: + with ( + patch.object(microvm_tests, "capture_snapshot") as capture, + patch.object(microvm_tests, "measure_once", measure), + patch.object( + microvm_tests, + "_snapshot_fingerprint", + return_value=("manifest", "state", "memory"), + ), + ): + microvm_tests.run_restore_processors( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "mshv", + [2, 8], + memory_mib=128, + timeout=60, + output_dir=Path(temporary), + ) - def test_restore_processors_fail_fast_and_record_restored_tsc_logs(self): - for check_tsc_sync in (False, True): - with self.subTest(check_tsc_sync=check_tsc_sync): - measure = _restore_processors_measure("mshv") + command = capture.call_args.args[0] + # No test-only CPU feature mask and no clock tuning on the command line. + self.assertEqual( + command[command.index("--cmdline") + 1], "quiet loglevel=0 maxcpus=1" + ) + self.assertEqual(measure.call_count, 3) + for entry in measure.call_args_list: + self.assertEqual( + entry.kwargs["failure_marker"], + b"NVX-RESTORE-PROCESSORS-FAIL", + ) + environment = entry.kwargs["environment"] + self.assertEqual( + environment["OPENVMM_LOG"], + "off,openvmm_core::worker::dispatch::time_abi=info", + ) + self.assertEqual(environment[benchmark.SNAPSHOT_PROFILE_ENV], "1") + + def test_restore_processors_require_the_count_and_a_passing_warp_probe(self): + def wrong_count(command: list[str], **kwargs: object) -> None: + _restore_processors_measure("kvm")(command, **kwargs) + log_path = cast(Path, kwargs["log_path"]) + log_path.write_bytes( + log_path.read_bytes().replace( + b"PROCESSORS-OK count=4", b"PROCESSORS-OK count=2" + ) + ) + + cases = ( + (wrong_count, "restore target 4 did not report"), + ( + _restore_processors_measure("kvm", warp_offset_ns=1500), + "restore target 4: cross-vCPU TSC skew check failed in round 1: " + "max_abs_offset_ns=1500 exceeds 1000", + ), + ) + for measure, message in cases: + with self.subTest(message=message): with tempfile.TemporaryDirectory() as temporary: with ( patch.object(microvm_tests, "capture_snapshot"), - patch.object(microvm_tests, "measure_once", measure), + patch.object( + microvm_tests, "measure_once", side_effect=measure + ), patch.object( microvm_tests, "_snapshot_fingerprint", return_value=("manifest", "state", "memory"), ), + self.assertRaisesRegex(RuntimeError, message), ): microvm_tests.run_restore_processors( Path("openvmm"), Path("vmlinux"), Path("initrd"), - "mshv", - [2, 8], + "kvm", + [4], memory_mib=128, timeout=60, output_dir=Path(temporary), - check_tsc_sync=check_tsc_sync, ) - self.assertEqual(measure.call_count, 3) - for entry in measure.call_args_list: - self.assertEqual( - entry.kwargs["failure_marker"], - b"NVX-RESTORE-PROCESSORS-FAIL", - ) - environment = entry.kwargs["environment"] - self.assertEqual( - environment["OPENVMM_LOG"], - "off,vmm_core::partition_unit::vp_set::tsc=debug," - "virt_mshv::x86_64::tsc=info", - ) - self.assertEqual(environment[benchmark.SNAPSHOT_PROFILE_ENV], "1") - - def test_restore_processors_classifies_unstable_tsc_with_fresh_boot_control(self): - with tempfile.TemporaryDirectory() as temporary: - output_dir = Path(temporary) - passing = _restore_processors_measure("mshv") - - def measure(command: list[str], **kwargs: object) -> None: - if _restore_target(command) == 4: - raise benchmark.GuestFailureReported( - "NVX-RESTORE-PROCESSORS-FAIL unstable-tsc", - "Measured 45 cycles TSC warp between CPUs\r\n", - ) - passing(command, **kwargs) - - with ( - patch.object(microvm_tests, "capture_snapshot"), - patch.object( - microvm_tests, "measure_once", side_effect=measure - ) as measure_once, - patch.object( - microvm_tests, - "_snapshot_fingerprint", - return_value=("manifest", "state", "memory"), - ), - patch.object( - microvm_tests, - "run_fresh_boot_tsc_control", - return_value="fresh-boot TSC control: verdict", - ) as control, - patch.object( - microvm_tests, - "_host_invariant_tsc_note", - return_value="host clock: note\n", - ), - ): - with self.assertRaises(RuntimeError) as raised: - microvm_tests.run_restore_processors( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "mshv", - [1, 2, 4, 8], - memory_mib=192, - timeout=30, - output_dir=output_dir, - check_tsc_sync=True, - ) - - self.assertEqual( - str(raised.exception), - "restore target 4: guest reported " - "NVX-RESTORE-PROCESSORS-FAIL unstable-tsc\n" - "fresh-boot TSC control: verdict\n" - "host clock: note\n" - "--- OpenVMM output ---\n" - "Measured 45 cycles TSC warp between CPUs\r\n", - ) - self.assertIsInstance( - raised.exception.__cause__, benchmark.GuestFailureReported - ) - self.assertEqual(measure_once.call_count, 3) - control.assert_called_once_with( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "mshv", - memory_mib=192, - timeout=30, - log_path=output_dir / "restore-processors-tsc-control.log", - ) - - def test_host_invariant_tsc_note_reads_the_cpu_flags(self): - with tempfile.TemporaryDirectory() as temporary: - cpuinfo = Path(temporary) / "cpuinfo" - cases = ( - ( - "processor\t: 0\nflags\t\t: fpu tsc constant_tsc nonstop_tsc\n", - "host CPU exposes an invariant TSC (nonstop_tsc)\n", - ), - ( - "processor\t: 0\nflags\t\t: fpu tsc constant_tsc tsc_known_freq\n", - "host CPU does not expose an invariant TSC (nonstop_tsc); " - "guests on this host intermittently see cross-vCPU TSC warps\n", - ), - ("processor\t: 0\n", ""), - ) - for text, expected in cases: - with self.subTest(text=text): - cpuinfo.write_text(text, encoding="utf-8") - self.assertEqual( - microvm_tests._host_invariant_tsc_note(cpuinfo), expected - ) - self.assertEqual( - microvm_tests._host_invariant_tsc_note(Path(temporary) / "missing"), - "", - ) - - def test_restore_processors_reports_other_guest_failures_without_control(self): + def test_restore_processors_report_guest_failures_with_the_target(self): failure = benchmark.GuestFailureReported( "NVX-RESTORE-PROCESSORS-FAIL expected=0-3 actual=0-2", "tail" ) @@ -3214,7 +3220,6 @@ def test_restore_processors_reports_other_guest_failures_without_control(self): "_snapshot_fingerprint", return_value=("manifest", "state", "memory"), ), - patch.object(microvm_tests, "run_fresh_boot_tsc_control") as control, ): with self.assertRaises(RuntimeError) as raised: microvm_tests.run_restore_processors( @@ -3235,201 +3240,19 @@ def test_restore_processors_reports_other_guest_failures_without_control(self): "--- OpenVMM output ---\ntail", ) self.assertIs(raised.exception.__cause__, failure) - control.assert_not_called() - - def test_fresh_boot_tsc_control_forces_the_warp_check_on_every_processor(self): - with tempfile.TemporaryDirectory() as temporary: - log_path = Path(temporary) / "control.log" - result: benchmark.GuestCommandResult = { - "text": ( - '> echo "NVX-TSC-CONTROL-RESULT unstable activations=0"\r\n' - "NVX-TSC-CONTROL-RESULT stable activations=147\r\n" - ), - "wall_ms": 1.0, - "peak_rss_bytes": 1, - } - with ( - patch.object( - microvm_tests, - "workload_boot_command", - return_value=["openvmm", "boot"], - ) as workload_boot_command, - patch.object( - microvm_tests, "run_guest_script", return_value=result - ) as run_guest_script, - ): - verdict = microvm_tests.run_fresh_boot_tsc_control( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "kvm", - memory_mib=128, - timeout=45, - log_path=log_path, - ) - - self.assertEqual( - verdict, - "fresh-boot TSC control: no TSC instability across 147 CPU " - "activations without snapshot restore", - ) - self.assertEqual( - workload_boot_command.call_args.args, - ( - Path("openvmm"), - "kvm", - Path("vmlinux"), - Path("initrd"), - 128, - "quiet loglevel=0 clearcpuid=tsc_adjust", - ), - ) - self.assertEqual(workload_boot_command.call_args.kwargs, {"processors": 8}) - command, script, marker = run_guest_script.call_args.args - self.assertEqual(command, ["openvmm", "boot"]) - self.assertEqual( - script, - microvm_tests._render_script( - "tsc-sync-control.sh.in", PROCESSORS="8", ROUNDS="20" - ), - ) - self.assertNotIn("@", script) - self.assertEqual(marker, b"NVX-TSC-CONTROL-DONE") - self.assertEqual( - run_guest_script.call_args.kwargs, {"timeout": 45, "log_path": log_path} - ) - def test_fresh_boot_tsc_control_reports_its_own_failure(self): - with patch.object( - microvm_tests, - "run_guest_script", - side_effect=TimeoutError("guest workload did not finish within 45s\ntail"), + def test_removed_tsc_guards_stay_removed(self): + # The warp probe replaced the restore-tsc-sync guard, its + # clearcpuid=tsc_adjust kernel option, and the fresh-boot TSC control. + self.assertNotIn("restore-tsc-sync", microvm_tests.MICROVM_TEST_SCENARIOS) + for name in ("restore-tsc-sync.sh", "tsc-sync-control.sh.in"): + self.assertFalse((microvm_tests.MICROVM_TEST_SCRIPTS_DIR / name).exists()) + for name in ( + "run_fresh_boot_tsc_control", + "_tsc_control_verdict", + "_host_invariant_tsc_note", ): - verdict = microvm_tests.run_fresh_boot_tsc_control( - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "mshv", - memory_mib=128, - timeout=45, - log_path=Path("control.log"), - ) - - self.assertEqual( - verdict, - "fresh-boot TSC control did not complete: " - "guest workload did not finish within 45s", - ) - - def test_tsc_control_verdict_accepts_only_guest_result_lines(self): - for text, expected in ( - ( - "NVX-TSC-CONTROL-RESULT unstable activations=12\r\n", - "fresh-boot TSC control: Linux also found TSC instability without " - "snapshot restore after 12 CPU activations", - ), - ( - '> echo "NVX-TSC-CONTROL-RESULT stable activations=$activations"\n', - "fresh-boot TSC control did not report a result", - ), - ( - "NVX-TSC-CONTROL-RESULT stable activations=many\n", - "fresh-boot TSC control did not report a result", - ), - ("", "fresh-boot TSC control did not report a result"), - ): - with self.subTest(text=text): - self.assertEqual(microvm_tests._tsc_control_verdict(text), expected) - - def test_tsc_sync_control_script_reactivates_aps_until_tsc_instability(self): - shell = _posix_shell() - if shell is None: - self.skipTest("POSIX shell is unavailable") - for flags, online, unstable, expected_status, expected_output in ( - ("tsc rdtscp", 4, False, 0, "NVX-TSC-CONTROL-RESULT stable activations=9"), - ( - "tsc rdtscp", - 4, - True, - 0, - "NVX-TSC-CONTROL-RESULT unstable activations=3", - ), - ("tsc tsc_adjust", 4, False, 61, "NVX-TSC-CONTROL-FAIL code=61"), - ("tsc rdtscp", 2, False, 60, "NVX-TSC-CONTROL-FAIL code=60"), - ): - with self.subTest(flags=flags, online=online, unstable=unstable): - with tempfile.TemporaryDirectory() as temporary: - cpu_root = Path(temporary) - for cpu in range(1, 4): - (cpu_root / f"cpu{cpu}").mkdir() - kernel_log = ( - "Measured 45 cycles TSC warp between CPUs" - if unstable - else "clocksource: Switched to clocksource tsc" - ) - script = ( - microvm_tests._render_script( - "tsc-sync-control.sh.in", PROCESSORS="4", ROUNDS="2" - ) - .replace("/sys/devices/system/cpu", cpu_root.as_posix()) - .replace("nvx-exit", "nvx_exit") - ) - result = subprocess.run( - [shell, "-s"], - input=( - f"getconf() {{ printf '{online}\\n'; }}\n" - f"cat() {{ printf 'flags : {flags}\\n'; }}\n" - f"dmesg() {{ printf '%s\\n' '{kernel_log}'; }}\n" - 'nvx_exit() { exit "$1"; }\n' + script - ), - text=True, - capture_output=True, - timeout=5, - check=False, - ) - reactivated = all( - (cpu_root / f"cpu{cpu}" / "online").is_file() - and (cpu_root / f"cpu{cpu}" / "online").read_text() == "1\n" - for cpu in range(1, 4) - ) - - self.assertEqual( - result.returncode, expected_status, result.stdout + result.stderr - ) - self.assertIn(expected_output, result.stdout) - self.assertEqual( - "NVX-TSC-CONTROL-DONE" in result.stdout, expected_status == 0 - ) - self.assertEqual(reactivated, expected_output.endswith("=9")) - if unstable: - self.assertIn(kernel_log, result.stdout) - - def test_tsc_sync_control_script_fails_when_dmesg_fails(self): - shell = _posix_shell() - if shell is None: - self.skipTest("POSIX shell is unavailable") - script = microvm_tests._render_script( - "tsc-sync-control.sh.in", PROCESSORS="4", ROUNDS="2" - ).replace("nvx-exit", "nvx_exit") - - result = subprocess.run( - [shell, "-s"], - input=( - "getconf() { printf '4\\n'; }\n" - "cat() { printf 'flags : tsc rdtscp\\n'; }\n" - "dmesg() { return 71; }\n" - 'nvx_exit() { exit "$1"; }\n' + script - ), - text=True, - capture_output=True, - timeout=5, - check=False, - ) - - self.assertEqual(result.returncode, 71, result.stdout + result.stderr) - self.assertIn("NVX-TSC-CONTROL-FAIL code=71", result.stdout) - self.assertNotIn("NVX-TSC-CONTROL-RESULT", result.stdout) - self.assertNotIn("NVX-TSC-CONTROL-DONE", result.stdout) + self.assertFalse(hasattr(microvm_tests, name), name) def test_restore_memory_reuses_one_base_snapshot_for_all_targets(self): with tempfile.TemporaryDirectory() as temporary: @@ -3759,7 +3582,7 @@ def require(path: Path, _description: str) -> Path: ): microvm_tests.run(args) - def test_runner_keeps_restore_tsc_logs_separate_from_processor_restore(self): + def test_runner_passes_processor_counts_to_the_restore_scenarios(self): def require(path: Path, _description: str) -> Path: return path @@ -3771,11 +3594,12 @@ def require(path: Path, _description: str) -> Path: "--backend", "whp", "--scenario", - "restore-processors", - "--scenario", - "restore-tsc-sync", + "smp-snapshot", "--scenario", - "restore-tsc-sync", + "restore-processors", + "--processors", + "2", + "4", "--output-dir", str(output_dir), ] @@ -3783,18 +3607,18 @@ def require(path: Path, _description: str) -> Path: with ( patch.object(microvm_tests, "validate_openvmm_test_backend"), patch.object(microvm_tests, "require_file", side_effect=require), - patch.object(microvm_tests, "run_restore_processors") as run, + patch.object(microvm_tests, "run_smp_snapshot") as smp_snapshot, + patch.object(microvm_tests, "run_restore_processors") as processors, ): self.assertEqual(microvm_tests.run(args), 0) - self.assertTrue((output_dir / "restore-tsc-sync").is_dir()) - self.assertEqual(run.call_count, 2) - self.assertEqual(run.call_args_list[0].kwargs["output_dir"], output_dir) - self.assertNotIn("check_tsc_sync", run.call_args_list[0].kwargs) - self.assertEqual( - run.call_args_list[1].kwargs["output_dir"], - output_dir / "restore-tsc-sync", - ) - self.assertTrue(run.call_args_list[1].kwargs["check_tsc_sync"]) + for run in (smp_snapshot, processors): + run.assert_called_once() + self.assertEqual(run.call_args.args[4], [2, 4]) + self.assertEqual(run.call_args.kwargs["output_dir"], output_dir) + with self.assertRaises(SystemExit), patch("sys.stderr"): + nvx.parse_args( + ["test-microvm", "--backend", "kvm", "--scenario", "restore-tsc-sync"] + ) def test_runner_dispatches_console_exit_for_each_requested_cpu_count(self): def require(path: Path, _description: str) -> Path: From 7f9e79cd57920adfe1d6a63a945067c44603311c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 06:05:17 -0700 Subject: [PATCH 110/188] ci: require the time ABI boot marker on every cold boot Time ABI v1 guests run the boot check in init before any other work and print a passing NVX-TIME-ABI boot line before the guest boot marker. The harness already failed on violation events, failed checks, and the 193/194/195 power-off statuses; it now also fails a cold boot whose guest reaches ALPINE-MICROVM-BOOT-OK or NVX-GUEST-BOOT-OK without a valid boot line, in every runner that feeds the monitor: the benchmarks, the guest-script and capture runners, and OpenvmmProcess. The line must report ABI version 1, generation 0, a TSC rate between 500 MHz and 10 GHz, and the backend's LAPIC rate. A report-only boot (NVX-TIME-REPORT) is rejected, as the spec requires. Restores and launches without --kernel carry no boot line and are not checked. REQUIRE_BOOT_MARKER exists so that legacy-path validation can disable the check; nothing in the repository clears it. Lands with the OpenVMM time ABI and the guest's init wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 5 ++++ doc/ci.md | 8 +++++- scripts/nvx_tools/time_abi.py | 15 ++++++----- scripts/test_time_abi.py | 50 ++++++++++++++++++++++++----------- 4 files changed, 55 insertions(+), 23 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index acacf5772..5f0444a1a 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -305,6 +305,11 @@ console-timed capture results, restore results without prequeued guest exit, mis any guest-exit teardown timeout. With at least ten samples, it also rejects snapshot-generation series whose p50 is more than 25% above p25; this prevents a transient host stall from entering performance history without hiding a uniformly slower product result. +Every launch is scanned for the guest's time ABI output as in the +[microVM correctness jobs](ci.md): a cold boot must print a passing `NVX-TIME-ABI` boot line +before `ALPINE-MICROVM-BOOT-OK`, and a violation event or a time ABI power-off fails the run. +The scan parses output the harness already reads, so it adds nothing to the measured intervals; +the guest's boot check is part of `openvmm_cold_start`. Lifecycle capture runs a deterministic affinity-pinned worker on every vCPU before the snapshot request. Explicit correctness scenarios also stage a post-restore probe. The capture probe is outside the snapshot-generation timing interval. Each worker proves that it diff --git a/doc/ci.md b/doc/ci.md index 070a5733b..31fad72c5 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -25,7 +25,13 @@ Ubuntu initramfs. Failure logs from the NVX layer are uploaded per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM console for the guest's [time ABI](design/time-abi.md) output. An `NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line -fails the scenario at once with the guest's code and detail. An OpenVMM exit +fails the scenario at once with the guest's code and detail. A cold boot whose +guest reaches its boot marker without a passing `NVX-TIME-ABI` boot line also +fails at once: init runs the boot check before any other guest work, so a +missing line means the guest image or OpenVMM does not implement the ABI. The +line must report ABI version 1, generation 0, a plausible TSC rate, and the +backend's LAPIC rate; a report-only boot (`NVX-TIME-REPORT`) is never accepted. +Restores carry no boot line. An OpenVMM exit status of 193, 194, or 195 is reported as the guest's time ABI conformance, runtime-violation, or restore-repair power-off, together with the event that preceded it, instead of as a generic exit status. diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index bb57a53cf..01d31c15b 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -77,6 +77,9 @@ def profile_id(self) -> str: ) # Guest boot markers that init prints after the time ABI boot check. GUEST_BOOT_MARKERS = ("ALPINE-MICROVM-BOOT-OK", "NVX-GUEST-BOOT-OK:") +# Init runs the boot check before any other guest work, so a cold boot whose +# guest reaches its shell without the boot marker is not conformant. +REQUIRE_BOOT_MARKER = True _MAX_PENDING_LINE = 64 * 1024 @@ -241,12 +244,15 @@ class TimeAbiMonitor: ``feed`` raises TimeAbiFailure as soon as a completed line carries a violation event or a failed conformance check, so a scenario fails fast - with the guest's own explanation instead of a marker timeout. + with the guest's own explanation instead of a marker timeout. On a cold + boot it also fails when the guest reaches its shell without a valid + boot marker. """ def __init__(self, command: Sequence[str] = ()) -> None: self.backend = _command_option(command, "--hypervisor") self.cold_boot = is_cold_boot_command(command) + self.require_boot_marker = REQUIRE_BOOT_MARKER and self.cold_boot self.boot: dict[str, str] | None = None self.restores: list[dict[str, str]] = [] self.uncertain: list[dict[str, str]] = [] @@ -300,6 +306,8 @@ def _line(self, line: str) -> None: f" {marker}" in line for marker in GUEST_BOOT_MARKERS ): self.guest_booted = True + if self.require_boot_marker: + self.require_boot("the guest boot marker") if not line.startswith(MARKER_PREFIX): return try: @@ -366,11 +374,6 @@ def require_boot( ) return self.boot - def require_boot_if_booted(self) -> None: - """Require the boot marker once a cold-booted guest reached its shell.""" - if self.cold_boot and self.guest_booted: - self.require_boot("the guest boot marker") - def warp_probe_command(*, cpus: str | None = None) -> str: """Return the guest command line that runs the warp probe.""" diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 458ffec3e..a64f568e6 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -151,7 +151,6 @@ def test_records_markers_across_chunk_boundaries(self): self.assertEqual(monitor.require_boot("test", online_cpus=4)["cpus"], "4") self.assertEqual(len(monitor.uncertain), 1) self.assertEqual(monitor.restores[0]["generation"], "1") - monitor.require_boot_if_booted() def test_fails_fast_on_violation_events_and_failed_checks(self): monitor = TimeAbiMonitor(KVM_BOOT) @@ -172,31 +171,36 @@ def test_fails_fast_on_violation_events_and_failed_checks(self): b"NVX-TIME-ABI: v=1 phase=boot status=maybe\n" ) - def test_ignores_report_only_lines_but_reports_them_when_the_marker_is_missing( - self, - ): + def test_rejects_a_report_only_boot_when_the_guest_reaches_its_shell(self): monitor = TimeAbiMonitor(KVM_BOOT) monitor.feed( b"NVX-TIME-REPORT-VIOLATION: v=1 code=G_TSC_WARP source=watcher " b'phase=boot generation=0 boottime_ns=1 detail="x"\n' b"NVX-TIME-REPORT: v=1 phase=boot status=fail cpus=1 failures=3\n" - b"NVX-GUEST-BOOT-OK: alpine\n" ) self.assertTrue(monitor.report_only) with self.assertRaisesRegex(TimeAbiFailure, "report-only"): - monitor.require_boot_if_booted() + monitor.feed(b"NVX-GUEST-BOOT-OK: alpine\n") - def test_requires_the_boot_marker_only_for_cold_boots_that_reached_the_shell( - self, - ): + def test_requires_a_valid_boot_marker_only_for_cold_boots(self): + with self.assertRaisesRegex( + TimeAbiFailure, + "did not print the NVX-TIME-ABI boot marker before the guest boot " + "marker; the guest image or OpenVMM does not implement time ABI v1", + ): + TimeAbiMonitor(KVM_BOOT).feed(b"NVX-GUEST-BOOT-OK: alpine\n") + mismatched = BOOT_LINE.replace("lapic_hz=1000000000", "lapic_hz=200000000") monitor = TimeAbiMonitor(KVM_BOOT) - monitor.require_boot_if_booted() - monitor.feed(b"NVX-GUEST-BOOT-OK: alpine\n") - with self.assertRaisesRegex(TimeAbiFailure, "does not implement time ABI"): - monitor.require_boot_if_booted() - restore = TimeAbiMonitor(MSHV_RESTORE) - restore.feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") - restore.require_boot_if_booted() + monitor.feed(mismatched.encode()) + with self.assertRaisesRegex(TimeAbiFailure, "is not the kvm rate"): + monitor.feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") + # Restores and launches without a kernel never print a boot marker. + TimeAbiMonitor(MSHV_RESTORE).feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") + TimeAbiMonitor(["openvmm"]).feed(b"NVX-GUEST-BOOT-OK: alpine\n") + with patch.object(time_abi, "REQUIRE_BOOT_MARKER", False): + legacy = TimeAbiMonitor(KVM_BOOT) + legacy.feed(b"NVX-GUEST-BOOT-OK: alpine\n") + self.assertTrue(legacy.guest_booted) def test_validates_boot_marker_fields_against_the_backend(self): cases = ( @@ -585,6 +589,20 @@ def test_run_guest_script_classifies_a_conformance_power_off(self): KVM_BOOT, "true\n", b"DONE", timeout=5, boot_marker=b"BOOT" ) + def test_cold_boot_runners_require_the_boot_marker(self): + boot = benchmark.BOOT_MARKER + b": 3.22.1\n" + interaction = FakeInteraction([boot, b"DONE\n"], 0) + with patch.object(benchmark, "InteractiveProcess", return_value=interaction): + with self.assertRaisesRegex(RuntimeError, "NVX-TIME-ABI boot marker"): + benchmark.run_guest_script(KVM_BOOT, "true\n", b"DONE", timeout=5) + self.assertEqual(interaction.writes, []) + conformant = FakeInteraction([BOOT_LINE.encode(), boot, b"DONE\n"], 0) + with patch.object(benchmark, "InteractiveProcess", return_value=conformant): + result = benchmark.run_guest_script(KVM_BOOT, "true\n", b"DONE", timeout=5) + self.assertIn("phase=boot status=ok", result["text"]) + with self.assertRaisesRegex(RuntimeError, "NVX-TIME-ABI boot marker"): + self._measure([boot, benchmark.RESTORE_MARKER + b"\n"], 0, KVM_BOOT) + def test_capture_snapshot_classifies_a_failed_capture_check(self): interaction = FakeInteraction([b"booting\n"], 193) with tempfile.TemporaryDirectory() as temporary: From 7c227fb654433ba5fdc73b45560048f840417843 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 06:19:16 -0700 Subject: [PATCH 111/188] ci: add a long-downtime restore scenario The time ABI makes monotonic time advance by the snapshot downtime and releases the guest's RCU stall suppression only after the grace period that was in flight at capture has ended. The spec's restore matrix therefore restores after a downtime longer than the 21 s RCU stall timeout, at 1 and 8 vCPUs, with and without rcupdate.rcu_expedited=1, and requires no stall. restore-downtime captures those four snapshots, waits one shared 30 s window, and restores each. The restored guest runs the warp probe; the harness waits up to 30 s for the daemon's NVX-TIME-ABI restore line, then checks two seconds later that /sys/kernel/rcu_stall_count is 0 and that the uptime is at least 30 s, which fails if the restore froze monotonic time. The watcher's own RCU stall events still fail the run at once. The scenario runs in every microVM job and costs about a minute per backend. Lands with the OpenVMM time ABI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 9 + .../restore-downtime.sh.in | 18 ++ scripts/nvx_tools/microvm_tests.py | 110 ++++++++++++ scripts/test_microvm_tests.py | 164 ++++++++++++++++++ 4 files changed, 301 insertions(+) create mode 100644 scripts/nvx_tools/microvm_test_scripts/restore-downtime.sh.in diff --git a/doc/ci.md b/doc/ci.md index 31fad72c5..f94101716 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -63,6 +63,15 @@ probe also powers the guest off with status 97. Restore-processor logs record OpenVMM's `time ABI rates declared` event, which reports the identity MSR route, the TSC synchronization method, the native and declared TSC rates, and the rate deviation from the snapshot. +The `restore-downtime` scenario covers the time ABI's long-downtime case. It +captures four snapshots, at 1 and 8 vCPUs, each with and without +`rcupdate.rcu_expedited=1`, then waits 30 s, longer than the guest's 21 s RCU +stall timeout, and restores each one. Every restored guest must run the warp +probe, print its `NVX-TIME-ABI ... phase=restore status=ok` line within 30 s of +the restore marker, report `/sys/kernel/rcu_stall_count` as 0 two seconds +later, and show an uptime of at least 30 s, which proves that monotonic time +advanced by the downtime. The captures share one downtime window, so the +scenario adds about a minute per backend. Every job that uses the `validate-runner` action first qualifies its runner for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm diff --git a/scripts/nvx_tools/microvm_test_scripts/restore-downtime.sh.in b/scripts/nvx_tools/microvm_test_scripts/restore-downtime.sh.in new file mode 100644 index 000000000..eaacfc324 --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/restore-downtime.sh.in @@ -0,0 +1,18 @@ +set -eu +minimum=@MIN_UPTIME_SECONDS@ +sleep @SETTLE_SECONDS@ +stalls="$(cat /sys/kernel/rcu_stall_count)" +uptime="$(cut -d. -f1 /proc/uptime)" +echo "NVX-RESTORE-DOWNTIME stalls=$stalls uptime_s=$uptime" +if [ "$stalls" -ne 0 ]; then + echo "NVX-RESTORE-DOWNTIME-FAIL rcu_stall_count=$stalls" + nvx-exit 98 + exit 98 +fi +if [ "$uptime" -lt "$minimum" ]; then + echo "NVX-RESTORE-DOWNTIME-FAIL uptime_s=$uptime minimum=$minimum" + nvx-exit 99 + exit 99 +fi +echo NVX-RESTORE-DOWNTIME-OK +nvx-exit 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 3ef3f2d9b..dac107350 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -83,6 +83,7 @@ "managed-lifecycle", "managed-exec-config", "network-snapshot", + "restore-downtime", "restore-memory", "restore-processors", "sandbox-blocks", @@ -170,6 +171,18 @@ # console line; targets that log while the guest runs, such as virt_kvm's # hidden-MSR #GPs, could split a marker in the merged console stream. TIME_ABI_RESTORE_LOG_FILTER = "off,openvmm_core::worker::dispatch::time_abi=info" +# The guest finishes a restore after the RCU grace-period release, which gives +# up after rcu_cpu_stall_timeout (21 s), and the deferred C7 check. +TIME_ABI_RESTORE_FINISH_SECONDS = 30.0 +# The spec's long-downtime case: longer than the 21 s RCU stall timeout, at 1 +# and 8 vCPUs, with and without expedited grace periods. +RESTORE_DOWNTIME_SECONDS = 30.0 +RESTORE_DOWNTIME_PROCESSORS = (1, 8) +RESTORE_DOWNTIME_COMPLETION_MARKER = b"NVX-RESTORE-DOWNTIME-OK" +RESTORE_DOWNTIME_PATH = "/tmp/nvx-restore-downtime" +# Gives the RCU stall detector time to report a stall that the release of +# the stall suppression exposed. +RESTORE_DOWNTIME_SETTLE_SECONDS = 2 OUTCOME_TOP_LEVEL_FIELDS = frozenset( { "schema_version", @@ -2575,6 +2588,89 @@ def run_restore_processors( ) +def run_restore_downtime( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Restore snapshots after a downtime longer than the RCU stall timeout. + + Every snapshot is captured first, so one shared downtime window covers + them all. The restored guest must finish its time ABI restore, report no + RCU stall, and show that monotonic time advanced by the downtime. + """ + cases = [ + (processors, expedited) + for processors in RESTORE_DOWNTIME_PROCESSORS + for expedited in (False, True) + ] + check = _render_script( + "restore-downtime.sh.in", + SETTLE_SECONDS=str(RESTORE_DOWNTIME_SETTLE_SECONDS), + MIN_UPTIME_SECONDS=str(int(RESTORE_DOWNTIME_SECONDS)), + ) + environment = {"OPENVMM_LOG": TIME_ABI_RESTORE_LOG_FILTER} + with tempfile.TemporaryDirectory(prefix="nvx-restore-downtime-") as temporary: + snapshots: list[tuple[str, int, Path]] = [] + for processors, expedited in cases: + name = f"{processors}-vcpu" + ("-expedited" if expedited else "") + cmdline = "quiet loglevel=0" + ( + " rcupdate.rcu_expedited=1" if expedited else "" + ) + snapshot_path = Path(temporary) / name + boot_command = workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + cmdline, + processors=processors, + ) + # The restored guest keeps running after the restore marker so the + # harness can wait for the daemon's restore marker. + capture_snapshot( + [*boot_command, "--snapshot-destination", str(snapshot_path)], + snapshot_path, + timeout=timeout, + processors=processors, + teardown_mode="host-terminate", + post_restore_script=warp_probe_script(), + log_path=output_dir / f"restore-downtime-{name}-capture.log", + ) + snapshots.append((name, processors, snapshot_path)) + restore_after = time.monotonic() + RESTORE_DOWNTIME_SECONDS + for name, processors, snapshot_path in snapshots: + time.sleep(max(0.0, restore_after - time.monotonic())) + context = f"{name} restore after a {RESTORE_DOWNTIME_SECONDS:g} s downtime" + with OpenvmmProcess( + snapshot_restore_command( + executable, + backend, + snapshot_path, + processors=processors, + ), + output_dir / f"restore-downtime-{name}.log", + environment=environment, + ) as process: + process.wait_for_line(RESTORE_MARKER, timeout) + process.wait_for_time_abi("restore", TIME_ABI_RESTORE_FINISH_SECONDS) + _stage_script( + process, RESTORE_DOWNTIME_PATH, "NVX_RESTORE_DOWNTIME", check + ) + process.wait_for_line(RESTORE_DOWNTIME_COMPLETION_MARKER, timeout) + result = process.wait(timeout) + if result.returncode != 0: + error = process.time_abi.exit_error(result.returncode) + raise RuntimeError(f"{context}: {error}") + _check_restore_warp(result.output, processors=processors, context=context) + + def run_restore_memory( executable: Path, kernel: Path, @@ -4382,6 +4478,20 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, output_dir=output_dir, ) + if "restore-downtime" in scenarios: + print( + "Running microVM long-downtime restore correctness on " + f"OpenVMM/{args.backend}" + ) + run_restore_downtime( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "restore-memory" in scenarios: print(f"Running microVM restore-memory correctness on OpenVMM/{args.backend}") run_restore_memory( diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 1cd284a62..caadaeea2 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -4,6 +4,7 @@ import argparse import json import queue +import re import shutil import socket import subprocess @@ -3241,6 +3242,169 @@ def test_restore_processors_report_guest_failures_with_the_target(self): ) self.assertIs(raised.exception.__cause__, failure) + def test_restore_downtime_shares_one_window_and_waits_for_the_restore_marker( + self, + ): + clock = [100.0] + sleeps: list[float] = [] + + def sleep(seconds: float) -> None: + sleeps.append(seconds) + clock[0] += seconds + + fake_time = MagicMock() + fake_time.monotonic.side_effect = lambda: clock[0] + fake_time.sleep.side_effect = sleep + results = [ + openvmm_process.OpenvmmProcessResult(0, _warp_probe_output(processors)) + for processors in (1, 1, 8, 8) + ] + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + with ( + patch.object( + microvm_tests, + "workload_boot_command", + return_value=["openvmm", "boot"], + ) as boot_command, + patch.object(microvm_tests, "capture_snapshot") as capture, + patch.object(microvm_tests, "OpenvmmProcess") as process, + patch.object(microvm_tests, "time", fake_time), + ): + active = process.return_value.__enter__.return_value + active.wait.side_effect = results + microvm_tests.run_restore_downtime( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + memory_mib=128, + timeout=60, + output_dir=output_dir, + ) + + self.assertEqual( + [ + (call.args[5], call.kwargs["processors"]) + for call in boot_command.call_args_list + ], + [ + ("quiet loglevel=0", 1), + ("quiet loglevel=0 rcupdate.rcu_expedited=1", 1), + ("quiet loglevel=0", 8), + ("quiet loglevel=0 rcupdate.rcu_expedited=1", 8), + ], + ) + for call in capture.call_args_list: + self.assertEqual(call.kwargs["teardown_mode"], "host-terminate") + self.assertEqual( + call.kwargs["post_restore_script"], time_abi.warp_probe_script() + ) + # Every capture precedes one shared 30 s window. + self.assertEqual(sleeps, [30.0, 0.0, 0.0, 0.0]) + self.assertEqual( + [call.args[1].name for call in process.call_args_list], + [ + "restore-downtime-1-vcpu.log", + "restore-downtime-1-vcpu-expedited.log", + "restore-downtime-8-vcpu.log", + "restore-downtime-8-vcpu-expedited.log", + ], + ) + for call in process.call_args_list: + self.assertEqual( + call.kwargs["environment"], + {"OPENVMM_LOG": "off,openvmm_core::worker::dispatch::time_abi=info"}, + ) + self.assertIn("--restore-snapshot", call.args[0]) + self.assertEqual( + active.wait_for_time_abi.call_args_list[0].args, ("restore", 30.0) + ) + self.assertEqual( + [call.args[0] for call in active.wait_for_line.call_args_list[:2]], + [benchmark.RESTORE_MARKER, b"NVX-RESTORE-DOWNTIME-OK"], + ) + staged = active.send_bytes.call_args_list[0].args[0].decode() + self.assertIn("sleep 2\n", staged) + self.assertIn("minimum=30\n", staged) + + def test_restore_downtime_rejects_a_failed_check_or_warp_probe(self): + fatal = "[E_TSC_SYNC_UNSUPPORTED] failed to launch vm worker" + cases = ( + ( + openvmm_process.OpenvmmProcessResult(99, b""), + None, + "downtime: OpenVMM exited with status 99$", + ), + ( + openvmm_process.OpenvmmProcessResult(1, b""), + fatal, + f"downtime: OpenVMM exited with status 1: {re.escape(fatal)}$", + ), + ( + openvmm_process.OpenvmmProcessResult( + 0, _warp_probe_output(1).replace(b"conclusive=1", b"conclusive=0") + ), + None, + "1-vcpu restore after a 0 s downtime: .*inconclusive", + ), + ) + for result, fatal_error, message in cases: + with self.subTest(message=message): + with tempfile.TemporaryDirectory() as temporary: + with ( + patch.object(microvm_tests, "capture_snapshot"), + patch.object(microvm_tests, "OpenvmmProcess") as process, + patch.object(microvm_tests, "RESTORE_DOWNTIME_SECONDS", 0.0), + ): + active = process.return_value.__enter__.return_value + active.wait.return_value = result + active.time_abi = time_abi.TimeAbiMonitor() + active.time_abi.fatal = fatal_error + with self.assertRaisesRegex(RuntimeError, message): + microvm_tests.run_restore_downtime( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "mshv", + memory_mib=128, + timeout=60, + output_dir=Path(temporary), + ) + + def test_restore_downtime_script_rejects_stalls_and_a_frozen_clock(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + script = microvm_tests._render_script( + "restore-downtime.sh.in", SETTLE_SECONDS="0", MIN_UPTIME_SECONDS="30" + ).replace("nvx-exit", "nvx_exit") + for stalls, uptime, expected in ( + ("0", "41", 0), + ("1", "41", 98), + ("0", "12", 99), + ): + with self.subTest(stalls=stalls, uptime=uptime): + result = subprocess.run( + [shell], + input=( + "sleep() { :; }\n" + f"cat() {{ printf '%s\\n' '{stalls}'; }}\n" + f"cut() {{ printf '%s\\n' '{uptime}'; }}\n" + 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' + script + ), + text=True, + capture_output=True, + timeout=5, + check=False, + ) + self.assertEqual(result.returncode, expected, result.stderr) + self.assertIn(f"NVX-EXIT {expected}", result.stdout) + self.assertEqual( + "NVX-RESTORE-DOWNTIME-OK" in result.stdout.splitlines(), + expected == 0, + ) + def test_removed_tsc_guards_stay_removed(self): # The warp probe replaced the restore-tsc-sync guard, its # clearcpuid=tsc_adjust kernel option, and the fresh-boot TSC control. From 1fbc8c8abe7c40bbcd0fa9cbd65ab62b61d1ae32 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 06:35:30 -0700 Subject: [PATCH 112/188] ci: add the exhaustive time ABI conformance scenario The spec's CI conformance suite runs the guest's exhaustive check, nvx-time exhaustive, on every online CPU: every leaf 0x40000006..=0x400000ff, a scan of the bases 0x40000100..=0x4000ff00 for another hypervisor signature, every C3 MSR, the write rules of HV_X64_MSR_TSC_INVARIANT_CONTROL, writes to the read-only identity MSRs, and reads of IA32_TSC_ADJUST and IA32_TSC_DEADLINE (X1 to X6). The boot check leaves these static backend properties to CI. time-abi-conformance boots the largest requested vCPU count, runs /sbin/nvx-time exhaustive, and records its exit status. The harness requires a passing NVX-TIME-ABI-EXHAUSTIVE line for every check on every CPU, a summary with status=ok, the requested CPU count, and no failures, and exit status 0. A failure names each failing check with the guest's detail, and a guest image without the subcommand fails with that diagnosis. The guest command is one console line, so the console's echo of it ends before the check prints. Lands with the OpenVMM time ABI and the guest's nvx-time exhaustive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 12 ++ scripts/nvx_tools/microvm_tests.py | 135 +++++++++++++++++++++++ scripts/test_microvm_tests.py | 170 +++++++++++++++++++++++++++++ 3 files changed, 317 insertions(+) diff --git a/doc/ci.md b/doc/ci.md index f94101716..cbf0c80cd 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -72,6 +72,18 @@ the restore marker, report `/sys/kernel/rcu_stall_count` as 0 two seconds later, and show an uptime of at least 30 s, which proves that monotonic time advanced by the downtime. The captures share one downtime window, so the scenario adds about a minute per backend. +The `time-abi-conformance` scenario boots the largest requested vCPU count and +runs the guest's exhaustive CI check, `/sbin/nvx-time exhaustive`, which the +boot check leaves to CI. On every online CPU it checks every leaf +`0x40000006..=0x400000ff` and every base `0x40000100..=0x4000ff00` for another +hypervisor signature, every `C3` MSR, the write rules of +`HV_X64_MSR_TSC_INVARIANT_CONTROL`, writes to the read-only identity MSRs, and +reads of `IA32_TSC_ADJUST` and `IA32_TSC_DEADLINE` (checks `X1` to `X6`). The +harness requires a passing `NVX-TIME-ABI-EXHAUSTIVE` line for every check on +every CPU, a summary with `status=ok`, the requested CPU count, and no +failures, and exit status 0; a failure lists each failing check with the +guest's detail. The guest command fits on one console line, so the console's +echo of it ends before the check prints. Every job that uses the `validate-runner` action first qualifies its runner for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index dac107350..271059e28 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -60,10 +60,12 @@ from .managed_exec_tests import run_managed_exec_configuration from .openvmm_process import OpenvmmProcess, TcpConsole from .time_abi import ( + ABI_VERSION, WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_FAILURE_MARKER, check_warp_probe, describe_exit_status, + parse_fields, warp_probe_script, warp_rounds, ) @@ -93,6 +95,7 @@ "snapshot-core", "snapshot-tiers", "structured-outcome", + "time-abi-conformance", "virtio-net", "workload-identity", ) @@ -183,6 +186,13 @@ # Gives the RCU stall detector time to report a stall that the release of # the stall suppression exposed. RESTORE_DOWNTIME_SETTLE_SECONDS = 2 +# The guest's exhaustive CI check reports each check on each online CPU. +TIME_ABI_EXHAUSTIVE_COMMAND = "/sbin/nvx-time exhaustive" +TIME_ABI_EXHAUSTIVE_PREFIX = "NVX-TIME-ABI-EXHAUSTIVE: " +TIME_ABI_EXHAUSTIVE_CHECKS = ("X1", "X2", "X3", "X4", "X5", "X6") +TIME_ABI_EXHAUSTIVE_EXIT_PREFIX = "NVX-EXHAUSTIVE-EXIT status=" +TIME_ABI_EXHAUSTIVE_COMPLETION_MARKER = b"NVX-EXHAUSTIVE-DONE" +TIME_ABI_EXHAUSTIVE_REPORTED_FAILURES = 8 OUTCOME_TOP_LEVEL_FIELDS = frozenset( { "schema_version", @@ -1239,6 +1249,115 @@ def run_smp( ) +def _check_exhaustive_report(text: str, *, processors: int) -> None: + """Check that ``nvx-time exhaustive`` passed every check on every CPU.""" + results: dict[tuple[str, int], dict[str, str]] = {} + summary: dict[str, str] | None = None + exit_status: str | None = None + problems: list[str] = [] + for raw in text.splitlines(): + line = raw.removesuffix("\r") + if line.startswith(TIME_ABI_EXHAUSTIVE_EXIT_PREFIX): + exit_status = line.removeprefix(TIME_ABI_EXHAUSTIVE_EXIT_PREFIX) + continue + if not line.startswith(TIME_ABI_EXHAUSTIVE_PREFIX): + continue + try: + fields = parse_fields(line.removeprefix(TIME_ABI_EXHAUSTIVE_PREFIX)) + if fields.get("v") != ABI_VERSION: + raise ValueError(f"version {fields.get('v')!r} is not {ABI_VERSION}") + if "check" not in fields: + summary = fields + continue + key = (fields["check"], int(fields["cpu"])) + except (KeyError, ValueError) as error: + raise RuntimeError( + f"time ABI exhaustive check: malformed line {line!r}: {error}" + ) from error + if key in results: + problems.append(f"{key[0]} reported cpu={key[1]} twice") + results[key] = fields + if exit_status is None: + raise RuntimeError( + "time ABI exhaustive check: the guest printed no exit status for " + f"{TIME_ABI_EXHAUSTIVE_COMMAND}" + ) + if summary is None and not results: + raise RuntimeError( + f"time ABI exhaustive check: {TIME_ABI_EXHAUSTIVE_COMMAND} exited with " + f"status {exit_status} and reported nothing; the guest image does not " + "provide the exhaustive check" + ) + failed = [ + f"{check} cpu={cpu} failed: {fields.get('detail', '')}" + for (check, cpu), fields in sorted(results.items()) + if fields.get("status") != "pass" + ] + if len(failed) > TIME_ABI_EXHAUSTIVE_REPORTED_FAILURES: + hidden = len(failed) - TIME_ABI_EXHAUSTIVE_REPORTED_FAILURES + failed = failed[:TIME_ABI_EXHAUSTIVE_REPORTED_FAILURES] + [ + f"{hidden} more failed checks" + ] + problems.extend(failed) + for check in TIME_ABI_EXHAUSTIVE_CHECKS: + missing = [cpu for cpu in range(processors) if (check, cpu) not in results] + if missing: + problems.append( + f"{check} reported nothing for cpu {', '.join(map(str, missing))}" + ) + if summary is None: + problems.append("no summary line") + else: + expected = {"status": "ok", "cpus": str(processors), "failures": "0"} + mismatched = [ + f"{name}={summary.get(name)}" + for name, value in expected.items() + if summary.get(name) != value + ] + if mismatched: + problems.append( + f"summary reports {' '.join(mismatched)} for {processors} vCPUs" + ) + if exit_status != "0": + problems.append(f"exit status {exit_status}") + if problems: + raise RuntimeError("time ABI exhaustive check: " + "; ".join(problems)) + + +def run_time_abi_conformance( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + processors: int, + *, + memory_mib: int, + timeout: float, + log_path: Path, +) -> None: + """Run the guest's exhaustive CI conformance check on every CPU.""" + command = workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + processors=processors, + ) + # One input line: the console echoes all of it before the check prints. + result = run_guest_script( + command, + f"{TIME_ABI_EXHAUSTIVE_COMMAND}; " + f'echo "{TIME_ABI_EXHAUSTIVE_EXIT_PREFIX}$?"; ' + f"echo {TIME_ABI_EXHAUSTIVE_COMPLETION_MARKER.decode()}; nvx-exit 0\n", + TIME_ABI_EXHAUSTIVE_COMPLETION_MARKER, + timeout=timeout, + log_path=log_path, + ) + _check_exhaustive_report(result["text"], processors=processors) + + def run_virtio_net( executable: Path, kernel: Path, @@ -4535,6 +4654,22 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, output_dir=output_dir, ) + if "time-abi-conformance" in scenarios: + processors = max(args.processors) + print( + f"Running microVM time ABI conformance ({processors} vCPU) " + f"on OpenVMM/{args.backend}" + ) + run_time_abi_conformance( + executable, + kernel, + initrd, + args.backend, + processors, + memory_mib=args.memory_mib, + timeout=args.timeout, + log_path=output_dir / "time-abi-conformance.log", + ) if "virtio-net" in scenarios: print(f"Running microVM virtio-net correctness on OpenVMM/{args.backend}") run_virtio_net( diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index caadaeea2..c453267c6 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3405,6 +3405,170 @@ def test_restore_downtime_script_rejects_stalls_and_a_frozen_clock(self): expected == 0, ) + @staticmethod + def _exhaustive_report( + processors: int, + *, + exit_status: int = 0, + failing: tuple[str, int] | None = None, + skip: tuple[str, int] | None = None, + ) -> str: + lines = [ + '/sbin/nvx-time exhaustive; echo "NVX-EXHAUSTIVE-EXIT status=$?"; ' + "echo NVX-EXHAUSTIVE-DONE; nvx-exit 0" + ] + failures = 0 + for cpu in range(processors): + for check in microvm_tests.TIME_ABI_EXHAUSTIVE_CHECKS: + if (check, cpu) == skip: + continue + status, detail = "pass", "" + if (check, cpu) == failing: + status, detail = "fail", "MSR 0x40000118 accepted 2" + failures += 1 + lines.append( + f"NVX-TIME-ABI-EXHAUSTIVE: v=1 check={check} cpu={cpu} " + f'status={status} detail="{detail}"' + ) + lines += [ + f"NVX-TIME-ABI-EXHAUSTIVE: v=1 status={'fail' if failures else 'ok'} " + f"cpus={processors} failures={failures}", + f"NVX-EXHAUSTIVE-EXIT status={exit_status}", + "NVX-EXHAUSTIVE-DONE", + ] + return "\r\n".join(lines) + "\r\n" + + def test_exhaustive_report_requires_every_check_on_every_cpu(self): + report = self._exhaustive_report(4) + microvm_tests._check_exhaustive_report(report, processors=4) + summary = "NVX-TIME-ABI-EXHAUSTIVE: v=1 status=ok cpus=4 failures=0\r\n" + first = "NVX-TIME-ABI-EXHAUSTIVE: v=1 check=X1 cpu=0 status=pass" + cases = ( + ( + self._exhaustive_report(4, failing=("X4", 2), exit_status=1), + 4, + "X4 cpu=2 failed: MSR 0x40000118 accepted 2; summary reports " + "status=fail failures=1 for 4 vCPUs; exit status 1$", + ), + ( + self._exhaustive_report(4, skip=("X2", 3)), + 4, + "X2 reported nothing for cpu 3", + ), + ( + self._exhaustive_report(2), + 4, + "X1 reported nothing for cpu 2, 3;.*summary reports cpus=2 for 4 vCPUs", + ), + (self._exhaustive_report(4, exit_status=1), 4, "exit status 1$"), + (report.replace(summary, ""), 4, "no summary line"), + ( + report.replace(first, first + "\r\n" + first), + 4, + "X1 reported cpu=0 twice", + ), + ( + report.replace("status=pass", "status=fail"), + 4, + "X1 cpu=0 failed: ;.*; 16 more failed checks", + ), + ( + report.replace("v=1 check=X3 cpu=1", "v=2 check=X3 cpu=1"), + 4, + "malformed", + ), + (report.replace("check=X3 cpu=1 ", "check=X3 "), 4, "malformed"), + ( + report.replace("NVX-EXHAUSTIVE-EXIT status=0\r\n", ""), + 4, + "printed no exit status", + ), + ( + "nvx-time: unknown command\r\nNVX-EXHAUSTIVE-EXIT status=2\r\n", + 8, + "status 2 and reported nothing; the guest image does not provide", + ), + ) + for text, processors, message in cases: + with self.subTest(message=message): + with self.assertRaisesRegex(RuntimeError, message): + microvm_tests._check_exhaustive_report(text, processors=processors) + + def test_conformance_runs_nvx_time_exhaustive_at_the_requested_vcpus(self): + with patch.object( + microvm_tests, + "run_guest_script", + return_value={"text": self._exhaustive_report(8)}, + ) as run: + microvm_tests.run_time_abi_conformance( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "mshv", + 8, + memory_mib=128, + timeout=60, + log_path=Path("time-abi-conformance.log"), + ) + command, script, marker = run.call_args.args + self.assertEqual(command[command.index("--processors") + 1], "8") + self.assertEqual( + script, + '/sbin/nvx-time exhaustive; echo "NVX-EXHAUSTIVE-EXIT status=$?"; ' + "echo NVX-EXHAUSTIVE-DONE; nvx-exit 0\n", + ) + self.assertEqual(marker, b"NVX-EXHAUSTIVE-DONE") + # The console echoes the input line, which must not complete the run. + self.assertFalse(microvm_tests.contains_output_line(script.encode(), marker)) + + def test_conformance_guest_line_reports_the_check_exit_status(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + with patch.object( + microvm_tests, + "run_guest_script", + return_value={"text": self._exhaustive_report(1)}, + ) as run: + microvm_tests.run_time_abi_conformance( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + 1, + memory_mib=128, + timeout=60, + log_path=Path("time-abi-conformance.log"), + ) + script = ( + run.call_args.args[1] + .replace("/sbin/nvx-time", "nvx_time") + .replace("nvx-exit", "nvx_exit") + ) + for status in (0, 1): + with self.subTest(status=status): + result = subprocess.run( + [shell], + input=( + f'nvx_time() {{ echo "nvx-time $1"; return {status}; }}\n' + 'nvx_exit() { printf "NVX-EXIT %s\\n" "$1"; }\n' + script + ), + text=True, + capture_output=True, + timeout=5, + check=False, + ) + self.assertEqual( + result.stdout.splitlines(), + [ + "nvx-time exhaustive", + f"NVX-EXHAUSTIVE-EXIT status={status}", + "NVX-EXHAUSTIVE-DONE", + "NVX-EXIT 0", + ], + result.stderr, + ) + def test_removed_tsc_guards_stay_removed(self): # The warp probe replaced the restore-tsc-sync guard, its # clearcpuid=tsc_adjust kernel option, and the fresh-boot TSC control. @@ -3761,6 +3925,8 @@ def require(path: Path, _description: str) -> Path: "smp-snapshot", "--scenario", "restore-processors", + "--scenario", + "time-abi-conformance", "--processors", "2", "4", @@ -3773,12 +3939,16 @@ def require(path: Path, _description: str) -> Path: patch.object(microvm_tests, "require_file", side_effect=require), patch.object(microvm_tests, "run_smp_snapshot") as smp_snapshot, patch.object(microvm_tests, "run_restore_processors") as processors, + patch.object(microvm_tests, "run_time_abi_conformance") as conformance, ): self.assertEqual(microvm_tests.run(args), 0) for run in (smp_snapshot, processors): run.assert_called_once() self.assertEqual(run.call_args.args[4], [2, 4]) self.assertEqual(run.call_args.kwargs["output_dir"], output_dir) + # The exhaustive check runs once, on the most CPUs requested. + conformance.assert_called_once() + self.assertEqual(conformance.call_args.args[4], 4) with self.assertRaises(SystemExit), patch("sys.stderr"): nvx.parse_args( ["test-microvm", "--backend", "kvm", "--scenario", "restore-tsc-sync"] From e26fcece8590e767e19c6207e5a95d4b928e478e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 07:04:30 -0700 Subject: [PATCH 113/188] ci: run the restore scenarios on the CI debug kernel The spec runs the same-host restore cases on a CI debug kernel with the soft-lockup and hung-task detectors enabled, which production kernels leave out, so that no watchdog fires across a restore. The guest agent added that kernel (kernel/config-microvm-debug, build/vmlinux-debug) and an opt-in debug-kernel input to build-guest-artifacts. test-microvm --debug-kernel boots build/vmlinux-debug and, unless --scenario is given, runs smp, smp-snapshot, restore-processors, restore-downtime, and snapshot-tiers. A watchdog report makes the guest's time ABI watcher power off with 194, which fails the run. The harness refuses a kernel whose vmlinux-debug.config lacks the detectors, because the guest's C11 check passes vacuously without them. A new GitHub-hosted debug-kernel job caches or builds the debug kernel beside the shared artifacts job and uploads it as guest-debug-kernel. build-guest-artifacts gains a guest-images input; with "false", only the debug kernel steps run. Both kernels share their cache inputs, so building them in one job would add a second kernel build to the critical path whenever the kernel inputs change; the separate job only delays the debug jobs, uses no self-hosted runner, and asks the WHP pool for the same runners at the same time as before. run-nvx-microvm-tests.yml gains a debug-kernel input that downloads the debug kernel, runs test-microvm --debug-kernel instead of the full suite, and skips the Ubuntu tests. To bound runner cost, pull requests run the debug kernel on KVM only (nvx-microvm-debug-kvm) and dev pushes add MSHV and WHP. The debug jobs and the debug kernel build gate the required status check, the development release, and performance persistence; the release and persistence jobs require the build to succeed, because they accept debug jobs that were skipped. Lands with the OpenVMM time ABI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../actions/build-guest-artifacts/action.yml | 69 +++++-- .github/workflows/ci.yml | 172 ++++++++++++++++++ .github/workflows/run-nvx-microvm-tests.yml | 41 +++-- doc/ci.md | 26 ++- scripts/nvx_tools/ci.py | 25 +++ scripts/nvx_tools/microvm_tests.py | 59 +++++- scripts/test_microvm_tests.py | 63 +++++++ scripts/test_nvx_tools.py | 135 +++++++++++++- 8 files changed, 557 insertions(+), 33 deletions(-) diff --git a/.github/actions/build-guest-artifacts/action.yml b/.github/actions/build-guest-artifacts/action.yml index 1897c93f7..e28d4aecc 100644 --- a/.github/actions/build-guest-artifacts/action.yml +++ b/.github/actions/build-guest-artifacts/action.yml @@ -12,6 +12,12 @@ inputs: description: Also cache or build the CI debug kernel (build/vmlinux-debug) required: false default: "false" + guest-images: + description: >- + Cache or build the kernel and guest images; "false" with debug-kernel + builds only the CI debug kernel, for a job that runs beside this one + required: false + default: "true" outputs: kernel-cache-key: @@ -83,6 +89,7 @@ runs: - name: Restore Linux kernel (zstd) id: cache-linux-kernel-zstd + if: inputs.guest-images == 'true' uses: actions/cache/restore@v5 with: path: | @@ -94,6 +101,7 @@ runs: - name: Restore Alpine disk image (zstd) id: cache-alpine-image-zstd + if: inputs.guest-images == 'true' uses: actions/cache/restore@v5 with: path: | @@ -105,6 +113,7 @@ runs: - name: Restore Ubuntu initramfs (zstd) id: cache-ubuntu-initramfs-zstd + if: inputs.guest-images == 'true' uses: actions/cache/restore@v5 with: path: | @@ -115,6 +124,7 @@ runs: - name: Restore Ubuntu distro layer (zstd) id: cache-ubuntu-distro-zstd + if: inputs.guest-images == 'true' uses: actions/cache/restore@v5 with: path: | @@ -125,6 +135,7 @@ runs: - name: Restore Azure Linux initramfs (zstd) id: cache-azurelinux-initramfs-zstd + if: inputs.guest-images == 'true' uses: actions/cache/restore@v5 with: path: | @@ -134,7 +145,10 @@ runs: enableCrossOsArchive: true - name: Build Linux kernel (Docker) - if: inputs.restore-only != 'true' && steps.cache-linux-kernel-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-linux-kernel-zstd.outputs.cache-hit != 'true' shell: bash run: >- docker build @@ -144,7 +158,10 @@ runs: . - name: Save Linux kernel (zstd) - if: inputs.restore-only != 'true' && steps.cache-linux-kernel-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-linux-kernel-zstd.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: path: | @@ -194,7 +211,10 @@ runs: enableCrossOsArchive: true - name: Build Alpine disk image (Docker) - if: inputs.restore-only != 'true' && steps.cache-alpine-image-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-alpine-image-zstd.outputs.cache-hit != 'true' shell: bash run: >- docker build @@ -204,7 +224,10 @@ runs: . - name: Save Alpine disk image (zstd) - if: inputs.restore-only != 'true' && steps.cache-alpine-image-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-alpine-image-zstd.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: path: | @@ -215,7 +238,10 @@ runs: enableCrossOsArchive: true - name: Build Ubuntu initramfs (Docker) - if: inputs.restore-only != 'true' && steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' shell: bash run: >- docker build @@ -228,8 +254,11 @@ runs: if: >- inputs.restore-only != 'true' && ( - steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' || - steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' + inputs.guest-images == 'true' && + ( + steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' || + steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' + ) ) shell: bash run: >- @@ -239,7 +268,10 @@ runs: . - name: Save Ubuntu initramfs (zstd) - if: inputs.restore-only != 'true' && steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-ubuntu-initramfs-zstd.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: path: | @@ -249,7 +281,10 @@ runs: enableCrossOsArchive: true - name: Build Azure Linux initramfs (Docker) - if: inputs.restore-only != 'true' && steps.cache-azurelinux-initramfs-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-azurelinux-initramfs-zstd.outputs.cache-hit != 'true' shell: bash env: AZURELINUX_IMAGE: ${{ steps.guest-versions.outputs.azurelinux-image }} @@ -264,7 +299,10 @@ runs: . - name: Save Azure Linux initramfs (zstd) - if: inputs.restore-only != 'true' && steps.cache-azurelinux-initramfs-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-azurelinux-initramfs-zstd.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: path: | @@ -274,7 +312,10 @@ runs: enableCrossOsArchive: true - name: Build Ubuntu distro layer (Docker) - if: inputs.restore-only != 'true' && steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' shell: bash run: >- docker build @@ -284,7 +325,10 @@ runs: . - name: Save Ubuntu distro layer (zstd) - if: inputs.restore-only != 'true' && steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' + if: >- + inputs.restore-only != 'true' && + inputs.guest-images == 'true' && + steps.cache-ubuntu-distro-zstd.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: path: | @@ -294,6 +338,7 @@ runs: enableCrossOsArchive: true - name: Prepare Ubuntu sandbox scratch template + if: inputs.guest-images == 'true' shell: bash run: | set -euo pipefail diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1485226d1..65f870816 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -187,6 +187,49 @@ jobs: if-no-files-found: error retention-days: 1 + # Beside the shared artifacts, so that a kernel rebuild doesn't delay the + # jobs that don't use the debug kernel. + debug-kernel: + name: Build CI debug kernel + needs: [openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + ( + github.event_name != 'pull_request' || + github.event.pull_request.draft == false + ) && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-workloads == 'true' + ) + }} + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + persist-credentials: false + + - name: Build the CI debug kernel + uses: ./.github/actions/build-guest-artifacts + with: + debug-kernel: "true" + guest-images: "false" + + - name: Upload guest debug kernel + uses: actions/upload-artifact@v7 + with: + name: guest-debug-kernel + path: | + build/vmlinux-debug + build/vmlinux-debug.config + build/vmlinux-debug.provenance.json + if-no-files-found: error + retention-days: 1 + build-openvmm-linux-gnu: name: OpenVMM binary / Linux GNU needs: [openvmm-changes] @@ -816,6 +859,93 @@ jobs: secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} + # Pull requests run the debug kernel on KVM only; dev pushes run every + # backend. + nvx-microvm-debug-kvm: + name: NVX microVM tests / Linux / KVM / Debug kernel + needs: [artifacts, debug-kernel, build-openvmm-linux-gnu, openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + ( + github.event_name != 'pull_request' || + github.event.pull_request.draft == false + ) && + ( + github.event_name == 'push' || + github.event.pull_request.head.repo.full_name == github.repository + ) && + needs.artifacts.result == 'success' && + needs.debug-kernel.result == 'success' && + needs.build-openvmm-linux-gnu.result == 'success' && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-tests == 'true' + ) + }} + uses: ./.github/workflows/run-nvx-microvm-tests.yml + with: + display-name: Linux / KVM / Debug kernel + runner: '["self-hosted", "linux", "kvm", "virtual-machine"]' + backend: kvm + openvmm-artifact: openvmm-linux-gnu + debug-kernel: true + secrets: + OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} + + nvx-microvm-debug-mshv: + name: NVX microVM tests / Linux / MSHV / Debug kernel + needs: [artifacts, debug-kernel, build-openvmm-linux-gnu, openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + github.event_name == 'push' && + needs.artifacts.result == 'success' && + needs.debug-kernel.result == 'success' && + needs.build-openvmm-linux-gnu.result == 'success' && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-tests == 'true' + ) + }} + uses: ./.github/workflows/run-nvx-microvm-tests.yml + with: + display-name: Linux / MSHV / Debug kernel + runner: '["self-hosted", "linux", "mshv", "virtual-machine"]' + backend: mshv + openvmm-artifact: openvmm-linux-gnu + debug-kernel: true + secrets: + OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} + + nvx-microvm-debug-whp: + name: NVX microVM tests / Windows / WHP / Debug kernel + needs: [artifacts, debug-kernel, build-openvmm-windows-msvc, openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + github.event_name == 'push' && + needs.artifacts.result == 'success' && + needs.debug-kernel.result == 'success' && + needs.build-openvmm-windows-msvc.result == 'success' && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-tests == 'true' + ) + }} + uses: ./.github/workflows/run-nvx-microvm-tests.yml + with: + display-name: Windows / WHP / Debug kernel + runner: '["self-hosted", "windows", "whp", "virtual-machine"]' + backend: whp + openvmm-artifact: openvmm-windows-msvc + debug-kernel: true + secrets: + OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} + platform-kvm: name: Platform / Linux / KVM / Virtual machine needs: [artifacts, build-openvmm-linux-gnu, openvmm-changes] @@ -933,6 +1063,7 @@ jobs: release: name: Publish development release needs: + - debug-kernel - quality - aci-edge-sandboxes - openvmm-vmm-tests @@ -940,6 +1071,9 @@ jobs: - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp + - nvx-microvm-debug-kvm + - nvx-microvm-debug-mshv + - nvx-microvm-debug-whp - platform-kvm - platform-mshv - platform-whp @@ -949,6 +1083,7 @@ jobs: github.event_name == 'push' && github.ref == 'refs/heads/dev' && !cancelled() && + needs.debug-kernel.result == 'success' && needs.quality.result == 'success' && needs.aci-edge-sandboxes.result == 'success' && ( @@ -971,6 +1106,18 @@ jobs: needs.nvx-microvm-tests-whp.result == 'success' || needs.nvx-microvm-tests-whp.result == 'skipped' ) && + ( + needs.nvx-microvm-debug-kvm.result == 'success' || + needs.nvx-microvm-debug-kvm.result == 'skipped' + ) && + ( + needs.nvx-microvm-debug-mshv.result == 'success' || + needs.nvx-microvm-debug-mshv.result == 'skipped' + ) && + ( + needs.nvx-microvm-debug-whp.result == 'success' || + needs.nvx-microvm-debug-whp.result == 'skipped' + ) && needs.platform-kvm.result == 'success' && needs.platform-mshv.result == 'success' && needs.platform-whp.result == 'success' @@ -1025,6 +1172,7 @@ jobs: - aci-edge-sandboxes - openvmm-changes - artifacts + - debug-kernel - build-openvmm-linux-gnu - build-openvmm-linux-musl - build-openvmm-windows-msvc @@ -1033,6 +1181,9 @@ jobs: - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp + - nvx-microvm-debug-kvm + - nvx-microvm-debug-mshv + - nvx-microvm-debug-whp - platform-kvm - platform-mshv - platform-whp @@ -1061,6 +1212,7 @@ jobs: ACI_EDGE_SANDBOXES_RESULT: ${{ needs.aci-edge-sandboxes.result }} CHANGES_RESULT: ${{ needs.openvmm-changes.result }} ARTIFACTS_RESULT: ${{ needs.artifacts.result }} + DEBUG_KERNEL_RESULT: ${{ needs.debug-kernel.result }} BUILD_LINUX_GNU_RESULT: ${{ needs.build-openvmm-linux-gnu.result }} BUILD_LINUX_MUSL_RESULT: ${{ needs.build-openvmm-linux-musl.result }} BUILD_WINDOWS_MSVC_RESULT: ${{ needs.build-openvmm-windows-msvc.result }} @@ -1069,6 +1221,9 @@ jobs: MICROVM_KVM_RESULT: ${{ needs.nvx-microvm-tests-kvm.result }} MICROVM_MSHV_RESULT: ${{ needs.nvx-microvm-tests-mshv.result }} MICROVM_WHP_RESULT: ${{ needs.nvx-microvm-tests-whp.result }} + MICROVM_DEBUG_KVM_RESULT: ${{ needs.nvx-microvm-debug-kvm.result }} + MICROVM_DEBUG_MSHV_RESULT: ${{ needs.nvx-microvm-debug-mshv.result }} + MICROVM_DEBUG_WHP_RESULT: ${{ needs.nvx-microvm-debug-whp.result }} PLATFORM_KVM_RESULT: ${{ needs.platform-kvm.result }} PLATFORM_MSHV_RESULT: ${{ needs.platform-mshv.result }} PLATFORM_WHP_RESULT: ${{ needs.platform-whp.result }} @@ -1083,11 +1238,15 @@ jobs: performance-persist: name: Persist performance baseline needs: + - debug-kernel - openvmm-vmm-tests - openvmm-unit-tests - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp + - nvx-microvm-debug-kvm + - nvx-microvm-debug-mshv + - nvx-microvm-debug-whp - platform-kvm - platform-mshv - platform-whp @@ -1097,6 +1256,7 @@ jobs: github.event_name == 'push' && github.ref == 'refs/heads/dev' && !cancelled() && + needs.debug-kernel.result == 'success' && ( needs.openvmm-vmm-tests.result == 'success' || needs.openvmm-vmm-tests.result == 'skipped' @@ -1117,6 +1277,18 @@ jobs: needs.nvx-microvm-tests-whp.result == 'success' || needs.nvx-microvm-tests-whp.result == 'skipped' ) && + ( + needs.nvx-microvm-debug-kvm.result == 'success' || + needs.nvx-microvm-debug-kvm.result == 'skipped' + ) && + ( + needs.nvx-microvm-debug-mshv.result == 'success' || + needs.nvx-microvm-debug-mshv.result == 'skipped' + ) && + ( + needs.nvx-microvm-debug-whp.result == 'success' || + needs.nvx-microvm-debug-whp.result == 'skipped' + ) && needs.platform-kvm.result == 'success' && needs.platform-mshv.result == 'success' && needs.platform-whp.result == 'success' diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 070286630..b1fea0ae1 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -19,6 +19,13 @@ on: description: Base name of the compatible OpenVMM artifacts required: true type: string + debug-kernel: + description: >- + Run the same-host restore scenarios on the CI debug kernel instead + of the full suite and the Ubuntu tests + required: false + type: boolean + default: false secrets: OPENVMM_DEPLOY_KEY: description: SSH deploy key used to checkout OpenVMM @@ -60,6 +67,13 @@ jobs: name: guest-artifacts path: build + - name: Download guest debug kernel + if: inputs.debug-kernel + uses: actions/download-artifact@v8 + with: + name: guest-debug-kernel + path: build + - name: Download OpenVMM executable uses: actions/download-artifact@v8 with: @@ -83,7 +97,8 @@ jobs: run: >- python3 scripts/nvx.py test-microvm --backend "${{ inputs.backend }}" - --output-dir "build/test-results/microvm-${{ inputs.backend }}" + ${{ inputs.debug-kernel && '--debug-kernel' || '' }} + --output-dir "build/test-results/microvm-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}" - name: Run NVX microVM tests on Windows if: runner.os == 'Windows' @@ -91,7 +106,8 @@ jobs: run: >- python scripts\nvx.py test-microvm --backend "${{ inputs.backend }}" - --output-dir "build/test-results/microvm-${{ inputs.backend }}" + ${{ inputs.debug-kernel && '--debug-kernel' || '' }} + --output-dir "build/test-results/microvm-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }}" - name: Run public managed execution configuration on Linux if: runner.os != 'Windows' @@ -112,7 +128,7 @@ jobs: --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Linux - if: runner.os != 'Windows' + if: runner.os != 'Windows' && !inputs.debug-kernel shell: bash run: >- python3 scripts/nvx.py test-microvm @@ -128,7 +144,7 @@ jobs: --output-dir "build/test-results/ubuntu-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Windows - if: runner.os == 'Windows' + if: runner.os == 'Windows' && !inputs.debug-kernel shell: powershell run: >- python scripts\nvx.py test-microvm @@ -144,7 +160,7 @@ jobs: --output-dir "build/test-results/ubuntu-${{ inputs.backend }}" - name: Run Azure Linux guest smoke tests on Linux - if: runner.os != 'Windows' + if: runner.os != 'Windows' && !inputs.debug-kernel shell: bash run: >- python3 scripts/nvx.py test-microvm @@ -161,7 +177,7 @@ jobs: --output-dir "build/test-results/azurelinux-${{ inputs.backend }}" - name: Run Azure Linux guest smoke tests on Windows - if: runner.os == 'Windows' + if: runner.os == 'Windows' && !inputs.debug-kernel shell: powershell run: >- python scripts\nvx.py test-microvm @@ -178,7 +194,7 @@ jobs: --output-dir "build/test-results/azurelinux-${{ inputs.backend }}" - name: Run extended Ubuntu guest tests on Linux/KVM - if: inputs.backend == 'kvm' + if: inputs.backend == 'kvm' && !inputs.debug-kernel shell: bash run: >- python3 scripts/nvx.py test-microvm @@ -192,7 +208,7 @@ jobs: --output-dir "build/test-results/ubuntu-kvm-extended" - name: Run Ubuntu sandbox layer smoke test on Linux - if: runner.os != 'Windows' + if: runner.os != 'Windows' && !inputs.debug-kernel shell: bash run: | set -euo pipefail @@ -345,7 +361,7 @@ jobs: rm -rf "${share}" "${scratch}" "${outside}" - name: Run Ubuntu sandbox layer smoke test on Windows - if: runner.os == 'Windows' + if: runner.os == 'Windows' && !inputs.debug-kernel shell: powershell run: | $manifest = Get-Content -Raw build\ubuntu-distro.erofs.manifest.json | @@ -534,7 +550,7 @@ jobs: Remove-Item -Recurse -Force $share, $scratch, $outside - name: Run aci_edge_sandboxes lifecycle test on Linux - if: runner.os != 'Windows' + if: runner.os != 'Windows' && !inputs.debug-kernel shell: bash run: >- python3 scripts/nvx.py test-aci-edge-sandboxes @@ -542,7 +558,7 @@ jobs: --output-dir "build/test-results/aci-edge-sandboxes-${{ inputs.backend }}" - name: Run aci_edge_sandboxes lifecycle test on Windows - if: runner.os == 'Windows' + if: runner.os == 'Windows' && !inputs.debug-kernel shell: powershell run: >- python scripts\nvx.py test-aci-edge-sandboxes @@ -553,10 +569,11 @@ jobs: if: failure() uses: actions/upload-artifact@v7 with: - name: nvx-microvm-tests-${{ inputs.backend }} + name: nvx-microvm-tests-${{ inputs.backend }}${{ inputs.debug-kernel && '-debug' || '' }} path: | build/test-results/public-exec-${{ inputs.backend }} build/test-results/microvm-${{ inputs.backend }} + build/test-results/microvm-${{ inputs.backend }}-debug build/test-results/ubuntu-${{ inputs.backend }} build/test-results/ubuntu-kvm-extended build/test-results/azurelinux-${{ inputs.backend }} diff --git a/doc/ci.md b/doc/ci.md index cbf0c80cd..90643b317 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -21,7 +21,10 @@ The `nvx-microvm-tests-{kvm,mshv,whp}` jobs consume the NVX Linux kernel and the NVX Linux kernel plus the selected Alpine or Ubuntu initramfs and exercises Linux, SMP, virtio, sandbox, and snapshot behavior through the public OpenVMM CLI. Alpine-control-only scenarios remain explicit and are rejected for the -Ubuntu initramfs. Failure logs from the NVX layer are uploaded per backend. +Ubuntu initramfs. Each job also boots the Azure Linux initramfs through its +one-vCPU smoke set, under the same time ABI checks; the debug-kernel jobs skip +it, as they skip the Ubuntu tests. Failure logs from the NVX layer are uploaded +per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM console for the guest's [time ABI](design/time-abi.md) output. An `NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line @@ -85,6 +88,24 @@ failures, and exit status 0; a failure lists each failing check with the guest's detail. The guest command fits on one console line, so the console's echo of it ends before the check prints. +The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run +`test-microvm --debug-kernel` on the CI debug kernel (`build/vmlinux-debug`, +built from `kernel/config-microvm-debug`), whose soft-lockup and hung-task +detectors production kernels leave out. It selects the same-host restore +scenarios `smp`, `smp-snapshot`, `restore-processors`, `restore-downtime`, and +`snapshot-tiers`. Any RCU stall, soft lockup, or hung task makes the guest's +time ABI watcher power off with status 194, which fails the run. The harness +refuses a kernel whose `vmlinux-debug.config` lacks the detectors, because the +guest's `C11` check passes vacuously without them. To bound the cost, pull +requests run the debug kernel on KVM only and `dev` pushes run it on every +backend; each job takes about five minutes on its own runner, in parallel with +the other microVM jobs. The jobs gate the required status check, the +development release, and performance persistence. The GitHub-hosted +`debug-kernel` job builds the debug kernel beside the shared `artifacts` job +(`build-guest-artifacts` with `guest-images: "false"`) and caches it under its +own key, so a kernel rebuild delays only the debug jobs, and a failed debug +kernel build fails the required status check and blocks the release. + Every job that uses the `validate-runner` action first qualifies its runner for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm --ci-schedule` (see [Host qualification](#host-qualification)): the backend, @@ -258,7 +279,8 @@ Linux/Windows result to be successful; failed, cancelled, or skipped crate checks cannot publish a release. Each `nvx-microvm-tests-{kvm,mshv,whp}` job then runs -`nvx.py test-aci-edge-sandboxes` on its self-hosted runner. This command drives a +`nvx.py test-aci-edge-sandboxes` on its self-hosted runner; the debug-kernel +jobs skip it, because it boots the production kernel. This command drives a complete provision, start, exec, stop, start, and deprovision cycle of the Alpine guest with the crate's OpenVMM backend. It also checks cancellation, that guest state lasts only until a stop, that a start terminates the VM of an diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index d820e2bda..683773ca1 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -32,6 +32,7 @@ "aci-edge-sandboxes": "ACI_EDGE_SANDBOXES_RESULT", "openvmm-changes": "CHANGES_RESULT", "artifacts": "ARTIFACTS_RESULT", + "debug-kernel": "DEBUG_KERNEL_RESULT", "build-openvmm-linux-gnu": "BUILD_LINUX_GNU_RESULT", "build-openvmm-linux-musl": "BUILD_LINUX_MUSL_RESULT", "build-openvmm-windows-msvc": "BUILD_WINDOWS_MSVC_RESULT", @@ -40,6 +41,9 @@ "nvx-microvm-tests-kvm": "MICROVM_KVM_RESULT", "nvx-microvm-tests-mshv": "MICROVM_MSHV_RESULT", "nvx-microvm-tests-whp": "MICROVM_WHP_RESULT", + "nvx-microvm-debug-kvm": "MICROVM_DEBUG_KVM_RESULT", + "nvx-microvm-debug-mshv": "MICROVM_DEBUG_MSHV_RESULT", + "nvx-microvm-debug-whp": "MICROVM_DEBUG_WHP_RESULT", "platform-kvm": "PLATFORM_KVM_RESULT", "platform-mshv": "PLATFORM_MSHV_RESULT", "platform-whp": "PLATFORM_WHP_RESULT", @@ -58,7 +62,16 @@ "nvx-microvm-tests-mshv", "nvx-microvm-tests-whp", ) +# The CI debug kernel runs on KVM for every pull request and on every backend +# for dev pushes. +REQUIRED_CI_MICROVM_DEBUG_JOBS = ("nvx-microvm-debug-kvm",) +REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS = ( + "nvx-microvm-debug-mshv", + "nvx-microvm-debug-whp", +) REQUIRED_CI_ARTIFACT_JOB = "artifacts" +# GitHub-hosted, beside the shared artifacts; only the debug jobs wait for it. +REQUIRED_CI_DEBUG_KERNEL_JOB = "debug-kernel" REQUIRED_CI_PLATFORM_JOBS = ( "platform-kvm", "platform-mshv", @@ -94,6 +107,7 @@ def required_ci_expected_results( "aci-edge-sandboxes": "success", "openvmm-changes": "success", REQUIRED_CI_ARTIFACT_JOB: "success" if run_workloads else "skipped", + REQUIRED_CI_DEBUG_KERNEL_JOB: "success" if run_workloads else "skipped", "performance-gate": ( "success" if event_name == "pull_request" @@ -128,7 +142,18 @@ def required_ci_expected_results( for job in ( *REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *REQUIRED_CI_MICROVM_TEST_JOBS, + *REQUIRED_CI_MICROVM_DEBUG_JOBS, + ) + } + ) + expected.update( + { + job: ( + "success" + if event_name == "push" and run_tests and run_workloads + else "skipped" ) + for job in REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS } ) expected.update( diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 271059e28..580b1a6b7 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -108,6 +108,15 @@ "snapshot-tiers", ) ) +# The spec runs the same-host restore cases on the CI debug kernel, whose +# soft-lockup and hung-task detectors the guest's time ABI watcher reports. +DEBUG_KERNEL_SCENARIOS = ( + "smp", + "smp-snapshot", + "restore-processors", + "restore-downtime", + "snapshot-tiers", +) MICROVM_PROCESSOR_COUNTS = (1, 2, 4, 8) MICROVM_TEST_SCRIPTS_DIR = Path(__file__).with_name("microvm_test_scripts") LIFECYCLE_COMPLETION_MARKER = b"NVX-LIFECYCLE-OK" @@ -233,6 +242,15 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: choices=MICROVM_TEST_SCENARIOS, help="scenario to run; repeat to select multiple (default: all)", ) + parser.add_argument( + "--debug-kernel", + action="store_true", + help=( + "boot the CI debug kernel (build/vmlinux-debug), which enables the " + "soft-lockup and hung-task detectors; selects the same-host restore " + "scenarios unless --scenario is given" + ), + ) parser.add_argument( "--processors", type=int, @@ -4309,15 +4327,47 @@ def run_snapshot_tiers( ) +def require_debug_kernel(config: Path) -> None: + """Fail unless a kernel config enables the debug kernel's detectors. + + The guest's C11 check passes vacuously on a kernel without them, so a + production kernel would otherwise pass the debug-kernel job. + """ + lines = set( + require_file(config, "microVM debug kernel config") + .read_text(encoding="utf-8") + .splitlines() + ) + missing = [ + option + for option in KernelBuildConstants.DEBUG_WATCHDOG_CONFIG + if option not in lines + ] + if missing: + raise ScriptError( + f"{config} is not the CI debug kernel config; it lacks " + + ", ".join(missing) + ) + + def run(args: argparse.Namespace) -> int: validate_openvmm_test_backend(args.backend) descriptor = guest_descriptor(args.guest) if args.memory_mib is None: args.memory_mib = descriptor.default_memory_mib executable = require_file(openvmm_binary_path(), "OpenVMM release binary") - kernel = require_file( - artifact_path(KernelBuildConstants.BINARY_NAME), "microVM Linux direct kernel" - ) + debug_kernel = getattr(args, "debug_kernel", False) + if debug_kernel: + require_debug_kernel(artifact_path(KernelBuildConstants.DEBUG_CONFIG_NAME)) + kernel = require_file( + artifact_path(KernelBuildConstants.DEBUG_BINARY_NAME), + "microVM Linux debug kernel", + ) + else: + kernel = require_file( + artifact_path(KernelBuildConstants.BINARY_NAME), + "microVM Linux direct kernel", + ) initrd = require_file( artifact_path(descriptor.initramfs_name), f"microVM {descriptor.distribution} initramfs", @@ -4330,9 +4380,10 @@ def run(args: argparse.Namespace) -> int: if not descriptor.sandbox_control: unsupported_scenarios.update(SANDBOX_CONTROL_SCENARIOS) if args.scenario is None: + defaults = DEBUG_KERNEL_SCENARIOS if debug_kernel else MICROVM_TEST_SCENARIOS scenarios = tuple( scenario - for scenario in MICROVM_TEST_SCENARIOS + for scenario in defaults if scenario not in unsupported_scenarios and scenario != "managed-exec-config" ) diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index c453267c6..a7a6ed3cd 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -12,6 +12,7 @@ import tempfile import unittest from collections.abc import Callable +from contextlib import ExitStack from pathlib import Path from typing import cast from unittest.mock import MagicMock, patch @@ -3910,6 +3911,68 @@ def require(path: Path, _description: str) -> Path: ): microvm_tests.run(args) + def test_debug_kernel_runs_the_same_host_restore_scenarios_on_vmlinux_debug( + self, + ): + def require(path: Path, _description: str) -> Path: + return path + + dispatch = { + "smp": "run_smp", + "smp-snapshot": "run_smp_snapshot", + "restore-processors": "run_restore_processors", + "restore-downtime": "run_restore_downtime", + "snapshot-tiers": "run_snapshot_tiers", + } + self.assertEqual(tuple(dispatch), microvm_tests.DEBUG_KERNEL_SCENARIOS) + with tempfile.TemporaryDirectory() as temporary: + build = Path(temporary) + config = build / "vmlinux-debug.config" + config.write_text( + "CONFIG_SOFTLOCKUP_DETECTOR=y\nCONFIG_DETECT_HUNG_TASK=y\n", + encoding="utf-8", + ) + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "kvm", + "--debug-kernel", + "--processors", + "2", + "--output-dir", + str(build / "logs"), + ] + ) + with ExitStack() as stack: + stack.enter_context( + patch.object(microvm_tests, "validate_openvmm_test_backend") + ) + stack.enter_context( + patch.object(microvm_tests, "require_file", side_effect=require) + ) + stack.enter_context( + patch.object( + microvm_tests, + "artifact_path", + side_effect=build.joinpath, + ) + ) + runs = { + scenario: stack.enter_context(patch.object(microvm_tests, runner)) + for scenario, runner in dispatch.items() + } + self.assertEqual(microvm_tests.run(args), 0) + for scenario, run in runs.items(): + with self.subTest(scenario=scenario): + run.assert_called() + self.assertEqual(run.call_args.args[1], build / "vmlinux-debug") + config.write_text("CONFIG_DETECT_HUNG_TASK=y\n", encoding="utf-8") + with self.assertRaisesRegex( + common.ScriptError, "lacks CONFIG_SOFTLOCKUP_DETECTOR=y" + ): + microvm_tests.run(args) + def test_runner_passes_processor_counts_to_the_restore_scenarios(self): def require(path: Path, _description: str) -> Path: return path diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 948f72a1f..fe8a181d0 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -2266,7 +2266,9 @@ def test_required_ci_result_policy(self): openvmm_tests = set(ci.REQUIRED_CI_OPENVMM_TEST_JOBS) openvmm_artifact_tests = set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS) microvm_tests = set(ci.REQUIRED_CI_MICROVM_TEST_JOBS) - artifacts = {ci.REQUIRED_CI_ARTIFACT_JOB} + debug_kernel = set(ci.REQUIRED_CI_MICROVM_DEBUG_JOBS) + debug_kernel_push = set(ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS) + artifacts = {ci.REQUIRED_CI_ARTIFACT_JOB, ci.REQUIRED_CI_DEBUG_KERNEL_JOB} platforms = set(ci.REQUIRED_CI_PLATFORM_JOBS) cases = ( ("pull_request", True, False, False, always_successful), @@ -2298,6 +2300,7 @@ def test_required_ci_result_policy(self): | openvmm_tests | openvmm_artifact_tests | microvm_tests + | debug_kernel | artifacts | platforms | {"performance-gate"}, @@ -2344,6 +2347,8 @@ def test_required_ci_result_policy(self): | openvmm_tests | openvmm_artifact_tests | microvm_tests + | debug_kernel + | debug_kernel_push | platforms, ), ) @@ -2669,6 +2674,21 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): "openvmm-windows-msvc", "run-nvx-microvm-tests.yml", ), + "nvx-microvm-debug-kvm": ( + "build-openvmm-linux-gnu", + "openvmm-linux-gnu", + "run-nvx-microvm-tests.yml", + ), + "nvx-microvm-debug-mshv": ( + "build-openvmm-linux-gnu", + "openvmm-linux-gnu", + "run-nvx-microvm-tests.yml", + ), + "nvx-microvm-debug-whp": ( + "build-openvmm-windows-msvc", + "openvmm-windows-msvc", + "run-nvx-microvm-tests.yml", + ), "platform-kvm": ( "build-openvmm-linux-gnu", "openvmm-linux-gnu", @@ -2688,8 +2708,13 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): for job_name, (producer, artifact, reusable_workflow) in consumers.items(): with self.subTest(consumer=job_name): job = _workflow_job(workflow, job_name) + debug_kernel = ( + "debug-kernel, " + if job_name.startswith("nvx-microvm-debug-") + else "" + ) self.assertIn( - f"needs: [artifacts, {producer}, openvmm-changes]", + f"needs: [artifacts, {debug_kernel}{producer}, openvmm-changes]", job, ) self.assertIn(f"needs.{producer}.result == 'success'", job) @@ -2717,7 +2742,7 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): self.assertEqual(build_workflow.count("overwrite: true"), 2) self.assertEqual(build_workflow.count("retention-days: 1"), 2) for consumer_workflow, download_count in ( - (microvm_workflow, 3), + (microvm_workflow, 4), (platform_workflow, 2), ): self.assertIn("path: openvmm/target/release", consumer_workflow) @@ -2962,6 +2987,83 @@ def test_guest_artifact_restore_only_leaves_restore_and_scratch_steps_active(sel continue self.assertNotIn("restore-only", condition) + def test_ci_runs_the_debug_kernel_on_kvm_per_pull_request_and_all_on_dev(self): + workflow = ( + BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" + ).read_text(encoding="utf-8") + microvm_workflow = ( + BuildConstants.REPO_ROOT + / ".github" + / "workflows" + / "run-nvx-microvm-tests.yml" + ).read_text(encoding="utf-8") + + artifacts = _workflow_job(workflow, "artifacts") + self.assertNotIn("debug-kernel", artifacts) + self.assertNotIn("vmlinux-debug", artifacts) + # The debug kernel builds on GitHub-hosted capacity beside the shared + # artifacts, under the same condition, and only the debug jobs wait. + debug_build = _workflow_job(workflow, "debug-kernel") + self.assertIn("runs-on: ubuntu-latest", debug_build) + self.assertIn('debug-kernel: "true"', debug_build) + self.assertIn('guest-images: "false"', debug_build) + self.assertIn("name: guest-debug-kernel", debug_build) + for name in ( + KernelBuildConstants.DEBUG_BINARY_NAME, + KernelBuildConstants.DEBUG_CONFIG_NAME, + ): + self.assertIn(f"build/{name}\n", debug_build) + + def condition(job: str) -> str: + return job[job.index(" if: >-\n") : job.index(" runs-on:")] + + self.assertEqual(condition(debug_build), condition(artifacts)) + for consumer in ("release", "performance-persist"): + with self.subTest(consumer=consumer): + job = _workflow_job(workflow, consumer) + self.assertIn(" - debug-kernel\n", job) + self.assertIn("needs.debug-kernel.result == 'success' &&", job) + kvm = _workflow_job(workflow, "nvx-microvm-debug-kvm") + self.assertIn("github.event.pull_request.head.repo.full_name", kvm) + for job_name in ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS: + with self.subTest(job=job_name): + job = _workflow_job(workflow, job_name) + self.assertIn("github.event_name == 'push' &&", job) + self.assertNotIn("pull_request", job) + for job_name in ( + *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, + *ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS, + ): + with self.subTest(job=job_name): + job = _workflow_job(workflow, job_name) + self.assertIn("debug-kernel: true", job) + self.assertIn("needs.debug-kernel.result == 'success' &&", job) + self.assertIn(job_name, ci.REQUIRED_CI_RESULT_ENVIRONMENTS) + for consumer in ("release", "performance-persist"): + self.assertIn( + f"needs.{job_name}.result == 'success' ||\n" + f" needs.{job_name}.result == 'skipped'", + _workflow_job(workflow, consumer), + ) + + self.assertIn(" debug-kernel:\n", microvm_workflow) + self.assertIn(" type: boolean\n", microvm_workflow) + download = microvm_workflow.index("name: guest-debug-kernel") + self.assertIn( + "if: inputs.debug-kernel", + microvm_workflow[download - 120 : download], + ) + self.assertEqual( + microvm_workflow.count( + "${{ inputs.debug-kernel && '--debug-kernel' || '' }}" + ), + 2, + ) + # The debug run replaces the Ubuntu and Azure Linux guest tests and the + # crate lifecycle test, which boots the production kernel, rather than + # adding to them. + self.assertEqual(microvm_workflow.count("!inputs.debug-kernel"), 9) + def test_benchmarks_rely_on_the_microvm_correctness_jobs(self): # #286: the benchmark action ran a second smp-lapic gate before # acceptance. The microVM correctness jobs already gate the required @@ -5728,6 +5830,33 @@ def test_ci_debug_kernel_is_opt_in_and_keyed_by_its_fragment(self): 2, ) + def test_ci_guest_images_input_leaves_only_the_debug_kernel(self): + action = ( + BuildConstants.REPO_ROOT + / ".github" + / "actions" + / "build-guest-artifacts" + / "action.yml" + ).read_text(encoding="utf-8") + guard = "inputs.guest-images == 'true'" + self.assertIn(" guest-images:\n", action) + self.assertIn('default: "true"', action) + steps = [ + line.strip().removeprefix("- name: ") + for line in action.splitlines() + if line.startswith(" - name: ") + ] + debug_steps = {name for name in steps if "debug kernel" in name} + self.assertEqual(len(debug_steps), 3) + always = {"Resolve guest artifact versions", "Resolve guest cache keys"} + for name in steps: + with self.subTest(step=name): + step = _composite_action_step(action, name) + if name in debug_steps or name in always: + self.assertNotIn("guest-images", step) + else: + self.assertIn(guard, step) + def test_ci_guest_cache_keys_include_shared_build_modules(self): action = ( BuildConstants.REPO_ROOT From d3d766468e4b038073ab4121b7779beab5417132 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 07:13:43 -0700 Subject: [PATCH 114/188] ci: read restore packet v4 entropy in the snapshot-core scenario snapshot-core read OpenVMM's version-1 restore packet itself, 83 one-byte reads of portb with the OPENVMM_ENTROPY_V1 magic, to compare the generation ID with the packet's entropy and to reseed the guest. Under the time ABI, nvx-time capture reads restore packet v4 during the restore and consumes it, and for an untiered restore without targets leaves its 64 entropy bytes in /run/nvx/restore-entropy for the caller. The scenario now reads the generation ID with nvx-time generation-id, checks that the entropy file holds 64 bytes whose first 16 are the new generation ID, and reseeds from it. Failure codes are unchanged. Lands with OpenVMM's restore packet v4 and the guest's nvx-time wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../microvm_test_scripts/snapshot-core.sh | 24 ++++------ scripts/test_microvm_tests.py | 47 +++++++++++++++++++ 2 files changed, 56 insertions(+), 15 deletions(-) diff --git a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh index 5e7dab64c..60e38a6b0 100644 --- a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh +++ b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh @@ -6,8 +6,7 @@ fail() { exit "$code" } -generation_id_before="$(/sbin/nvx-port-io read-generation-id 233 234)" || - fail 48 +generation_id_before="$(/sbin/nvx-time generation-id)" || fail 48 [ "${#generation_id_before}" -eq 32 ] || fail 49 sleep 3600 & sleeping_pid=$! sleep 5 & timer_pid=$! @@ -35,23 +34,18 @@ echo "NVX-SNAPSHOT-DOWNTIME-$((wall_restored - wall_before))-$((uptime_restored echo "NVX-SNAPSHOT-UPTIME-CS-$((uptime_restored_cs - uptime_before_cs))" echo "NVX-SNAPSHOT-CPU-$((process_cpu_restored - process_cpu_before))-$((thread_cpu_restored - thread_cpu_before))" echo "NVX-SNAPSHOT-TIMER-WAIT-$((timer_wait_after - timer_wait_before))" -generation_id_after="$(/sbin/nvx-port-io read-generation-id 233 234)" || - fail 50 +generation_id_after="$(/sbin/nvx-time generation-id)" || fail 50 [ "$generation_id_before" != "$generation_id_after" ] || fail 51 -printf '\245' | dd of=/dev/port bs=1 seek=234 count=1 conv=notrunc 2>/dev/null -rm -f /tmp/nvx-snapshot-packet /tmp/nvx-snapshot-entropy -index=0 -while [ "$index" -lt 83 ]; do - dd if=/dev/port bs=1 skip=233 count=1 2>/dev/null >>/tmp/nvx-snapshot-packet - index=$((index + 1)) -done -head -c 18 /tmp/nvx-snapshot-packet | grep -q OPENVMM_ENTROPY_V1 || fail 44 -tail -c 64 /tmp/nvx-snapshot-packet >/tmp/nvx-snapshot-entropy +# nvx-time capture consumed restore packet v4 and, for an untiered restore +# without targets, left its 64 entropy bytes to the caller. The generation ID +# is their first 16 bytes. +entropy=/run/nvx/restore-entropy +[ "$(wc -c <"$entropy")" -eq 64 ] || fail 44 generation_id_packet="$( - head -c 16 /tmp/nvx-snapshot-entropy | od -An -tx1 -v | tr -d '[:space:]' + head -c 16 "$entropy" | od -An -tx1 -v | tr -d '[:space:]' )" [ "$generation_id_after" = "$generation_id_packet" ] || fail 52 -/sbin/nvx-reseed /tmp/nvx-snapshot-entropy || fail 45 +/sbin/nvx-reseed "$entropy" || fail 45 rng="$(/sbin/nvx-reseed --sample)" || fail 53 uuid="$(cat /proc/sys/kernel/random/uuid)" temp_path="$(mktemp /tmp/nvx-clone.XXXXXX)" || fail 54 diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index a7a6ed3cd..a2bdb1825 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -1892,6 +1892,53 @@ def test_snapshot_core_script_handles_no_clocksource(self): self.assertIn("NVX-SNAPSHOT-UUID-", script) self.assertIn("NVX-SNAPSHOT-TEMP-ID-", script) + def test_snapshot_core_reads_the_entropy_that_nvx_time_kept(self): + # Restore packet v4 is consumed by nvx-time capture, which leaves the + # entropy of an untiered restore in /run/nvx/restore-entropy. + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + script = microvm_tests._read_script("snapshot-core.sh") + self.assertNotIn("OPENVMM_ENTROPY_V1", script) + self.assertNotIn("/dev/port", script) + section = ( + "generation_id_after=" + + script.split("generation_id_after=", 1)[1].split("rng=", 1)[0] + ) + generation = bytes(range(16)) + cases = ( + (generation + bytes(48), "00" * 16, "reseeded", 0), + (generation + bytes(47), "00" * 16, "FAIL 44", 44), + (bytes(16) + bytes(48), "00" * 16, "FAIL 52", 52), + (generation + bytes(48), generation.hex(), "FAIL 51", 51), + ) + for entropy, before, expected, status in cases: + with self.subTest(expected=expected): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "restore-entropy" + path.write_bytes(entropy) + body = ( + section.replace("/run/nvx/restore-entropy", path.as_posix()) + .replace("/sbin/nvx-time", "nvx_time") + .replace("/sbin/nvx-reseed", "nvx_reseed") + ) + result = subprocess.run( + [shell], + input=( + "set -eu\n" + 'fail() { echo "FAIL $1"; exit "$1"; }\n' + f"nvx_time() {{ echo {generation.hex()}; }}\n" + 'nvx_reseed() { echo "reseeded"; }\n' + f"generation_id_before={before}\n" + body + ), + text=True, + capture_output=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode, status, result.stderr) + self.assertIn(expected, result.stdout) + def test_smp_worker_requires_bounded_local_timer_progress(self): shell = _posix_shell() if shell is None: From c50b087bc9dadc42411b7cee7173428444bc638c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 08:31:45 -0700 Subject: [PATCH 115/188] ci: qualify VM runners with OpenVMM's profile check and preflight Host qualification gates on the CPU profile through OpenVMM's fingerprint tool (H2) and preflight verification (H3, openvmm --x-time-abi-verify). Both need the OpenVMM binary, and H3 also the guest artifacts, which the microVM and platform jobs download after the action used to run, so validate-runner qualified every job with --no-openvmm. validate-runner gains a verify-openvmm input that drops --no-openvmm, adds H3, and writes the fingerprint to $RUNNER_TEMP. A failed qualification uploads the fingerprint, which names the host's whole CPU surface for a new generation or profile revision. The microVM and platform workflows now run the action right after downloading OpenVMM and the guest artifacts, so H4 also compares the measured TSC rate with the backend's native rate. The platform workflow downloads the guest artifacts itself instead of inside run-benchmark, so they exist before qualification. Jobs that only build or unit-test OpenVMM keep H1, H2 without the profile check, and H4. Lands with the OpenVMM flip: until then the pinned OpenVMM has neither --cpu-fingerprint nor the time ABI's verification mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/run-benchmark/action.yml | 10 ++-- .github/actions/validate-runner/action.yml | 32 +++++++++--- .github/workflows/run-nvx-microvm-tests.yml | 12 +++-- .github/workflows/run-platform.yml | 18 +++++-- doc/ci.md | 36 +++++++------ scripts/test_nvx_tools.py | 57 +++++++++++++++++++-- 6 files changed, 120 insertions(+), 45 deletions(-) diff --git a/.github/actions/run-benchmark/action.yml b/.github/actions/run-benchmark/action.yml index 4920a2831..279778654 100644 --- a/.github/actions/run-benchmark/action.yml +++ b/.github/actions/run-benchmark/action.yml @@ -1,5 +1,7 @@ name: Run benchmark -description: Run and publish an OpenVMM performance series +description: >- + Run and publish an OpenVMM performance series; the caller downloads the + guest artifacts and OpenVMM into their build paths inputs: backend: @@ -35,12 +37,6 @@ inputs: runs: using: composite steps: - - name: Download guest artifacts - uses: actions/download-artifact@v8 - with: - name: guest-artifacts - path: build - - name: Run Linux acceptance test if: runner.os != 'Windows' shell: bash diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index 5cf0e156d..210de1674 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -5,6 +5,13 @@ inputs: backend: description: Virtualization backend expected on the runner required: true + verify-openvmm: + description: >- + Also check the CPU profile with the downloaded OpenVMM (doctor H2) and + verify its time ABI preflight (doctor H3); OpenVMM and the guest + artifacts must be in their build paths + required: false + default: "false" runs: using: composite @@ -119,11 +126,11 @@ runs: # Time ABI host qualification (doc/design/time-abi.md, "Host # qualification"): the backend, the CPU fingerprint and generation, and - # the TSC rate stability on its short CI schedule. It replaces the - # nonstop_tsc check, which is now recorded as evidence only. Jobs qualify - # before they have an OpenVMM binary, so H2 does not run OpenVMM's CPU - # profile check. The guest warp probe runs in the microVM scenarios, - # which boot the time ABI. + # the TSC rate stability on its short CI schedule, and, in jobs that + # downloaded OpenVMM, its CPU profile check (H2) and preflight (H3). Other + # jobs qualify without OpenVMM. It replaces the nonstop_tsc check, which + # is now recorded as evidence only. The guest warp probe runs in the + # microVM scenarios, which boot the time ABI. - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash @@ -131,7 +138,7 @@ runs: python3 scripts/nvx.py doctor --backend "${{ inputs.backend }}" --checks H1 H2 H4 - --no-openvmm + ${{ inputs.verify-openvmm == 'true' && 'H3 --cpu-fingerprint "${RUNNER_TEMP}/nvx-cpu-fingerprint.json"' || '--no-openvmm' }} --ci-schedule --summary "${GITHUB_STEP_SUMMARY}" @@ -142,10 +149,21 @@ runs: python scripts\nvx.py doctor --backend "${{ inputs.backend }}" --checks H1 H2 H4 - --no-openvmm + ${{ inputs.verify-openvmm == 'true' && 'H3 --cpu-fingerprint "$env:RUNNER_TEMP\nvx-cpu-fingerprint.json"' || '--no-openvmm' }} --ci-schedule --summary "$env:GITHUB_STEP_SUMMARY" + # A failed CPU profile check keeps OpenVMM's fingerprint, which names the + # host's whole CPU surface for a new generation or profile revision. + - name: Upload the CPU fingerprint + if: failure() && inputs.verify-openvmm == 'true' + uses: actions/upload-artifact@v7 + with: + path: ${{ runner.temp }}/nvx-cpu-fingerprint.json + name: cpu-fingerprint-${{ github.job }}-${{ inputs.backend }}-${{ runner.name }}-attempt-${{ github.run_attempt }} + if-no-files-found: ignore + overwrite: true + retention-days: 7 - name: Validate KVM if: inputs.backend == 'kvm' shell: bash diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index b1fea0ae1..612963e4b 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -45,11 +45,6 @@ jobs: with: persist-credentials: false - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: ${{ inputs.backend }} - - name: Validate NVX CLI uses: ./.github/actions/validate-nvx @@ -91,6 +86,13 @@ jobs: shell: bash run: chmod +x openvmm/target/release/openvmm + # Qualification verifies the downloaded OpenVMM's time ABI preflight. + - name: Validate runner + uses: ./.github/actions/validate-runner + with: + backend: ${{ inputs.backend }} + verify-openvmm: "true" + - name: Run NVX microVM tests on Linux if: runner.os != 'Windows' shell: bash diff --git a/.github/workflows/run-platform.yml b/.github/workflows/run-platform.yml index 11ce747f7..f63fe0387 100644 --- a/.github/workflows/run-platform.yml +++ b/.github/workflows/run-platform.yml @@ -66,11 +66,6 @@ jobs: with: persist-credentials: false - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: ${{ inputs.backend }} - - name: Validate NVX CLI uses: ./.github/actions/validate-nvx @@ -89,6 +84,12 @@ jobs: shell: powershell run: python scripts\nvx.py verify + - name: Download guest artifacts + uses: actions/download-artifact@v8 + with: + name: guest-artifacts + path: build + - name: Download OpenVMM executable uses: actions/download-artifact@v8 with: @@ -106,6 +107,13 @@ jobs: shell: bash run: chmod +x openvmm/target/release/openvmm + # Qualification verifies the downloaded OpenVMM's time ABI preflight. + - name: Validate runner + uses: ./.github/actions/validate-runner + with: + backend: ${{ inputs.backend }} + verify-openvmm: "true" + - name: Run benchmark uses: ./.github/actions/run-benchmark with: diff --git a/doc/ci.md b/doc/ci.md index 90643b317..b2fb6ab2b 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -106,16 +106,18 @@ development release, and performance persistence. The GitHub-hosted own key, so a kernel rebuild delays only the debug jobs, and a failed debug kernel build fails the required status check and blocks the release. -Every job that uses the `validate-runner` action first qualifies its runner -for the time ABI with `nvx.py doctor --checks H1 H2 H4 --no-openvmm ---ci-schedule` (see [Host qualification](#host-qualification)): the backend, -the CPU fingerprint and generation, and the TSC rate stability on its short -schedule. The job has no OpenVMM binary yet, so H2 skips OpenVMM's CPU profile -check. This replaces the earlier `nonstop_tsc` check and takes a few seconds. -It reports the CPU generation, the CPU profile that `auto` selects, and the -measured TSC rate in the log and the job summary, and fails the job with a -stable code when the runner is not qualified, for example -`E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation. +Every job that uses the `validate-runner` action qualifies its runner for the +time ABI with `nvx.py doctor --checks H1 H2 H4 --ci-schedule` (see +[Host qualification](#host-qualification)): the backend, the CPU fingerprint +and generation, and the TSC rate stability on its short schedule. The microVM +and platform jobs run it after downloading OpenVMM and add OpenVMM's CPU +profile check (H2) and preflight (H3), and keep the fingerprint as an artifact +when the profile check fails; the other jobs pass `--no-openvmm`. This replaces +the earlier `nonstop_tsc` check and takes a few seconds. It reports the CPU +generation, the CPU profile that `auto` selects, and the measured TSC rate in +the log and the job summary, and fails the job with a stable code when the +runner is not qualified, for example `E_PROFILE_HOST_UNKNOWN` on an unknown CPU +generation. Qualification gates only on measured properties, alike on every backend: the host OS's invariant-TSC flags and clocksource are recorded as evidence, and the guest warp probe in the microVM scenarios measures the skew that a host @@ -255,13 +257,13 @@ Qualification gates on measured properties, alike on every backend: the guest warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource are evidence only. In CI, `validate-runner` runs the cheap host-level checks -H1, H2, and H4 before every job, with `--no-openvmm`, because jobs download -OpenVMM and the guest artifacts only later, and with `--ci-schedule`. The -guest warp probe needs a time ABI boot, so it belongs in the microVM boot and -restore scenarios, which run CI's warp schedule after every boot and restore -and assert its verdict once OpenVMM boots the time ABI by default. Without H3, -H4 checks the rate stability without comparing it against the backend's rate. -H5 and H7 remain available for interactive qualification. +H1, H2, and H4 in every job that uses it, on the short `--ci-schedule`. The +microVM and platform jobs run it after downloading OpenVMM and the guest +artifacts and add H2's CPU profile check and H3, so that H4 also compares the +measured rate with the backend's native rate; the other jobs pass +`--no-openvmm`. The guest warp probe needs a time ABI boot, so the microVM boot +and restore scenarios run CI's warp schedule after every boot and restore and +assert its verdict. H5 and H7 remain available for interactive qualification. ## Rust crate diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index fe8a181d0..7b947f17a 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -2743,7 +2743,7 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): self.assertEqual(build_workflow.count("retention-days: 1"), 2) for consumer_workflow, download_count in ( (microvm_workflow, 4), - (platform_workflow, 2), + (platform_workflow, 3), ): self.assertIn("path: openvmm/target/release", consumer_workflow) self.assertIn("path: build", consumer_workflow) @@ -3346,14 +3346,63 @@ def test_runners_qualify_their_host_time_before_every_job(self): self.assertIn(command, step) self.assertIn('--backend "${{ inputs.backend }}"', step) self.assertIn("--checks H1 H2 H4\n", step) - self.assertIn("--no-openvmm\n", step) + fingerprint = ( + '"${RUNNER_TEMP}/nvx-cpu-fingerprint.json"' + if shell == "bash" + else '"$env:RUNNER_TEMP\\nvx-cpu-fingerprint.json"' + ) + self.assertIn( + "${{ inputs.verify-openvmm == 'true' && " + f"'H3 --cpu-fingerprint {fingerprint}' || '--no-openvmm' }}}}\n", + step, + ) self.assertIn("--ci-schedule\n", step) self.assertIn(summary, step) args = nvx.parse_args( - ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "--no-openvmm"] + ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4", "H3"] + + ["--cpu-fingerprint", "fingerprint.json", "--ci-schedule"] + ) + self.assertEqual(args.checks, ["H1", "H2", "H4", "H3"]) + self.assertEqual(args.cpu_fingerprint, Path("fingerprint.json")) + args = nvx.parse_args( + ["doctor", "--backend", "kvm", "--checks", "H1", "H2", "H4"] + + ["--no-openvmm", "--ci-schedule"] ) - self.assertEqual(args.checks, ["H1", "H2", "H4"]) self.assertTrue(args.no_openvmm) + self.assertTrue(args.ci_schedule) + self.assertIn(" verify-openvmm:\n", validate_runner) + self.assertIn(' default: "false"\n', validate_runner) + # A failed CPU profile check keeps OpenVMM's fingerprint. + upload = validate_runner.split(" - name: Upload the CPU fingerprint\n")[1] + upload = upload.split("\n\n", 1)[0] + self.assertIn("if: failure() && inputs.verify-openvmm == 'true'\n", upload) + self.assertIn("path: ${{ runner.temp }}/nvx-cpu-fingerprint.json\n", upload) + self.assertIn("if-no-files-found: ignore\n", upload) + # Jobs that run OpenVMM qualify the runner after downloading it. + benchmark = ( + BuildConstants.REPO_ROOT + / ".github" + / "actions" + / "run-benchmark" + / "action.yml" + ).read_text(encoding="utf-8") + self.assertNotIn("download-artifact", benchmark) + for workflow_name in ("run-nvx-microvm-tests.yml", "run-platform.yml"): + with self.subTest(workflow=workflow_name): + workflow = ( + BuildConstants.REPO_ROOT / ".github" / "workflows" / workflow_name + ).read_text(encoding="utf-8") + validate = workflow.index(" - name: Validate runner\n") + self.assertEqual(workflow.count(" - name: Validate runner\n"), 1) + self.assertIn( + 'verify-openvmm: "true"', workflow[validate : validate + 220] + ) + for step in ( + " - name: Download guest artifacts\n", + " - name: Download OpenVMM executable\n", + " - name: Make OpenVMM executable\n", + ): + self.assertLess(workflow.index(step), validate) self.assertLess( validate_runner.index("Validate Linux toolchain"), validate_runner.index("Qualify host time on Linux"), From 4a1ad6682efe316c05d3a18aa7f4fde01e503c61 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 11:02:28 -0700 Subject: [PATCH 116/188] ci: print the time ABI evidence of every microVM test run CI uploads the guest logs of a microVM test run only when it fails, so a green run keeps none of its time ABI evidence. The reliability window before the time ABI lands records that evidence for every attempt. test-microvm now ends a passing run with one NVX-TIME-ABI-EVIDENCE line, which it also adds to the GitHub job summary: the number of warp probe runs with their worst max_abs_offset_ns and max_backward_ns, the count and elapsed_us range of the boot, capture, and restore markers, the exhaustive check's summary, and the restore-downtime stall counts. A run without such lines prints nothing. Lands with the OpenVMM time ABI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 7 +++ scripts/nvx_tools/microvm_tests.py | 86 ++++++++++++++++++++++++++++++ scripts/test_microvm_tests.py | 74 ++++++++++++++++++++++++- 3 files changed, 166 insertions(+), 1 deletion(-) diff --git a/doc/ci.md b/doc/ci.md index b2fb6ab2b..abe82b1e5 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -88,6 +88,13 @@ failures, and exit status 0; a failure lists each failing check with the guest's detail. The guest command fits on one console line, so the console's echo of it ends before the check prints. +At the end of a passing run, `test-microvm` prints one `NVX-TIME-ABI-EVIDENCE:` +line and adds it to the GitHub job summary, because CI keeps the guest logs only +for failed jobs: the number of warp probe runs with their worst +`max_abs_offset_ns` and `max_backward_ns`, the count and `elapsed_us` range of +the boot, capture, and restore markers, the exhaustive check's summary, and the +`restore-downtime` stall counts. + The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run `test-microvm --debug-kernel` on the CI debug kernel (`build/vmlinux-debug`, built from `kernel/config-microvm-debug`), whose soft-lockup and hung-task diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 580b1a6b7..8481a9842 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -63,9 +63,11 @@ ABI_VERSION, WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_FAILURE_MARKER, + WARP_SUMMARY_PREFIX, check_warp_probe, describe_exit_status, parse_fields, + parse_marker, warp_probe_script, warp_rounds, ) @@ -202,6 +204,10 @@ TIME_ABI_EXHAUSTIVE_EXIT_PREFIX = "NVX-EXHAUSTIVE-EXIT status=" TIME_ABI_EXHAUSTIVE_COMPLETION_MARKER = b"NVX-EXHAUSTIVE-DONE" TIME_ABI_EXHAUSTIVE_REPORTED_FAILURES = 8 +# One line per test-microvm run that sums up the time ABI evidence in its guest +# logs, which CI uploads only for failed jobs. +TIME_ABI_EVIDENCE_PREFIX = "NVX-TIME-ABI-EVIDENCE: " +RESTORE_DOWNTIME_REPORT_PREFIX = "NVX-RESTORE-DOWNTIME " OUTCOME_TOP_LEVEL_FIELDS = frozenset( { "schema_version", @@ -1342,6 +1348,80 @@ def _check_exhaustive_report(text: str, *, processors: int) -> None: raise RuntimeError("time ABI exhaustive check: " + "; ".join(problems)) +def time_abi_evidence(output_dir: Path) -> dict[str, str]: + """Sum up the time ABI evidence in the guest logs of one run.""" + offsets: list[int] = [] + backward: list[int] = [] + elapsed: dict[str, list[int]] = {"boot": [], "capture": [], "restore": []} + exhaustive: list[str] = [] + stalls: list[str] = [] + for path in sorted(output_dir.rglob("*.log")): + for raw in path.read_text(encoding="utf-8", errors="replace").splitlines(): + line = raw.removesuffix("\r") + try: + if line.startswith(WARP_SUMMARY_PREFIX): + fields = parse_fields(line.removeprefix(WARP_SUMMARY_PREFIX)) + offsets.append(int(fields["max_abs_offset_ns"])) + backward.append(int(fields["max_backward_ns"])) + elif line.startswith(TIME_ABI_EXHAUSTIVE_PREFIX): + fields = parse_fields(line.removeprefix(TIME_ABI_EXHAUSTIVE_PREFIX)) + if "check" not in fields: + exhaustive.append( + f"{fields['status']}/{fields['cpus']}/{fields['failures']}" + ) + elif line.startswith(RESTORE_DOWNTIME_REPORT_PREFIX): + fields = parse_fields( + line.removeprefix(RESTORE_DOWNTIME_REPORT_PREFIX) + ) + stalls.append(fields["stalls"]) + else: + marker = parse_marker(line) + if ( + marker is not None + and marker["status"] == "ok" + and marker["phase"] in elapsed + ): + elapsed[marker["phase"]].append(int(marker["elapsed_us"])) + except (KeyError, ValueError): + continue + evidence: dict[str, str] = {} + if offsets: + evidence["warp_runs"] = str(len(offsets)) + evidence["warp_max_abs_offset_ns"] = str(max(offsets)) + evidence["warp_max_backward_ns"] = str(max(backward)) + for phase, values in elapsed.items(): + if values: + evidence[f"{phase}_markers"] = str(len(values)) + evidence[f"{phase}_elapsed_us"] = f"{min(values)}-{max(values)}" + if exhaustive: + evidence["exhaustive"] = ",".join(exhaustive) + if stalls: + evidence["downtime_stalls"] = ",".join(stalls) + return evidence + + +def report_time_abi_evidence( + output_dir: Path, *, backend: str, guest: str, debug_kernel: bool +) -> str | None: + """Print the run's evidence line and add it to the GitHub job summary.""" + evidence = time_abi_evidence(output_dir) + if not evidence: + return None + fields = { + "backend": backend, + "guest": guest, + "kernel": "debug" if debug_kernel else "default", + **evidence, + } + line = TIME_ABI_EVIDENCE_PREFIX + " ".join(f"{k}={v}" for k, v in fields.items()) + print(line) + summary = os.environ.get("GITHUB_STEP_SUMMARY") + if summary: + with open(summary, "a", encoding="utf-8") as handle: + handle.write(f"\n`{line}`\n") + return line + + def run_time_abi_conformance( executable: Path, kernel: Path, @@ -4733,5 +4813,11 @@ def run(args: argparse.Namespace) -> int: log_path=output_dir / "virtio-net.log", ) + report_time_abi_evidence( + output_dir, + backend=args.backend, + guest=descriptor.name, + debug_kernel=debug_kernel, + ) print(f"Wrote microVM correctness logs to {output_dir}") return 0 diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index a2bdb1825..096ee0397 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2,7 +2,9 @@ # pyright: reportPrivateUsage=false import argparse +import io import json +import os import queue import re import shutil @@ -12,7 +14,7 @@ import tempfile import unittest from collections.abc import Callable -from contextlib import ExitStack +from contextlib import ExitStack, redirect_stdout from pathlib import Path from typing import cast from unittest.mock import MagicMock, patch @@ -3617,6 +3619,76 @@ def test_conformance_guest_line_reports_the_check_exit_status(self): result.stderr, ) + def test_time_abi_evidence_sums_up_the_guest_logs(self): + boot = ( + "NVX-TIME-ABI: v=1 phase=boot status=ok cpus=8 tsc_hz=2793437000 " + "lapic_hz=1000000000 generation=0 elapsed_us={elapsed}\r\n" + ) + restore = boot.replace("phase=boot", "phase=restore").replace( + "generation=0", "generation=1" + ) + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + (output_dir / "smp-8.log").write_bytes( + boot.format(elapsed=10894).encode() + + _warp_probe_output(8, offset_ns=41) + ) + (output_dir / "restore-downtime-8-vcpu.log").write_bytes( + restore.format(elapsed=1401).encode() + + _warp_probe_output(8, offset_ns=3) + + b"NVX-RESTORE-DOWNTIME stalls=0 uptime_s=37\r\n" + + restore.format(elapsed=6410).encode() + ) + (output_dir / "time-abi-conformance.log").write_bytes( + b'/ # /sbin/nvx-time exhaustive; echo "NVX-EXHAUSTIVE-EXIT status=$?"\r\n' + b'NVX-TIME-ABI-EXHAUSTIVE: v=1 check=X1 cpu=0 status=pass detail=""\r\n' + b"NVX-TIME-ABI-EXHAUSTIVE: v=1 status=ok cpus=8 failures=0\r\n" + + boot.format(elapsed=1384).encode() + ) + (output_dir / "lifecycle.log").write_text("no time ABI lines\n") + evidence = microvm_tests.time_abi_evidence(output_dir) + self.assertEqual( + evidence, + { + "warp_runs": "4", + "warp_max_abs_offset_ns": "41", + "warp_max_backward_ns": "0", + "boot_markers": "2", + "boot_elapsed_us": "1384-10894", + "restore_markers": "2", + "restore_elapsed_us": "1401-6410", + "exhaustive": "ok/8/0", + "downtime_stalls": "0", + }, + ) + + def test_time_abi_evidence_line_goes_to_the_log_and_the_job_summary(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) / "out" + output_dir.mkdir() + empty = Path(temporary) / "empty" + empty.mkdir() + summary = Path(temporary) / "summary.md" + (output_dir / "smp-2.log").write_bytes(_warp_probe_output(2, offset_ns=7)) + with ( + patch.dict(os.environ, {"GITHUB_STEP_SUMMARY": str(summary)}), + redirect_stdout(io.StringIO()) as stdout, + ): + line = microvm_tests.report_time_abi_evidence( + output_dir, backend="mshv", guest="alpine", debug_kernel=True + ) + nothing = microvm_tests.report_time_abi_evidence( + empty, backend="kvm", guest="alpine", debug_kernel=False + ) + self.assertEqual( + line, + "NVX-TIME-ABI-EVIDENCE: backend=mshv guest=alpine kernel=debug " + "warp_runs=2 warp_max_abs_offset_ns=7 warp_max_backward_ns=0", + ) + self.assertIsNone(nothing) + self.assertEqual(stdout.getvalue(), f"{line}\n") + self.assertEqual(summary.read_text(encoding="utf-8"), f"\n`{line}`\n") + def test_removed_tsc_guards_stay_removed(self): # The warp probe replaced the restore-tsc-sync guard, its # clearcpuid=tsc_adjust kernel option, and the fresh-boot TSC control. From 5a92c86f1966b0ee6dc3036a5938cc778a0ff0b3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 14:15:13 -0700 Subject: [PATCH 117/188] ci: stop passing --restore-entropy, which restore packet v4 replaces Spec f62c4cd: every time ABI restore exposes restore packet v4, which carries fresh entropy, so --restore-entropy has no effect. OpenVMM still accepts it, and the harness stops passing it. The nvx.py run wrapper and benchmark's snapshot_restore_command, which every restore scenario and benchmark uses, drop the option, and the manual restore command in doc/run.md follows the wrapper. Lands with the OpenVMM flip: until then the option still gives the restored guest its entropy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/run.md | 9 +++++---- scripts/nvx.py | 4 +--- scripts/nvx_tools/benchmark.py | 1 - scripts/test_nvx_tools.py | 3 ++- 4 files changed, 8 insertions(+), 9 deletions(-) diff --git a/doc/run.md b/doc/run.md index f3277e1a9..3c4d6f77c 100644 --- a/doc/run.md +++ b/doc/run.md @@ -248,9 +248,11 @@ translations and additions: | `--dry-run` | No equivalent; this only prints the generated command | Always include `--single-process`. When restoring, omit `--memory`, `--kernel`, -and `--initrd`, and add `--restore-entropy`; the wrapper adds this option -automatically. Do not pass `--guest ubuntu` during restore; the captured RAM -and machine contract already identify the restored guest. For example: +and `--initrd`. Every restore gives the guest fresh entropy through the time +ABI's restore packet, so `--restore-entropy` is no longer needed; OpenVMM still +accepts it without effect. Do not pass `--guest ubuntu` during restore; the +captured RAM and machine contract already identify the restored guest. For +example: ```bash ./bin/openvmm \ @@ -259,7 +261,6 @@ and machine contract already identify the restored guest. For example: --processors 8 \ --hypervisor kvm \ --restore-snapshot /var/lib/nvx/snapshot \ - --restore-entropy \ --restore-processors 4 \ --restore-memory 1024M \ --restore-ready-path /run/nvx/restore-ready.sock diff --git a/scripts/nvx.py b/scripts/nvx.py index 96a56a85d..7f20aa1cc 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -368,9 +368,7 @@ def command_run(args: argparse.Namespace) -> None: _hypervisor(args.hypervisor), ] if args.restore_snapshot is not None: - command.extend( - ["--restore-snapshot", str(args.restore_snapshot), "--restore-entropy"] - ) + command.extend(["--restore-snapshot", str(args.restore_snapshot)]) if args.restore_processors is not None: command.extend(["--restore-processors", str(args.restore_processors)]) if args.restore_memory_mib is not None: diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index c022e39d9..d488f8c48 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -4979,7 +4979,6 @@ def snapshot_restore_command( hypervisor, "--restore-snapshot", str(snapshot_path), - "--restore-entropy", ] if restore_processors is not None: command.extend(("--restore-processors", str(restore_processors))) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 7b947f17a..d856ebb8c 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -1571,7 +1571,6 @@ def test_run_exposes_restore_readiness(self): "mshv", "--restore-snapshot", "snapshot", - "--restore-entropy", "--restore-processors", "2", "--restore-ready-path", @@ -10333,6 +10332,8 @@ def test_snapshot_restore_command_separates_capacity_from_online_target(self): self.assertEqual(command[command.index("--processors") + 1], "8") self.assertEqual(command[command.index("--restore-processors") + 1], "4") + # Restore packet v4 gives every restore fresh entropy. + self.assertNotIn("--restore-entropy", command) def test_snapshot_restore_command_sets_memory_target_separately(self): command = benchmark.snapshot_restore_command( From 3a4dd7e0190fcc30e31e6cf3c9c6f819eca7738e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 15:09:35 -0700 Subject: [PATCH 118/188] ci: ask quiet time ABI guests for their status Console bytes are port exits that cost 10 to 45 us each and broke the restore performance gate, so time ABI guests keep the console quiet (spec 1efcd28 and a9ef0f5): only violation events reach it, and /sbin/nvx-time status prints one line per recorded check phase (boot, capture, restore, oldest first) and a runtime line on demand, after waiting up to 30 s for pending checks. This replaces 8dded28's automatic requirement (REQUIRE_BOOT_MARKER), which failed a cold boot whose guest reached its boot marker without a boot line. The test runners now ask: - After a cold boot, the guest-script and capture runners and OpenvmmProcess send the status query before any other input and wait for its NVX-TIME-STATUS-EXIT line. The guest's tty echoes input that arrives while a command runs, so input sent behind the query could split its lines. The query must exit 0 with a valid boot line. OpenvmmProcess asks only when the guest's boot marker appeared on its own console, so a guest whose shell is on a TCP console, or that execs a workload, is not asked. - After each smp-snapshot, restore-processors, and restore-downtime restore, the post-restore script runs the warp probe and then the status query, which waits for the restore's deferred checks. The restore line must report the backend's LAPIC rate, the restored CPU count, and generation 1, which spec bb1e88c states for every restore of a snapshot captured at generation 0. The query also repeats the source's boot and capture lines, which keep the CPU counts their checks covered, so only the restore line is checked against the online CPUs. - Doctor H6 asks through the guest-script runner. A check that is still pending when the guest stops waiting (status=pending) fails with that reason, and the harness waits 60 s so that the guest reports it first. An nvx-time without the status subcommand (exit status 2) fails with that reason, and a report-only guest fails as report-only even when it exits 1. The runtime line replaces the old non-fatal status=uncertain line and never gates. The evidence line counts each query's newest phase line, so a restored guest does not count its source's boot again. Benchmarks never ask, so their measured intervals stay quiet; a violation event or a 193/194/195 power-off still fails every run, including a boot check that fails after shell-ready. Lands with the guest's quiet-console wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 9 +- doc/ci.md | 48 +++- scripts/nvx_tools/benchmark.py | 46 +++- scripts/nvx_tools/doctor.py | 4 +- scripts/nvx_tools/microvm_tests.py | 113 ++++++-- scripts/nvx_tools/openvmm_process.py | 50 +++- scripts/nvx_tools/time_abi.py | 192 +++++++++++--- scripts/test_microvm_tests.py | 127 +++++++-- scripts/test_time_abi.py | 384 ++++++++++++++++++++++++--- 9 files changed, 845 insertions(+), 128 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 5f0444a1a..2bbd9b579 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -306,10 +306,11 @@ any guest-exit teardown timeout. With at least ten samples, it also rejects snap series whose p50 is more than 25% above p25; this prevents a transient host stall from entering performance history without hiding a uniformly slower product result. Every launch is scanned for the guest's time ABI output as in the -[microVM correctness jobs](ci.md): a cold boot must print a passing `NVX-TIME-ABI` boot line -before `ALPINE-MICROVM-BOOT-OK`, and a violation event or a time ABI power-off fails the run. -The scan parses output the harness already reads, so it adds nothing to the measured intervals; -the guest's boot check is part of `openvmm_cold_start`. +[microVM correctness jobs](ci.md): a violation event or a time ABI power-off fails the run. +Benchmarks never ask the guest for its time ABI status, as the correctness jobs do after cold +boots and restores, so the guest prints nothing extra and the scan only parses output the harness +already reads; it adds nothing to the measured intervals. The guest's boot check, which powers the +guest off with status 193 if it fails, is part of `openvmm_cold_start`. Lifecycle capture runs a deterministic affinity-pinned worker on every vCPU before the snapshot request. Explicit correctness scenarios also stage a post-restore probe. The capture probe is outside the snapshot-generation timing interval. Each worker proves that it diff --git a/doc/ci.md b/doc/ci.md index abe82b1e5..5ba70e0c4 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -28,13 +28,24 @@ per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM console for the guest's [time ABI](design/time-abi.md) output. An `NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line -fails the scenario at once with the guest's code and detail. A cold boot whose -guest reaches its boot marker without a passing `NVX-TIME-ABI` boot line also -fails at once: init runs the boot check before any other guest work, so a -missing line means the guest image or OpenVMM does not implement the ABI. The -line must report ABI version 1, generation 0, a plausible TSC rate, and the -backend's LAPIC rate; a report-only boot (`NVX-TIME-REPORT`) is never accepted. -Restores carry no boot line. An OpenVMM exit +fails the scenario at once with the guest's code and detail. Guests keep the +console quiet, because every console byte costs a port exit. Only the initial +clock step precedes the guest's boot marker; the other boot checks finish +afterwards and still power off with status 193 on failure, possibly in the +middle of a scenario. A guest prints one `NVX-TIME-ABI` line per recorded +check phase (boot, then capture and restore after a restore) and a runtime +line only when `/sbin/nvx-time status` asks, after waiting up to 30 s for +pending checks. The test runners ask after every cold boot whose shell is on +the OpenVMM console, before any other input, and wait for the query to exit, +so the console's echo of later input cannot split its lines. The query must +exit 0 with a passing `NVX-TIME-ABI` boot line, which reports ABI version 1, +generation 0, a plausible TSC rate, and the backend's LAPIC rate; a missing +line means the guest image or OpenVMM does not implement the ABI, a check +still pending after the guest's 30 s wait fails, and a report-only guest +(`NVX-TIME-REPORT`) is never accepted. A guest image whose `nvx-time` has no +`status` subcommand fails with that reason. The runtime line only records +the wall-clock discipline's state. Benchmarks never ask, so their measured +intervals stay quiet. An OpenVMM exit status of 193, 194, or 195 is reported as the guest's time ABI conformance, runtime-violation, or restore-repair power-off, together with the event that preceded it, instead of as a generic exit status. @@ -52,6 +63,14 @@ it once, and the first snapshot a second time to prove that a restore leaves it reusable. `restore-processors` captures one boot-online CPU with capacity 8 and probes the CPUs that each restore target activated. After the 1/2/4/8-CPU restores, it restores the same snapshot once without `--restore-processors`. +After the warp probe, every `smp-snapshot` and `restore-processors` restore +runs `/sbin/nvx-time status`, which waits for the restore's deferred checks; +it must exit 0 with a passing `NVX-TIME-ABI ... phase=restore` line that +reports the backend's LAPIC rate, the restored CPU count, and generation 1: +every scenario captures a cold-booted guest, and OpenVMM cannot capture a +restored one. The query also prints the source's boot and capture lines, +which keep the CPU count their checks covered, so only the restore line must +match the restored CPU count. Every restore runs with OpenVMM lifecycle profiling and must report exactly one `startup.vp_thread_bind` record. Its `startup.vp_bind_*` records must show that an explicit MSHV target binds exactly VPs `0..N-1`, while untargeted MSHV @@ -70,11 +89,11 @@ The `restore-downtime` scenario covers the time ABI's long-downtime case. It captures four snapshots, at 1 and 8 vCPUs, each with and without `rcupdate.rcu_expedited=1`, then waits 30 s, longer than the guest's 21 s RCU stall timeout, and restores each one. Every restored guest must run the warp -probe, print its `NVX-TIME-ABI ... phase=restore status=ok` line within 30 s of -the restore marker, report `/sys/kernel/rcu_stall_count` as 0 two seconds -later, and show an uptime of at least 30 s, which proves that monotonic time -advanced by the downtime. The captures share one downtime window, so the -scenario adds about a minute per backend. +probe, report a passing restore line through `nvx-time status` before the +harness stages its check, report `/sys/kernel/rcu_stall_count` as 0 two +seconds later, and show an uptime of at least 30 s, which proves that +monotonic time advanced by the downtime. The captures share one downtime +window, so the scenario adds about a minute per backend. The `time-abi-conformance` scenario boots the largest requested vCPU count and runs the guest's exhaustive CI check, `/sbin/nvx-time exhaustive`, which the boot check leaves to CI. On every online CPU it checks every leaf @@ -92,7 +111,8 @@ At the end of a passing run, `test-microvm` prints one `NVX-TIME-ABI-EVIDENCE:` line and adds it to the GitHub job summary, because CI keeps the guest logs only for failed jobs: the number of warp probe runs with their worst `max_abs_offset_ns` and `max_backward_ns`, the count and `elapsed_us` range of -the boot, capture, and restore markers, the exhaustive check's summary, and the +the newest check that each `nvx-time status` query reported (boot after a cold +boot, restore after a restore), the exhaustive check's summary, and the `restore-downtime` stall counts. The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run @@ -247,7 +267,7 @@ ABI failure code starts its detail with the code in brackets, for example | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | | H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, `CLOCK_MONOTONIC` on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | -| H6 | A microVM with the largest supported vCPU count up to 8 prints a valid `NVX-TIME-ABI` boot marker and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | +| H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | | H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows | H2, H4, and H5 use a dependency-free host probe, diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index d488f8c48..92beb8124 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -42,7 +42,7 @@ OpenVMMBuildConstants, ) from .common import bytes_to_mib, sha256_file -from .time_abi import TimeAbiMonitor +from .time_abi import TimeAbiMonitor, status_script BOOT_MARKER = b"ALPINE-MICROVM-BOOT-OK" RESTORE_MARKER = b"OPENVMM-SNAPSHOT-RESTORE-OK" @@ -2434,7 +2434,15 @@ def run_guest_script( log_path: Path | None = None, boot_marker: bytes = BOOT_MARKER, contain_process_tree: bool = False, + time_abi_status: bool = False, ) -> GuestCommandResult: + """Run ``script`` once the guest boots and wait for ``completion_marker``. + + With ``time_abi_status``, a cold boot first asks the guest for its time + ABI status and sends ``script`` only after the query exits, so the + guest's tty cannot echo script bytes into the status lines. The run + fails unless the guest reported a valid boot line. + """ environment = os.environ.copy() environment["OPENVMM_LOG"] = "off" started_ns = time.perf_counter_ns() @@ -2445,6 +2453,7 @@ def run_guest_script( ) process = interaction.process monitor = TimeAbiMonitor(command) + query_status = time_abi_status and monitor.cold_boot if windows_cpus is not None: set_windows_affinity(process.pid, windows_cpus) @@ -2454,6 +2463,7 @@ def run_guest_script( ).start() deadline = time.monotonic() + timeout output = bytearray() + status_sent = False input_sent = False completed = False peak_bytes = 0 @@ -2479,7 +2489,14 @@ def run_guest_script( output.extend(chunk) monitor.feed(chunk) peak_bytes = _try_peak_rss(process, peak_bytes) - if not input_sent and boot_marker in output: + if query_status and not status_sent and boot_marker in output: + interaction.write_input(status_script().encode("utf-8")) + status_sent = True + if ( + not input_sent + and boot_marker in output + and (not query_status or monitor.status_queries > 0) + ): interaction.write_input(script.encode("utf-8")) input_sent = True if input_sent and contains_output_line(output, completion_marker): @@ -2497,6 +2514,8 @@ def run_guest_script( if teardown_mode == "guest-exit" and returncode != 0: raise monitor.exit_error(returncode) if not input_sent: + if status_sent: + monitor.require_status("the guest exited") raise RuntimeError("guest exited before its boot marker") if not completed: raise RuntimeError( @@ -4670,11 +4689,14 @@ def capture_snapshot( post_restore_script: str | None = None, log_path: Path | None = None, boot_marker: bytes = BOOT_MARKER, + time_abi_status: bool = False, ) -> tuple[float, float, float, int]: """Capture one guest-requested snapshot and time its generation. Guest markers match only the guest console, and the profile records that - time generation come only from OpenVMM's stderr. + time generation come only from OpenVMM's stderr. With ``time_abi_status``, + the source guest first answers a time ABI status query, and the capture + starts only after the query exits. """ if snapshot_path.exists(): shutil.rmtree(snapshot_path) @@ -4685,6 +4707,7 @@ def capture_snapshot( interaction = InteractiveProcess(command, environment, separate_stderr=True) process = interaction.process monitor = TimeAbiMonitor(command) + query_status = time_abi_status and monitor.cold_boot profile = SnapshotProfileCollector( process.pid, process_started_ns, @@ -4695,6 +4718,7 @@ def capture_snapshot( output = SeparatedOutput(interaction) deadline = time.monotonic() + timeout + status_sent = False boot_seen = False snapshot_requested = False snapshot_started_ns = None @@ -4762,7 +4786,19 @@ def request_snapshot() -> None: and contains_output_line(console, SNAPSHOT_GUEST_DISPATCH_MARKER) ): snapshot_guest_dispatched_ns = time.perf_counter_ns() - if not boot_seen and boot_marker in console: + if ( + not boot_seen + and query_status + and not status_sent + and (boot_marker in console) + ): + interaction.write_input(status_script().encode("utf-8")) + status_sent = True + if ( + not boot_seen + and boot_marker in console + and (not query_status or monitor.status_queries > 0) + ): boot_seen = True if processors is None: request_snapshot() @@ -4793,6 +4829,8 @@ def request_snapshot() -> None: snapshot_published_ns = source_exited_ns if returncode != 0: raise monitor.exit_error(returncode, "snapshot source") + if status_sent: + monitor.require_status("the snapshot source exited") if not snapshot_requested: raise RuntimeError("source guest exited before its snapshot request") if not snapshot_path.is_dir(): diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 5b54a370c..ea42d3611 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -769,11 +769,13 @@ def _warp_guest( warp_probe_script(gaps) + "nvx-exit 0\n", WARP_PROBE_COMPLETION_MARKER, timeout=context.timeout, + time_abi_status=True, ) monitor = TimeAbiMonitor(command) monitor.feed(result["text"].encode()) monitor.finish() - boot = monitor.require_boot("the guest boot marker", online_cpus=vcpus) + monitor.require_status("the warp probe") + boot = monitor.require_boot("the warp probe", online_cpus=vcpus) rounds = check_warp_probe( result["text"], cpus=vcpus, context="H6", rounds=warp_rounds(vcpus, gaps) ) diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 8481a9842..24d7372bd 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -64,10 +64,13 @@ WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_FAILURE_MARKER, WARP_SUMMARY_PREFIX, + TimeAbiFailure, + TimeAbiMonitor, check_warp_probe, describe_exit_status, parse_fields, parse_marker, + status_script, warp_probe_script, warp_rounds, ) @@ -188,6 +191,9 @@ # The guest finishes a restore after the RCU grace-period release, which gives # up after rcu_cpu_stall_timeout (21 s), and the deferred C7 check. TIME_ABI_RESTORE_FINISH_SECONDS = 30.0 +# Every scenario captures a cold-booted guest (generation 0), and OpenVMM +# cannot capture a restored one, so every restore carries generation 1. +RESTORED_GENERATION = 1 # The spec's long-downtime case: longer than the 21 s RCU stall timeout, at 1 # and 8 vCPUs, with and without expedited grace periods. RESTORE_DOWNTIME_SECONDS = 30.0 @@ -307,6 +313,7 @@ def run_guest_script( log_path=log_path, boot_marker=BOOT_MARKER, contain_process_tree=contain_process_tree, + time_abi_status=True, ) @@ -339,6 +346,7 @@ def capture_snapshot( post_restore_script=post_restore_script, log_path=log_path, boot_marker=BOOT_MARKER, + time_abi_status=True, ) @@ -1349,13 +1357,20 @@ def _check_exhaustive_report(text: str, *, processors: int) -> None: def time_abi_evidence(output_dir: Path) -> dict[str, str]: - """Sum up the time ABI evidence in the guest logs of one run.""" + """Sum up the time ABI evidence in the guest logs of one run. + + nvx-time status prints every recorded phase, oldest first, and then its + runtime line, so a restored guest repeats its source's boot and capture + lines. Only each query's newest phase line counts: one boot per cold-boot + query and one restore per post-restore query. + """ offsets: list[int] = [] backward: list[int] = [] elapsed: dict[str, list[int]] = {"boot": [], "capture": [], "restore": []} exhaustive: list[str] = [] stalls: list[str] = [] for path in sorted(output_dir.rglob("*.log")): + newest: dict[str, str] | None = None for raw in path.read_text(encoding="utf-8", errors="replace").splitlines(): line = raw.removesuffix("\r") try: @@ -1376,12 +1391,18 @@ def time_abi_evidence(output_dir: Path) -> dict[str, str]: stalls.append(fields["stalls"]) else: marker = parse_marker(line) + if marker is None: + continue + if marker["phase"] != "runtime": + newest = marker + continue + last, newest = newest, None if ( - marker is not None - and marker["status"] == "ok" - and marker["phase"] in elapsed + last is not None + and last["status"] == "ok" + and last["phase"] in elapsed ): - elapsed[marker["phase"]].append(int(marker["elapsed_us"])) + elapsed[last["phase"]].append(int(last["elapsed_us"])) except (KeyError, ValueError): continue evidence: dict[str, str] = {} @@ -2618,6 +2639,35 @@ def _check_restore_warp(output: bytes, *, processors: int, context: str) -> None ) +def post_restore_checks() -> str: + """Return the guest checks that run after every restore of the restore + scenarios: the warp probe, then the time ABI status, which waits for the + restore's deferred checks.""" + return warp_probe_script() + status_script() + + +def _check_restore_status( + output: bytes, + command: Sequence[str], + *, + processors: int, + context: str, +) -> dict[str, str]: + """Validate the restore line that the post-restore status query printed.""" + monitor = TimeAbiMonitor(command) + try: + monitor.feed(output) + monitor.finish() + monitor.require_status("the restore checks finished") + return monitor.require_restore( + "the restore checks finished", + online_cpus=processors, + generation=RESTORED_GENERATION, + ) + except TimeAbiFailure as error: + raise RuntimeError(f"{context}: {error}") from error + + def run_smp_snapshot( executable: Path, kernel: Path, @@ -2651,19 +2701,20 @@ def run_smp_snapshot( timeout=timeout, processors=processors, post_restore_script=smp_probe_script(processors, exit_guest=False) - + warp_probe_script(), + + post_restore_checks(), log_path=output_dir / f"smp-snapshot-{processors}-capture.log", ) fingerprint = _snapshot_fingerprint(snapshot_path) for restore_index in range(restores): context = f"{processors}-vCPU SMP restore {restore_index}" + restore_command = snapshot_restore_command( + executable, + backend, + snapshot_path, + processors=processors, + ) output = _measure_restore( - snapshot_restore_command( - executable, - backend, - snapshot_path, - processors=processors, - ), + restore_command, context=context, environment=_restore_environment(), timeout=timeout, @@ -2672,6 +2723,9 @@ def run_smp_snapshot( failure_marker=WARP_PROBE_FAILURE_MARKER, ) _check_restore_warp(output, processors=processors, context=context) + _check_restore_status( + output, restore_command, processors=processors, context=context + ) if _snapshot_fingerprint(snapshot_path) != fingerprint: raise RuntimeError(f"{context} modified snapshot artifacts") @@ -2740,7 +2794,7 @@ def run_restore_processors( cmdline = f"quiet loglevel=0 maxcpus={boot_online}" # The warp probe replaces the guest TSC warp guard: it measures the skew # between every pair of CPUs, including the ones the restore activated. - script = _read_script("restore-processors.sh") + warp_probe_script() + script = _read_script("restore-processors.sh") + post_restore_checks() # The VP-binding lifecycle records identify the VPs that each restore # instantiates without changing restore behavior. environment = _restore_environment() @@ -2771,14 +2825,15 @@ def run_restore_processors( for target in targets: name = "untargeted" if target is None else str(target) online = boot_online if target is None else target + restore_command = snapshot_restore_command( + executable, + backend, + snapshot_path, + processors=capacity, + restore_processors=target, + ) output = _measure_restore( - snapshot_restore_command( - executable, - backend, - snapshot_path, - processors=capacity, - restore_processors=target, - ), + restore_command, context=_restore_label(target), environment=environment, timeout=timeout, @@ -2793,6 +2848,12 @@ def run_restore_processors( _check_restore_warp( output, processors=online, context=_restore_label(target) ) + _check_restore_status( + output, + restore_command, + processors=online, + context=_restore_label(target), + ) _check_restore_vp_bindings( output, backend, @@ -2850,14 +2911,15 @@ def run_restore_downtime( processors=processors, ) # The restored guest keeps running after the restore marker so the - # harness can wait for the daemon's restore marker. + # harness can stage its downtime check; the post-restore status + # query waits for the restore to finish first. capture_snapshot( [*boot_command, "--snapshot-destination", str(snapshot_path)], snapshot_path, timeout=timeout, processors=processors, teardown_mode="host-terminate", - post_restore_script=warp_probe_script(), + post_restore_script=post_restore_checks(), log_path=output_dir / f"restore-downtime-{name}-capture.log", ) snapshots.append((name, processors, snapshot_path)) @@ -2886,6 +2948,13 @@ def run_restore_downtime( error = process.time_abi.exit_error(result.returncode) raise RuntimeError(f"{context}: {error}") _check_restore_warp(result.output, processors=processors, context=context) + try: + process.time_abi.require_status(context) + process.time_abi.require_restore( + context, online_cpus=processors, generation=RESTORED_GENERATION + ) + except TimeAbiFailure as error: + raise RuntimeError(f"{context}: {error}") from error def run_restore_memory( diff --git a/scripts/nvx_tools/openvmm_process.py b/scripts/nvx_tools/openvmm_process.py index a9e8635a8..0502e2a0e 100644 --- a/scripts/nvx_tools/openvmm_process.py +++ b/scripts/nvx_tools/openvmm_process.py @@ -15,7 +15,15 @@ from .benchmark import InteractiveProcess, terminate from .common import remaining_timeout -from .time_abi import TimeAbiFailure, TimeAbiMonitor +from .time_abi import ( + GUEST_BOOT_MARKERS, + STATUS_TIMEOUT_SECONDS, + TimeAbiFailure, + TimeAbiMonitor, + status_script, +) + +_GUEST_BOOT_MARKERS = tuple(marker.encode() for marker in GUEST_BOOT_MARKERS) def _line_marker_end( @@ -59,7 +67,16 @@ def __init__( *, environment: Mapping[str, str] | None = None, output_read_delay: float = 0.0, + time_abi_status: bool = True, ) -> None: + """Start OpenVMM and scan its console output for the time ABI. + + With ``time_abi_status``, a cold-booted guest whose boot marker + appeared on this console answers a time ABI status query before its + first other input, which waits until the query exits so the guest's + tty cannot echo that input into the status lines. The query fails + unless the guest reported a valid boot line. + """ if output_read_delay < 0: raise ValueError("OpenVMM output read delay cannot be negative") process_environment = os.environ.copy() @@ -86,6 +103,8 @@ def __init__( self._log_path = log_path self._finished = False self._monitor = TimeAbiMonitor(command) + self._query_status = time_abi_status and self._monitor.cold_boot + self._status_sent = False @property def process(self): @@ -100,11 +119,40 @@ def time_abi(self) -> TimeAbiMonitor: return self._monitor def send_bytes(self, data: bytes) -> None: + self._query_status_first() self._interaction.write_input(data) def send_line(self, line: str) -> None: self.send_bytes(f"{line}\n".encode()) + def _query_status_first(self) -> None: + if ( + not self._query_status + or self._status_sent + # The guest's shell is not on this console, or has not booted. + or not any(marker in self._output for marker in _GUEST_BOOT_MARKERS) + ): + return + self._status_sent = True + self._interaction.write_input(status_script().encode()) + deadline = time.monotonic() + STATUS_TIMEOUT_SECONDS + while self._monitor.status_queries == 0: + remaining = deadline - time.monotonic() + if remaining <= 0: + self._fail( + TimeoutError( + "nvx-time status did not exit within " + f"{STATUS_TIMEOUT_SECONDS:g}s" + ) + ) + try: + chunk = self._chunks.get(timeout=min(remaining, 0.1)) + except queue.Empty: + continue + if chunk is None: + self._fail(self._exited("nvx-time status exited")) + self._consume(chunk) + def _consume(self, chunk: bytes) -> None: self._output.extend(chunk) try: diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 01d31c15b..ca94d7d82 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -75,11 +75,27 @@ def profile_id(self) -> str: CpuGeneration("icelake-sp", "GenuineIntel", 6, 106, range(16)), CpuGeneration("emeraldrapids", "GenuineIntel", 6, 207, range(16)), ) -# Guest boot markers that init prints after the time ABI boot check. +# Guest boot markers that init prints once the guest is shell-ready. Only the +# time ABI's initial clock step (C12) precedes them; the other boot checks +# finish asynchronously. GUEST_BOOT_MARKERS = ("ALPINE-MICROVM-BOOT-OK", "NVX-GUEST-BOOT-OK:") -# Init runs the boot check before any other guest work, so a cold boot whose -# guest reaches its shell without the boot marker is not conformant. -REQUIRE_BOOT_MARKER = True +# Guests keep the console quiet, because every console byte is a port exit: +# they print one line per recorded check phase (boot, capture, restore, oldest +# first) and a runtime line only when nvx-time status asks. Test runners ask +# once after a cold boot, before any other input, and after each restore; +# benchmarks never ask, so measured intervals stay quiet. +STATUS_COMMAND = "/sbin/nvx-time status" +# nvx-time status waits this long for pending checks, then reports +# status=pending and exits 1. +STATUS_WAIT_SECONDS = 30 +STATUS_EXIT_PREFIX = "NVX-TIME-STATUS-EXIT status=" +# The token ends its line even when the query's last output lacked a newline; +# the shell's echo of the query line ends in "$?" instead of a status. +_STATUS_EXIT = re.compile(rf"{re.escape(STATUS_EXIT_PREFIX)}(\d+)$") +# The exit status of an nvx-time that has no status subcommand. +STATUS_USAGE_EXIT = 2 +# Leaves the guest time to report a check that is still pending. +STATUS_TIMEOUT_SECONDS = STATUS_WAIT_SECONDS + 30.0 _MAX_PENDING_LINE = 64 * 1024 @@ -207,18 +223,20 @@ def is_cold_boot_command(command: Sequence[str]) -> bool: return "--kernel" in command and "--restore-snapshot" not in command -def validate_boot_marker( +def _validate_marker( fields: Mapping[str, str], *, + phase: str, backend: str | None, - online_cpus: int | None = None, + online_cpus: int | None, + generation: int | None = None, ) -> None: - """Check the boot marker's fields against the backend's declared rates.""" + """Check a boot or restore line's fields against the backend's rates.""" problems: list[str] = [] if fields.get("v") != ABI_VERSION: problems.append(f"version {fields.get('v')!r} is not {ABI_VERSION}") - if fields.get("phase") != "boot" or fields.get("status") != "ok": - problems.append("it is not a passing boot check") + if fields.get("phase") != phase or fields.get("status") != "ok": + problems.append(f"it is not a passing {phase} check") try: tsc_hz = int(fields.get("tsc_hz", "")) if not MIN_TSC_HZ <= tsc_hz <= MAX_TSC_HZ: @@ -229,37 +247,75 @@ def validate_boot_marker( expected_lapic = LAPIC_HZ.get(backend) if backend is not None else None if expected_lapic is not None and lapic != str(expected_lapic): problems.append(f"lapic_hz={lapic} is not the {backend} rate {expected_lapic}") - if fields.get("generation") != "0": - problems.append(f"generation={fields.get('generation')} is not 0 at cold boot") + actual = fields.get("generation", "") + if phase == "boot" and actual != "0": + problems.append(f"generation={actual} is not 0 at cold boot") + if phase == "restore" and not (actual.isdecimal() and int(actual) >= 1): + problems.append(f"generation={actual} is not 1 or later after a restore") + elif generation is not None and actual != str(generation): + problems.append(f"generation={actual} is not {generation}") if online_cpus is not None and fields.get("cpus") != str(online_cpus): problems.append(f"cpus={fields.get('cpus')} is not {online_cpus}") if problems: raise TimeAbiFailure( - "guest time ABI boot marker is invalid: " + "; ".join(problems) + f"guest time ABI {phase} marker is invalid: " + "; ".join(problems) ) +def validate_boot_marker( + fields: Mapping[str, str], + *, + backend: str | None, + online_cpus: int | None = None, +) -> None: + """Check the boot marker's fields against the backend's declared rates.""" + _validate_marker(fields, phase="boot", backend=backend, online_cpus=online_cpus) + + +def validate_restore_marker( + fields: Mapping[str, str], + *, + backend: str | None, + online_cpus: int | None = None, + generation: int | None = None, +) -> None: + """Check a restore marker's fields against the backend's declared rates. + + ``generation`` is the lineage generation the restore must carry: one more + than the generation of the process that was captured. + """ + _validate_marker( + fields, + phase="restore", + backend=backend, + online_cpus=online_cpus, + generation=generation, + ) + + class TimeAbiMonitor: """Scan OpenVMM console output for time ABI markers and violations. ``feed`` raises TimeAbiFailure as soon as a completed line carries a violation event or a failed conformance check, so a scenario fails fast - with the guest's own explanation instead of a marker timeout. On a cold - boot it also fails when the guest reaches its shell without a valid - boot marker. + with the guest's own explanation instead of a marker timeout. Guests print + their boot and restore lines only for ``nvx-time status``: when a query + ends, a cold boot must have reported a valid boot line, and a restored + guest a restore line. The query's runtime line is kept as ``runtime``. """ def __init__(self, command: Sequence[str] = ()) -> None: self.backend = _command_option(command, "--hypervisor") self.cold_boot = is_cold_boot_command(command) - self.require_boot_marker = REQUIRE_BOOT_MARKER and self.cold_boot + self.restored = "--restore-snapshot" in command self.boot: dict[str, str] | None = None self.restores: list[dict[str, str]] = [] - self.uncertain: list[dict[str, str]] = [] + self.runtime: dict[str, str] | None = None self.violation: str | None = None self.fatal: str | None = None self.report_only = False self.guest_booted = False + self.status_queries = 0 self._pending = bytearray() def feed(self, chunk: bytes | bytearray) -> None: @@ -306,8 +362,10 @@ def _line(self, line: str) -> None: f" {marker}" in line for marker in GUEST_BOOT_MARKERS ): self.guest_booted = True - if self.require_boot_marker: - self.require_boot("the guest boot marker") + status_exit = _STATUS_EXIT.search(line) + if status_exit is not None: + self._finish_status(int(status_exit.group(1))) + return if not line.startswith(MARKER_PREFIX): return try: @@ -316,19 +374,27 @@ def _line(self, line: str) -> None: raise TimeAbiFailure(f"malformed time ABI marker: {error}") from error assert fields is not None status = fields["status"] + phase = fields["phase"] + if phase == "runtime": + # The wall-clock discipline's state; it never gates. + self.runtime = fields + return if status == "fail": raise TimeAbiFailure( - f"guest time ABI {fields['phase']} check {fields.get('check', '?')} " + f"guest time ABI {phase} check {fields.get('check', '?')} " f"failed: {fields.get('detail', line)}" ) - if status == "uncertain": - self.uncertain.append(fields) - elif status != "ok": + if status == "pending": + raise TimeAbiFailure( + f"guest time ABI {phase} check was still pending when nvx-time " + f"status stopped waiting after {STATUS_WAIT_SECONDS} s" + ) + if status != "ok": raise TimeAbiFailure(f"unknown time ABI marker status: {line}") - elif fields["phase"] == "boot": + if phase == "boot": if self.boot is None: self.boot = fields - elif fields["phase"] == "restore": + elif phase == "restore": self.restores.append(fields) def check_exit(self, returncode: int | None) -> None: @@ -350,6 +416,36 @@ def exit_error( message += f": {self.fatal}" return RuntimeError(message) + def _finish_status(self, code: int) -> None: + """Check what one ``nvx-time status`` query printed before it exited.""" + self.status_queries += 1 + if code == STATUS_USAGE_EXIT: + raise TimeAbiFailure( + f"nvx-time status exited {code}: the guest image has no status " + "subcommand and predates the quiet console" + ) + # A report-only guest exits 1 when a check failed; the requirement + # below names report-only mode instead. + if code != 0 and not self.report_only: + raise TimeAbiFailure(f"nvx-time status exited {code}") + if self.cold_boot: + self.require_boot("nvx-time status exited") + elif self.restored: + self.require_restore("nvx-time status exited") + + def require_status(self, context: str) -> None: + """Fail unless an ``nvx-time status`` query finished before ``context``.""" + if self.status_queries == 0: + raise TimeAbiFailure( + f"nvx-time status did not finish before {context}; the guest never " + "reported its time ABI state" + ) + + def _missing_reason(self) -> str: + if self.report_only: + return "the guest ran nvx-time in report-only mode" + return "the guest image or OpenVMM does not implement time ABI v1" + def require_boot( self, context: str, @@ -358,14 +454,9 @@ def require_boot( ) -> dict[str, str]: """Return the validated boot marker of a cold boot, or fail clearly.""" if self.boot is None: - reason = ( - "the guest ran nvx-time in report-only mode" - if self.report_only - else "the guest image or OpenVMM does not implement time ABI v1" - ) raise TimeAbiFailure( - f"guest did not print the NVX-TIME-ABI boot marker before {context}; " - f"{reason}" + f"guest did not report the NVX-TIME-ABI boot marker before {context}; " + f"{self._missing_reason()}" ) validate_boot_marker( self.boot, @@ -374,6 +465,43 @@ def require_boot( ) return self.boot + def require_restore( + self, + context: str, + *, + online_cpus: int | None = None, + generation: int | None = None, + ) -> dict[str, str]: + """Return the validated latest restore marker, or fail clearly. + + After a restore, nvx-time status prints every recorded phase, oldest + first, and each line keeps the values from when its check ran: only + the restore line describes the CPUs that are online now. + """ + if not self.restores: + raise TimeAbiFailure( + f"guest did not report an NVX-TIME-ABI restore marker before " + f"{context}; {self._missing_reason()}" + ) + validate_restore_marker( + self.restores[-1], + backend=self.backend, + online_cpus=online_cpus, + generation=generation, + ) + return self.restores[-1] + + +def status_script() -> str: + """Return a guest command line that prints the time ABI status lines. + + nvx-time status first waits for pending checks: the asynchronous boot + checks, or a restore's deferred work. The line then echoes + STATUS_EXIT_PREFIX with the exit status, where the monitor checks what + the query printed. + """ + return f'{STATUS_COMMAND}; echo "{STATUS_EXIT_PREFIX}$?"\n' + def warp_probe_command(*, cpus: str | None = None) -> str: """Return the guest command line that runs the warp probe.""" diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 096ee0397..166d8821e 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -99,6 +99,28 @@ def _warp_probe_output(cpus: int, *, offset_ns: int = 40) -> bytes: return (probe_round * time_abi.warp_rounds(cpus) + "NVX-WARP-PROBE-OK\r\n").encode() +def _restore_status_output( + backend: str, cpus: int, *, boot_cpus: int | None = None +) -> bytes: + """Return what the post-restore ``nvx-time status`` query prints: every + recorded phase, oldest first, with the values from when its check ran, + then the runtime line.""" + boot_cpus = cpus if boot_cpus is None else boot_cpus + rates = f"tsc_hz=2793437000 lapic_hz={time_abi.LAPIC_HZ[backend]}" + return ( + f"NVX-TIME-ABI: v=1 phase=boot status=ok cpus={boot_cpus} {rates} " + "generation=0 elapsed_us=2390\r\n" + f"NVX-TIME-ABI: v=1 phase=capture status=ok cpus={boot_cpus} {rates} " + "generation=0 elapsed_us=95\r\n" + f"NVX-TIME-ABI: v=1 phase=restore status=ok cpus={cpus} {rates} " + "generation=1 elapsed_us=310\r\n" + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=1 " + "discontinuities=1 offset_ns=-1200 uncertainty_ns=900 rejected_samples=0 " + "last_sample_error=none\r\n" + "NVX-TIME-STATUS-EXIT status=0\r\n" + ).encode() + + def _restore_processors_measure( backend: str, *, @@ -116,6 +138,8 @@ def measure(command: list[str], **kwargs: object) -> None: log_path.write_bytes( f"NVX-RESTORE-PROCESSORS-OK count={online}\r\n".encode() + _warp_probe_output(online, offset_ns=warp_offset_ns) + # The snapshot's guest booted with one online CPU. + + _restore_status_output(backend, online, boot_cpus=1) + benchmark.RESTORE_MARKER + b"\r\n" + _vp_binding_profile(list(range(vp_count))) @@ -2825,7 +2849,10 @@ def measure(command: list[str], **kwargs: object) -> None: processors = int(command[command.index("--processors") + 1]) log_path = cast(Path, kwargs["log_path"]) log_path.parent.mkdir(parents=True, exist_ok=True) - log_path.write_bytes(_warp_probe_output(processors)) + log_path.write_bytes( + _warp_probe_output(processors) + + _restore_status_output("whp", processors) + ) with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) / "logs" @@ -2873,7 +2900,7 @@ def measure(command: list[str], **kwargs: object) -> None: self.assertTrue( all( call.kwargs["post_restore_script"] - == "probe\n" + time_abi.warp_probe_script() + == "probe\n" + time_abi.warp_probe_script() + time_abi.status_script() for call in capture_snapshot.call_args_list ) ) @@ -2912,6 +2939,25 @@ def measure(_command: list[str], **kwargs: object) -> None: cases = ( (b"", "did not finish its warp probe"), (_warp_probe_output(4, offset_ns=5000), "max_abs_offset_ns=5000"), + ( + _warp_probe_output(4), + "nvx-time status did not finish before the restore checks", + ), + ( + _warp_probe_output(4) + _restore_status_output("kvm", 2), + "restore marker is invalid: cpus=2 is not 4", + ), + ( + _warp_probe_output(4) + + _restore_status_output("kvm", 4).replace( + b"generation=1", b"generation=2" + ), + "restore marker is invalid: generation=2 is not 1", + ), + ( + _warp_probe_output(4) + b"NVX-TIME-STATUS-EXIT status=0\r\n", + "did not report an NVX-TIME-ABI restore marker", + ), ) for output, message in cases: with self.subTest(message=message): @@ -3004,7 +3050,8 @@ def test_restore_processors_uses_capacity_eight_and_each_target(self): self.assertEqual( script, microvm_tests._read_script("restore-processors.sh") - + time_abi.warp_probe_script(), + + time_abi.warp_probe_script() + + time_abi.status_script(), ) self.assertEqual( logs, @@ -3053,11 +3100,14 @@ def test_restore_processors_requires_full_capacity_off_mshv(self): with self.subTest(backend=backend): with tempfile.TemporaryDirectory() as temporary: - def measure(command: list[str], **kwargs: object) -> None: + def measure( + command: list[str], backend: str = backend, **kwargs: object + ) -> None: log_path = cast(Path, kwargs["log_path"]) log_path.write_bytes( b"NVX-RESTORE-PROCESSORS-OK count=2\r\n" + _warp_probe_output(2) + + _restore_status_output(backend, 2, boot_cpus=1) + _vp_binding_profile([0, 1]) ) @@ -3348,8 +3398,17 @@ def sleep(seconds: float) -> None: for call in capture.call_args_list: self.assertEqual(call.kwargs["teardown_mode"], "host-terminate") self.assertEqual( - call.kwargs["post_restore_script"], time_abi.warp_probe_script() + call.kwargs["post_restore_script"], + time_abi.warp_probe_script() + time_abi.status_script(), ) + self.assertEqual( + [ + (call.kwargs["online_cpus"], call.kwargs["generation"]) + for call in active.time_abi.require_restore.call_args_list + ], + [(1, 1), (1, 1), (8, 1), (8, 1)], + ) + self.assertEqual(active.time_abi.require_status.call_count, 4) # Every capture precedes one shared 30 s window. self.assertEqual(sleeps, [30.0, 0.0, 0.0, 0.0]) self.assertEqual( @@ -3380,15 +3439,19 @@ def sleep(seconds: float) -> None: def test_restore_downtime_rejects_a_failed_check_or_warp_probe(self): fatal = "[E_TSC_SYNC_UNSUPPORTED] failed to launch vm worker" + context = "1-vcpu restore after a 0 s downtime" + passed = openvmm_process.OpenvmmProcessResult(0, _warp_probe_output(1)) cases = ( ( openvmm_process.OpenvmmProcessResult(99, b""), None, + b"", "downtime: OpenVMM exited with status 99$", ), ( openvmm_process.OpenvmmProcessResult(1, b""), fatal, + b"", f"downtime: OpenVMM exited with status 1: {re.escape(fatal)}$", ), ( @@ -3396,10 +3459,18 @@ def test_restore_downtime_rejects_a_failed_check_or_warp_probe(self): 0, _warp_probe_output(1).replace(b"conclusive=1", b"conclusive=0") ), None, - "1-vcpu restore after a 0 s downtime: .*inconclusive", + b"", + f"{context}: .*inconclusive", + ), + (passed, None, b"", f"{context}: nvx-time status did not finish"), + ( + passed, + None, + _restore_status_output("mshv", 2), + f"{context}: guest time ABI restore marker is invalid: cpus=2 is not 1", ), ) - for result, fatal_error, message in cases: + for result, fatal_error, status, message in cases: with self.subTest(message=message): with tempfile.TemporaryDirectory() as temporary: with ( @@ -3411,6 +3482,7 @@ def test_restore_downtime_rejects_a_failed_check_or_warp_probe(self): active.wait.return_value = result active.time_abi = time_abi.TimeAbiMonitor() active.time_abi.fatal = fatal_error + active.time_abi.feed(status) with self.assertRaisesRegex(RuntimeError, message): microvm_tests.run_restore_downtime( Path("openvmm"), @@ -3620,32 +3692,47 @@ def test_conformance_guest_line_reports_the_check_exit_status(self): ) def test_time_abi_evidence_sums_up_the_guest_logs(self): - boot = ( - "NVX-TIME-ABI: v=1 phase=boot status=ok cpus=8 tsc_hz=2793437000 " - "lapic_hz=1000000000 generation=0 elapsed_us={elapsed}\r\n" - ) - restore = boot.replace("phase=boot", "phase=restore").replace( - "generation=0", "generation=1" + runtime = ( + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation={g} " + "discontinuities={g} offset_ns=0 uncertainty_ns=900 rejected_samples=0 " + "last_sample_error=none\r\n" ) + + def line(phase: str, generation: int, elapsed: int) -> str: + return ( + f"NVX-TIME-ABI: v=1 phase={phase} status=ok cpus=8 tsc_hz=2793437000 " + f"lapic_hz=1000000000 generation={generation} elapsed_us={elapsed}\r\n" + ) + + def restored(elapsed: int) -> bytes: + # A restored guest repeats its source's boot and capture lines. + return ( + line("boot", 0, 10894) + + line("capture", 0, 95) + + line("restore", 1, elapsed) + + runtime.format(g=1) + ).encode() + with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) (output_dir / "smp-8.log").write_bytes( - boot.format(elapsed=10894).encode() + (line("boot", 0, 10894) + runtime.format(g=0)).encode() + _warp_probe_output(8, offset_ns=41) ) (output_dir / "restore-downtime-8-vcpu.log").write_bytes( - restore.format(elapsed=1401).encode() - + _warp_probe_output(8, offset_ns=3) + _warp_probe_output(8, offset_ns=3) + + restored(1401) + b"NVX-RESTORE-DOWNTIME stalls=0 uptime_s=37\r\n" - + restore.format(elapsed=6410).encode() ) + (output_dir / "smp-snapshot-8-restore-0.log").write_bytes(restored(6410)) (output_dir / "time-abi-conformance.log").write_bytes( - b'/ # /sbin/nvx-time exhaustive; echo "NVX-EXHAUSTIVE-EXIT status=$?"\r\n' + (line("boot", 0, 1384) + runtime.format(g=0)).encode() + + b'/ # /sbin/nvx-time exhaustive; echo "NVX-EXHAUSTIVE-EXIT status=$?"\r\n' b'NVX-TIME-ABI-EXHAUSTIVE: v=1 check=X1 cpu=0 status=pass detail=""\r\n' b"NVX-TIME-ABI-EXHAUSTIVE: v=1 status=ok cpus=8 failures=0\r\n" - + boot.format(elapsed=1384).encode() ) - (output_dir / "lifecycle.log").write_text("no time ABI lines\n") + # A phase line outside a status query is not counted. + (output_dir / "lifecycle.log").write_text(line("boot", 0, 7)) evidence = microvm_tests.time_abi_evidence(output_dir) self.assertEqual( evidence, diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index a64f568e6..a33040517 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -24,6 +24,17 @@ "NVX-TIME-ABI: v=1 phase=boot status=ok cpus=4 tsc_hz=2194804000 " "lapic_hz=1000000000 generation=0 elapsed_us=1873\n" ) +RESTORE_LINE = ( + "NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 tsc_hz=2194804000 " + "lapic_hz=1000000000 generation=2 elapsed_us=41\n" +) +# nvx-time status prints the last check's marker, then this line. +RUNTIME_LINE = ( + "NVX-TIME-ABI: v=1 phase=runtime status=unsynchronized generation=0 " + "discontinuities=0 offset_ns=0 uncertainty_ns=0 rejected_samples=0 " + "last_sample_error=none\n" +) +STATUS_OK = "NVX-TIME-STATUS-EXIT status=0\n" KVM_BOOT = ["openvmm", "--hypervisor", "kvm", "--kernel", "vmlinux"] MSHV_RESTORE = ["openvmm", "--hypervisor", "mshv", "--restore-snapshot", "snap"] # OpenVMM's openvmm_entry fatal_error_message, after a guest prompt. @@ -138,10 +149,9 @@ def test_records_markers_across_chunk_boundaries(self): monitor = TimeAbiMonitor(KVM_BOOT) data = ( "[ 0.1] early\n" - + BOOT_LINE + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" - + "NVX-TIME-ABI: v=1 phase=runtime status=uncertain " - + "code=G_SAMPLE_UNCERTAIN epsilon_ns=-1\n" + + BOOT_LINE + + RUNTIME_LINE + "NVX-TIME-ABI: v=1 phase=restore status=ok cpus=4 tsc_hz=2194804000 " + "lapic_hz=1000000000 generation=1 elapsed_us=12\n" ).encode() @@ -149,8 +159,25 @@ def test_records_markers_across_chunk_boundaries(self): monitor.feed(data[index : index + 7]) self.assertTrue(monitor.guest_booted) self.assertEqual(monitor.require_boot("test", online_cpus=4)["cpus"], "4") - self.assertEqual(len(monitor.uncertain), 1) + assert monitor.runtime is not None + self.assertEqual(monitor.runtime["status"], "unsynchronized") self.assertEqual(monitor.restores[0]["generation"], "1") + # The runtime line is the discipline's state and never fails a run. + TimeAbiMonitor(KVM_BOOT).feed( + RUNTIME_LINE.replace("unsynchronized", "uncertain").encode() + ) + + def test_reports_a_check_that_stayed_pending(self): + pending = ( + "NVX-TIME-ABI: v=1 phase=restore status=pending cpus=4 " + "tsc_hz=2194804000 lapic_hz=200000000 generation=1\n" + ) + with self.assertRaisesRegex( + TimeAbiFailure, + "restore check was still pending when nvx-time status stopped " + "waiting after 30 s", + ): + TimeAbiMonitor(MSHV_RESTORE).feed(pending.encode() + STATUS_OK.encode()) def test_fails_fast_on_violation_events_and_failed_checks(self): monitor = TimeAbiMonitor(KVM_BOOT) @@ -171,36 +198,130 @@ def test_fails_fast_on_violation_events_and_failed_checks(self): b"NVX-TIME-ABI: v=1 phase=boot status=maybe\n" ) - def test_rejects_a_report_only_boot_when_the_guest_reaches_its_shell(self): + def test_rejects_a_report_only_boot_when_a_status_query_exits(self): monitor = TimeAbiMonitor(KVM_BOOT) monitor.feed( b"NVX-TIME-REPORT-VIOLATION: v=1 code=G_TSC_WARP source=watcher " b'phase=boot generation=0 boottime_ns=1 detail="x"\n' - b"NVX-TIME-REPORT: v=1 phase=boot status=fail cpus=1 failures=3\n" + b"NVX-GUEST-BOOT-OK: alpine\n" ) self.assertTrue(monitor.report_only) + # A report-only guest whose check failed exits 1; the failure names + # report-only mode, not the exit status. + report = ( + b"NVX-TIME-REPORT: v=1 phase=boot status=fail cpus=1 tsc_hz=2194804000 " + b"lapic_hz=1000000000 generation=0 elapsed_us=9 failures=3\n" + ) with self.assertRaisesRegex(TimeAbiFailure, "report-only"): - monitor.feed(b"NVX-GUEST-BOOT-OK: alpine\n") + monitor.feed(report + b"NVX-TIME-STATUS-EXIT status=1\n") + restored = TimeAbiMonitor(MSHV_RESTORE) + with self.assertRaisesRegex( + TimeAbiFailure, + "restore marker before nvx-time status exited; .*report-only", + ): + restored.feed( + report.replace(b"phase=boot", b"phase=restore") + + b"NVX-TIME-STATUS-EXIT status=0\n" + ) - def test_requires_a_valid_boot_marker_only_for_cold_boots(self): + def test_status_queries_require_the_line_of_the_launch_kind(self): + # A quiet guest reaches its shell without printing any time ABI line. + quiet = TimeAbiMonitor(KVM_BOOT) + quiet.feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\nNVX-GUEST-BOOT-OK: alpine\n") + self.assertTrue(quiet.guest_booted) with self.assertRaisesRegex( TimeAbiFailure, - "did not print the NVX-TIME-ABI boot marker before the guest boot " - "marker; the guest image or OpenVMM does not implement time ABI v1", + "nvx-time status did not finish before the scenario; the guest never " + "reported its time ABI state", ): - TimeAbiMonitor(KVM_BOOT).feed(b"NVX-GUEST-BOOT-OK: alpine\n") + quiet.require_status("the scenario") + # The shell's echo of the query is not its exit line. + quiet.feed( + b'/ # /sbin/nvx-time status; echo "NVX-TIME-STATUS-EXIT status=$?"\n' + ) + self.assertEqual(quiet.status_queries, 0) + with self.assertRaisesRegex( + TimeAbiFailure, + "did not report the NVX-TIME-ABI boot marker before nvx-time status " + "exited; the guest image or OpenVMM does not implement time ABI v1", + ): + quiet.feed(b"NVX-TIME-STATUS-EXIT status=0\r\n") + + booted = TimeAbiMonitor(KVM_BOOT) + booted.feed((BOOT_LINE + RUNTIME_LINE + STATUS_OK).encode()) + self.assertEqual(booted.status_queries, 1) + booted.require_status("the scenario") + self.assertEqual(booted.require_boot("x", online_cpus=4)["cpus"], "4") mismatched = BOOT_LINE.replace("lapic_hz=1000000000", "lapic_hz=200000000") monitor = TimeAbiMonitor(KVM_BOOT) monitor.feed(mismatched.encode()) with self.assertRaisesRegex(TimeAbiFailure, "is not the kvm rate"): - monitor.feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") - # Restores and launches without a kernel never print a boot marker. - TimeAbiMonitor(MSHV_RESTORE).feed(b" ALPINE-MICROVM-BOOT-OK: 3.22.1\n") - TimeAbiMonitor(["openvmm"]).feed(b"NVX-GUEST-BOOT-OK: alpine\n") - with patch.object(time_abi, "REQUIRE_BOOT_MARKER", False): - legacy = TimeAbiMonitor(KVM_BOOT) - legacy.feed(b"NVX-GUEST-BOOT-OK: alpine\n") - self.assertTrue(legacy.guest_booted) + monitor.feed(b"NVX-TIME-STATUS-EXIT status=0\n") + + with self.assertRaisesRegex( + TimeAbiFailure, + "did not report an NVX-TIME-ABI restore marker before nvx-time status " + "exited", + ): + TimeAbiMonitor(MSHV_RESTORE).feed(b"NVX-TIME-STATUS-EXIT status=0\n") + restored = TimeAbiMonitor(MSHV_RESTORE) + restored.feed( + RESTORE_LINE.replace("lapic_hz=1000000000", "lapic_hz=200000000").encode() + + RUNTIME_LINE.replace("generation=0", "generation=2").encode() + + b"NVX-TIME-STATUS-EXIT status=0\n" + ) + self.assertEqual( + restored.require_restore("x", online_cpus=4)["generation"], "2" + ) + # After a restore, status prints every recorded phase, oldest first, + # each with the values from when its check ran: the boot and capture + # lines keep their boot-time CPU count, and only the newest line counts. + activated = TimeAbiMonitor(MSHV_RESTORE) + mshv = "lapic_hz=200000000" + activated.feed( + ( + BOOT_LINE.replace("cpus=4", "cpus=1").replace( + "lapic_hz=1000000000", mshv + ) + + BOOT_LINE.replace("phase=boot", "phase=capture") + .replace("cpus=4", "cpus=1") + .replace("lapic_hz=1000000000", mshv) + + RESTORE_LINE.replace("generation=2", "generation=1").replace( + "lapic_hz=1000000000", mshv + ) + + RUNTIME_LINE.replace("generation=0", "generation=1") + + STATUS_OK + ).encode() + ) + self.assertEqual( + activated.require_restore("x", online_cpus=4, generation=1)["cpus"], "4" + ) + assert activated.boot is not None + self.assertEqual(activated.boot["cpus"], "1") + # A launch that neither boots a kernel nor restores has no line to require. + TimeAbiMonitor(["openvmm"]).feed(b"NVX-TIME-STATUS-EXIT status=0\n") + + with self.assertRaisesRegex( + TimeAbiFailure, "exited 2: the guest image has no status subcommand" + ): + # An old nvx-time's usage text can end without a newline. + TimeAbiMonitor(KVM_BOOT).feed( + b"usage: nvx-time boot|exhaustiveNVX-TIME-STATUS-EXIT status=2\n" + ) + with self.assertRaisesRegex(TimeAbiFailure, "nvx-time status exited 1$"): + TimeAbiMonitor(KVM_BOOT).feed( + BOOT_LINE.encode() + b"NVX-TIME-STATUS-EXIT status=1\n" + ) + + def test_builds_the_status_query(self): + self.assertEqual( + time_abi.status_script(), + '/sbin/nvx-time status; echo "NVX-TIME-STATUS-EXIT status=$?"\n', + ) + # The harness waits longer than the guest, which reports pending checks. + self.assertGreater( + time_abi.STATUS_TIMEOUT_SECONDS, time_abi.STATUS_WAIT_SECONDS + ) def test_validates_boot_marker_fields_against_the_backend(self): cases = ( @@ -223,6 +344,32 @@ def test_validates_boot_marker_fields_against_the_backend(self): assert mshv is not None time_abi.validate_boot_marker(mshv, backend="mshv") + def test_validates_restore_marker_fields_against_the_backend(self): + restore = time_abi.parse_marker(RESTORE_LINE) + assert restore is not None + time_abi.validate_restore_marker(restore, backend="kvm", online_cpus=4) + cases = ( + ("generation=2", "generation=0", "generation=0 is not 1 or later"), + ("generation=2", "generation=x", "generation=x is not 1 or later"), + ("phase=restore", "phase=boot", "not a passing restore check"), + ("lapic_hz=1000000000", "lapic_hz=200000000", "kvm rate"), + ("cpus=4", "cpus=8", "cpus=8 is not 4"), + ) + for valid, invalid, message in cases: + fields = time_abi.parse_marker(RESTORE_LINE.replace(valid, invalid)) + assert fields is not None + with self.subTest(invalid=invalid): + with self.assertRaisesRegex( + TimeAbiFailure, f"restore marker is invalid: .*{message}" + ): + time_abi.validate_restore_marker( + fields, backend="kvm", online_cpus=4 + ) + # A restore of a snapshot captured at generation 0 carries generation 1. + with self.assertRaisesRegex(TimeAbiFailure, "generation=2 is not 1$"): + time_abi.validate_restore_marker(restore, backend="kvm", generation=1) + time_abi.validate_restore_marker(restore, backend="kvm", generation=2) + def test_classifies_time_abi_exit_statuses_with_the_violation_event(self): monitor = TimeAbiMonitor(MSHV_RESTORE) monitor.check_exit(0) @@ -487,12 +634,53 @@ def close(self) -> None: pass +class ScriptedInteraction(FakeInteraction): + """A guest that answers each console write with the next scripted reply + and exits after the last one.""" + + def __init__( + self, chunks: list[bytes], status: int, replies: list[list[bytes]] + ) -> None: + super().__init__(chunks, status) + self.replies = list(replies) + self._sink: queue.Queue[bytes | None] | None = None + + def read_output(self, chunks: queue.Queue[bytes | None]) -> None: + self._sink = chunks + for chunk in self.chunks: + chunks.put(chunk) + if not self.replies: + self._exit() + + def write_input(self, data: bytes) -> None: + self.writes.append(data) + assert self._sink is not None + for chunk in self.replies.pop(0): + self._sink.put(chunk) + if not self.replies: + self._exit() + + def _exit(self) -> None: + assert self._sink is not None + self.process.exited = True + self._sink.put(None) + + class RunnerWiringTests(unittest.TestCase): + @staticmethod + def _fake_exit(process: FakeProcess, _timeout: float) -> int: + """Wait for a fake OpenVMM, whose PID is not a real process.""" + return process.wait() + def _measure(self, chunks: list[bytes], status: int, command: list[str]): interaction = FakeInteraction(chunks, status) with ( patch.object(benchmark, "InteractiveProcess", return_value=interaction), patch.object(benchmark, "live_peak_rss_bytes", return_value=1024), + # Never pidfd_open the fake's PID. + patch.object( + benchmark, "wait_for_process_exit", side_effect=self._fake_exit + ), ): return benchmark.measure_once( command, @@ -589,19 +777,145 @@ def test_run_guest_script_classifies_a_conformance_power_off(self): KVM_BOOT, "true\n", b"DONE", timeout=5, boot_marker=b"BOOT" ) - def test_cold_boot_runners_require_the_boot_marker(self): + def test_cold_boot_runners_query_the_status_before_other_input(self): + status = time_abi.status_script().encode() boot = benchmark.BOOT_MARKER + b": 3.22.1\n" - interaction = FakeInteraction([boot, b"DONE\n"], 0) - with patch.object(benchmark, "InteractiveProcess", return_value=interaction): + # A quiet guest that answers the query without a boot line fails, and + # never receives the script. + quiet = ScriptedInteraction([boot], 0, [[STATUS_OK.encode()], [b"DONE\n"]]) + with patch.object(benchmark, "InteractiveProcess", return_value=quiet): with self.assertRaisesRegex(RuntimeError, "NVX-TIME-ABI boot marker"): - benchmark.run_guest_script(KVM_BOOT, "true\n", b"DONE", timeout=5) - self.assertEqual(interaction.writes, []) - conformant = FakeInteraction([BOOT_LINE.encode(), boot, b"DONE\n"], 0) + benchmark.run_guest_script( + KVM_BOOT, "true\n", b"DONE", timeout=5, time_abi_status=True + ) + self.assertEqual(quiet.writes, [status]) + answered = [BOOT_LINE.encode(), STATUS_OK.encode()] + conformant = ScriptedInteraction([boot], 0, [answered, [b"DONE\n"]]) with patch.object(benchmark, "InteractiveProcess", return_value=conformant): - result = benchmark.run_guest_script(KVM_BOOT, "true\n", b"DONE", timeout=5) + result = benchmark.run_guest_script( + KVM_BOOT, "true\n", b"DONE", timeout=5, time_abi_status=True + ) + self.assertEqual(conformant.writes, [status, b"true\n"]) self.assertIn("phase=boot status=ok", result["text"]) - with self.assertRaisesRegex(RuntimeError, "NVX-TIME-ABI boot marker"): - self._measure([boot, benchmark.RESTORE_MARKER + b"\n"], 0, KVM_BOOT) + # A guest that exits during the query names the query. + exiting = ScriptedInteraction([boot], 0, [[]]) + with patch.object(benchmark, "InteractiveProcess", return_value=exiting): + with self.assertRaisesRegex( + RuntimeError, "nvx-time status did not finish before the guest exited" + ): + benchmark.run_guest_script( + KVM_BOOT, "true\n", b"DONE", timeout=5, time_abi_status=True + ) + # Benchmarks and restores never query: the console stays quiet. + for command in (KVM_BOOT, MSHV_RESTORE): + silent = ScriptedInteraction([boot], 0, [[b"DONE\n"]]) + with patch.object(benchmark, "InteractiveProcess", return_value=silent): + benchmark.run_guest_script( + command, + "true\n", + b"DONE", + timeout=5, + time_abi_status=command is MSHV_RESTORE, + ) + self.assertEqual(silent.writes, [b"true\n"]) + self._measure([boot, benchmark.RESTORE_MARKER + b"\n"], 0, KVM_BOOT) + + def test_capture_snapshot_queries_the_status_before_the_capture_script(self): + boot = benchmark.BOOT_MARKER + b": 3.22.1\n" + answered = [BOOT_LINE.encode(), STATUS_OK.encode()] + for queried in (False, True): + replies = [[b"capture script exited\n"]] + if queried: + replies.insert(0, answered) + interaction = ScriptedInteraction([boot], 0, replies) + with tempfile.TemporaryDirectory() as temporary: + with patch.object( + benchmark, "InteractiveProcess", return_value=interaction + ): + with self.assertRaisesRegex( + RuntimeError, "exited before its snapshot request" + ): + benchmark.capture_snapshot( + KVM_BOOT, + Path(temporary) / "snapshot", + timeout=5, + processors=1, + time_abi_status=queried, + ) + with self.subTest(queried=queried): + capture = benchmark.prepare_snapshot_capture_script( + 1, teardown_mode="guest-exit" + ).encode() + status = time_abi.status_script().encode() + expected = [status, capture] if queried else [capture] + self.assertEqual(interaction.writes, expected) + quiet = ScriptedInteraction([boot], 0, [[STATUS_OK.encode()]]) + with tempfile.TemporaryDirectory() as temporary: + with patch.object(benchmark, "InteractiveProcess", return_value=quiet): + with self.assertRaisesRegex(RuntimeError, "NVX-TIME-ABI boot marker"): + benchmark.capture_snapshot( + KVM_BOOT, + Path(temporary) / "snapshot", + timeout=5, + processors=1, + time_abi_status=True, + ) + self.assertEqual(len(quiet.writes), 1) + + def test_openvmm_process_queries_the_status_before_the_first_input(self): + status = time_abi.status_script().encode() + answered = [BOOT_LINE.encode(), STATUS_OK.encode()] + with tempfile.TemporaryDirectory() as temporary: + log_path = Path(temporary) / "process.log" + # The scenario's substring wait can return before the boot line + # ends; its input still follows the finished query. + guest = ScriptedInteraction( + [b"NVX-GUEST-BOOT-OK:"], 0, [[b" alpine\n", *answered], [b"OK\n"]] + ) + with patch.object( + openvmm_process, "InteractiveProcess", return_value=guest + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + process.wait_for(b"NVX-GUEST-BOOT-OK:", 1) + process.send_line("echo OK") + self.assertEqual(process.time_abi.status_queries, 1) + process.wait_for_line(b"OK", 1) + self.assertEqual(process.wait(1).returncode, 0) + self.assertEqual(guest.writes, [status, b"echo OK\n"]) + + quiet = ScriptedInteraction( + [b"NVX-GUEST-BOOT-OK: alpine\n"], 0, [[STATUS_OK.encode()], []] + ) + with patch.object( + openvmm_process, "InteractiveProcess", return_value=quiet + ): + with openvmm_process.OpenvmmProcess(KVM_BOOT, log_path) as process: + process.wait_for(b"NVX-GUEST-BOOT-OK:", 1) + with self.assertRaisesRegex( + RuntimeError, "did not report the NVX-TIME-ABI boot marker" + ): + process.send_line("echo OK") + self.assertEqual(quiet.writes, [status]) + + # Restores, guests whose shell is on another console, and opted-out + # processes receive only the scenario's input. + for command, output, enabled in ( + (MSHV_RESTORE, b"NVX-GUEST-BOOT-OK: alpine\n", True), + (KVM_BOOT, b"booting\n", True), + (KVM_BOOT, b"NVX-GUEST-BOOT-OK: alpine\n", False), + ): + other = ScriptedInteraction([output], 0, [[]]) + with patch.object( + openvmm_process, "InteractiveProcess", return_value=other + ): + with openvmm_process.OpenvmmProcess( + command, log_path, time_abi_status=enabled + ) as process: + process.wait_for(output.rstrip(b"\n"), 1) + process.send_bytes(b"\x01") + self.assertEqual(process.wait(1).returncode, 0) + with self.subTest(command=command, output=output, enabled=enabled): + self.assertEqual(other.writes, [b"\x01"]) def test_capture_snapshot_classifies_a_failed_capture_check(self): interaction = FakeInteraction([b"booting\n"], 193) @@ -1214,7 +1528,7 @@ def test_guest_warp_check_runs_the_idle_schedules_and_applies_the_bound(self): def boot(cpus: int) -> str: line = BOOT_LINE.replace("cpus=4", f"cpus={cpus}") - return line + " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + return " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + line + STATUS_OK smp = boot(8) + warp_output(pairs=28) * 4 + warp_output(pairs=28, offset=60) single = boot(1) + warp_output(pairs=0, offset=0) @@ -1240,6 +1554,7 @@ def boot(cpus: int) -> str: self.assertEqual(command[command.index("--processors") + 1], processors) self.assertEqual(script, time_abi.warp_probe_script(gaps) + "nvx-exit 0\n") self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + self.assertIs(call.kwargs["time_abi_status"], True) # CI's schedule runs the probe twice with a 1 s gap. context.schedule = doctor.CI_SCHEDULE ci = boot(8) + warp_output(pairs=28) * 2 @@ -1270,7 +1585,16 @@ def boot(cpus: int) -> str: "vcpus=8", "in round 5: max_abs_offset_ns=1500", ), - ([warp_output(pairs=28) * 5], "vcpus=8", "NVX-TIME-ABI boot marker"), + ( + [STATUS_OK + warp_output(pairs=28) * 5], + "vcpus=8", + "NVX-TIME-ABI boot marker", + ), + ( + [BOOT_LINE.replace("cpus=4", "cpus=8") + warp_output(pairs=28) * 5], + "vcpus=8", + "nvx-time status did not finish before the warp probe", + ), ( [smp, boot(1) + warp_output(pairs=0, backward=2000, verdict="FAIL")], "vcpus=1", From e93fdc09089696dcddc4f764606b1f6b90c7f723 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 20:54:32 -0700 Subject: [PATCH 119/188] ci: drop the pre-flip --x-time-abi-v1 hints, which the flip removes The OpenVMM flip (nanvix/openvmm nvx/time-abi-v1-flip, "make the NVX time ABI the only microVM time path") removes the --x-time-abi-v1 development switch: every microVM runs the time ABI, and passing the switch fails with "unexpected argument". It keeps --x-time-abi-verify, which needs the microVM machine profile that doctor's H3 command already selects with --machine microvm. - doc/ci.md and doctor.py no longer tell users to pass --openvmm-arg=--x-time-abi-v1 before the flip. - H3 reports an OpenVMM that predates the verification mode only when OpenVMM rejects --x-time-abi-verify itself. Before, any unexpected argument whose name contained "x-time-abi" got that explanation, so a stale --openvmm-arg=--x-time-abi-v1 on a flipped OpenVMM would have been misreported. OpenVMM's own message is now reported unchanged. - The pass-through test uses a test hook instead of the removed switch. Lands with the OpenVMM flip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 2 +- scripts/nvx_tools/doctor.py | 3 +-- scripts/test_time_abi.py | 21 +++++++++++++++------ 3 files changed, 17 insertions(+), 9 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 5ba70e0c4..fdc47f464 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -264,7 +264,7 @@ ABI failure code starts its detail with the code in brackets, for example | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | | H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. Then `openvmm --hypervisor --cpu-fingerprint ` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line. H2 requires exit status 0, `status=pass`, the same backend and generation, and a revision of the generation's profile, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | -| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]`. Before the flip, pass `--openvmm-arg=--x-time-abi-v1` | +| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` | | H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, `CLOCK_MONOTONIC` on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | | H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index ea42d3611..cc6260432 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -522,7 +522,6 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: os.fspath(context.kernel), "--initrd", os.fspath(context.initrd), - # Before the flip, pass --openvmm-arg=--x-time-abi-v1. *context.openvmm_args, "--x-time-abi-verify", ] @@ -550,7 +549,7 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: (line.strip() for line in reversed(output.splitlines()) if line.strip()), "no output", ) - if "x-time-abi" in output and "unexpected argument" in output: + if "unexpected argument '--x-time-abi-verify'" in output: last = "this OpenVMM predates the time ABI's --x-time-abi-verify mode" return CheckResult( "H3", diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index a33040517..6619e27bf 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -1264,16 +1264,18 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: self.assertEqual(context.facts["profile"], "intel.icelake-sp.v1") command = run.call_args.args[0] self.assertEqual(command[-1], "--x-time-abi-verify") - self.assertNotIn("--x-time-abi-v1", command) + self.assertEqual(command[command.index("--machine") + 1], "microvm") self.assertEqual(command[command.index("--kernel") + 1], str(context.kernel)) - switched = context_with_files() - switched.openvmm_args = ("--x-time-abi-v1",) + # --openvmm-arg passes development options, such as a test hook. + hooked = context_with_files() + hooked.openvmm_args = ("--x-time-abi-test-hook", "force-utc-downtime") with patch.object( doctor.subprocess, "run", return_value=completed(line) ) as run: - self.assertTrue(doctor.check_openvmm_preflight(switched).passed) + self.assertTrue(doctor.check_openvmm_preflight(hooked).passed) self.assertEqual( - run.call_args.args[0][-2:], ["--x-time-abi-v1", "--x-time-abi-verify"] + run.call_args.args[0][-3:], + ["--x-time-abi-test-hook", "force-utc-downtime", "--x-time-abi-verify"], ) failure = ( "NVX-TIME-ABI-VERIFY: v=1 status=fail backend=kvm " @@ -1285,9 +1287,16 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: '[E_TSC_SYNC_UNSUPPORTED] OpenVMM verification failed (exit 1): failed: "sync" unsupported', ), ( - completed("", 2, "error: unexpected argument '--x-time-abi-v1' found"), + completed( + "", 2, "error: unexpected argument '--x-time-abi-verify' found" + ), "predates the time ABI", ), + # The flip removed --x-time-abi-v1; passing it is reported as is. + ( + completed("", 2, "error: unexpected argument '--x-time-abi-v1' found"), + "unexpected argument '--x-time-abi-v1' found", + ), (completed(line.replace("1000000000", "200000000")), "[E_LAPIC_RATE_"), ( completed(line.replace("tsc_hz=2793437000", "tsc_hz=4000", 1)), From c6ce218a4ffebb780f8657eb48eb5e79c53439cc Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 1 Oct 2026 22:11:19 -0700 Subject: [PATCH 120/188] ci: report the time ABI checks' CPU time against the spec's budget Spec 6f44e2c (Q22) adds cpu_us to the phase lines of nvx-time status: the CPU time of the boot, capture, or restore check. Each backend has a budget per phase (spec 45ec1a15): a base plus an increment per additional vCPU. The table holds the spec's current values, the bare-metal floors from the guest's v7 measurements and KVM's restore budget from azure-kvm-5; its final values are the larger of the bare-metal and Azure maxima. elapsed_us stays the check's wall time, which includes waits for a CPU at SCHED_IDLE, and has no budget. - The evidence line gives the cpu_us range of each status query's newest line and, for the run's backend, how many checks exceed their phase's budget. A guest image without cpu_us reports elapsed_us only. - Doctor H6's detail adds the boot check's cpu_us where the guest reports it. - time_abi.CHECK_CPU_BUDGET_US holds one budget per backend and phase, the only place to change when the spec updates its values, and check_cpu_budget_us() computes it; doc/ci.md points to the table rather than repeating the values. CI reports both values and gates on neither: the performance gate is the A/B comparison outside CI. Lands with the guest's quiet-console wiring, whose status lines it reads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 9 ++++++- scripts/nvx_tools/doctor.py | 5 +++- scripts/nvx_tools/microvm_tests.py | 36 +++++++++++++++++++++------ scripts/nvx_tools/time_abi.py | 26 +++++++++++++++++++ scripts/test_microvm_tests.py | 40 ++++++++++++++++++++++++++++++ scripts/test_time_abi.py | 30 +++++++++++++++++++--- 6 files changed, 134 insertions(+), 12 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index fdc47f464..e13c83908 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -113,7 +113,14 @@ for failed jobs: the number of warp probe runs with their worst `max_abs_offset_ns` and `max_backward_ns`, the count and `elapsed_us` range of the newest check that each `nvx-time status` query reported (boot after a cold boot, restore after a restore), the exhaustive check's summary, and the -`restore-downtime` stall counts. +`restore-downtime` stall counts. Where the guest reports a check's CPU time +(`cpu_us`), the line also gives its range and how many checks exceed the +backend's [CPU-time budget](design/time-abi.md#performance-expectations-and-acceptance-gate) +for their phase, which the spec sets from guest measurements. +`CHECK_CPU_BUDGET_US` in `scripts/nvx_tools/time_abi.py` holds the budgets, one +per backend and phase. `elapsed_us` is wall time, including waits behind the +workload, and has no budget. CI gates on neither: the performance gate is the +A/B comparison outside CI. The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run `test-microvm --debug-kernel` on the CI debug kernel (`build/vmlinux-debug`, diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index cc6260432..831c84469 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -807,10 +807,13 @@ def check_guest_warp(context: DoctorContext) -> CheckResult: return CheckResult("H6", False, f"vcpus={count}: {first}") boots.append(boot) rounds.extend(count_rounds) + # The boot check's wall time and, where the guest reports it, its CPU + # time, which the spec budgets; both are evidence only. + cpu_us = f" boot_cpu_us={boot['cpu_us']}" if "cpu_us" in boot else "" runs.append( f"vcpus={count} rounds={len(count_rounds)} " f"idle_gaps_s={','.join(count_gaps) or 'none'} " - f"boot_elapsed_us={boot.get('elapsed_us', '?')}" + f"boot_elapsed_us={boot.get('elapsed_us', '?')}{cpu_us}" ) def worst(name: str) -> int: diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 24d7372bd..a5640982f 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -61,11 +61,13 @@ from .openvmm_process import OpenvmmProcess, TcpConsole from .time_abi import ( ABI_VERSION, + CHECK_CPU_BUDGET_US, WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_FAILURE_MARKER, WARP_SUMMARY_PREFIX, TimeAbiFailure, TimeAbiMonitor, + check_cpu_budget_us, check_warp_probe, describe_exit_status, parse_fields, @@ -1356,17 +1358,21 @@ def _check_exhaustive_report(text: str, *, processors: int) -> None: raise RuntimeError("time ABI exhaustive check: " + "; ".join(problems)) -def time_abi_evidence(output_dir: Path) -> dict[str, str]: +def time_abi_evidence(output_dir: Path, backend: str | None = None) -> dict[str, str]: """Sum up the time ABI evidence in the guest logs of one run. nvx-time status prints every recorded phase, oldest first, and then its runtime line, so a restored guest repeats its source's boot and capture lines. Only each query's newest phase line counts: one boot per cold-boot - query and one restore per post-restore query. + query and one restore per post-restore query. Where the guest reports a + check's CPU time (cpu_us), the evidence also counts the checks over + ``backend``'s CPU-time budget for their phase; it never gates on either. """ offsets: list[int] = [] backward: list[int] = [] elapsed: dict[str, list[int]] = {"boot": [], "capture": [], "restore": []} + cpu: dict[str, list[int]] = {"boot": [], "capture": [], "restore": []} + over_budget: dict[str, int] = {"boot": 0, "capture": 0, "restore": 0} exhaustive: list[str] = [] stalls: list[str] = [] for path in sorted(output_dir.rglob("*.log")): @@ -1398,11 +1404,23 @@ def time_abi_evidence(output_dir: Path) -> dict[str, str]: continue last, newest = newest, None if ( - last is not None - and last["status"] == "ok" - and last["phase"] in elapsed + last is None + or last["status"] != "ok" + or last["phase"] not in elapsed ): - elapsed[last["phase"]].append(int(last["elapsed_us"])) + continue + phase = last["phase"] + elapsed[phase].append(int(last["elapsed_us"])) + if "cpu_us" in last: + cpu_us = int(last["cpu_us"]) + cpu[phase].append(cpu_us) + budget = ( + None + if backend is None + else check_cpu_budget_us(backend, phase, int(last["cpus"])) + ) + if budget is not None and cpu_us > budget: + over_budget[phase] += 1 except (KeyError, ValueError): continue evidence: dict[str, str] = {} @@ -1414,6 +1432,10 @@ def time_abi_evidence(output_dir: Path) -> dict[str, str]: if values: evidence[f"{phase}_markers"] = str(len(values)) evidence[f"{phase}_elapsed_us"] = f"{min(values)}-{max(values)}" + if cpu[phase]: + evidence[f"{phase}_cpu_us"] = f"{min(cpu[phase])}-{max(cpu[phase])}" + if backend is not None and phase in CHECK_CPU_BUDGET_US.get(backend, {}): + evidence[f"{phase}_cpu_over_budget"] = str(over_budget[phase]) if exhaustive: evidence["exhaustive"] = ",".join(exhaustive) if stalls: @@ -1425,7 +1447,7 @@ def report_time_abi_evidence( output_dir: Path, *, backend: str, guest: str, debug_kernel: bool ) -> str | None: """Print the run's evidence line and add it to the GitHub job summary.""" - evidence = time_abi_evidence(output_dir) + evidence = time_abi_evidence(output_dir, backend) if not evidence: return None fields = { diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index ca94d7d82..48b28d71a 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -33,6 +33,23 @@ } MIN_TSC_HZ = 500_000_000 MAX_TSC_HZ = 10_000_000_000 +# doc/design/time-abi.md, "Performance": each backend has a CPU-time +# (cpu_us) budget per phase (boot, capture, and restore): a base plus an +# increment per additional online CPU, in microseconds. These are the spec's +# bare-metal floors from the guest's v7 measurements; its final values are the +# larger of those and the Azure maxima. This table is the one place the +# harness reads them. The checks' wall time (elapsed_us) has no budget. CI +# reports both and gates on neither: the performance gate is the A/B +# comparison outside CI. +CHECK_CPU_BUDGET_US: Mapping[str, Mapping[str, tuple[int, int]]] = { + "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (5_000, 1_500)}, + "mshv": {"boot": (3_000, 500), "capture": (1_000, 400), "restore": (1_500, 500)}, + "whp": { + "boot": (5_000, 1_000), + "capture": (1_000, 400), + "restore": (20_000, 2_000), + }, +} WARP_BOUND_NS = 1000 WARP_PROBE_PATH = "/sbin/nvx-time-probe" WARP_SUMMARY_PREFIX = "NVX-TIME-PROBE warp " @@ -204,6 +221,15 @@ def parse_violation(line: str) -> dict[str, str] | None: return fields +def check_cpu_budget_us(backend: str, phase: str, cpus: int) -> int | None: + """Return the CPU-time budget of one time ABI ``phase`` check at ``cpus`` CPUs.""" + budget = CHECK_CPU_BUDGET_US.get(backend, {}).get(phase) + if budget is None or cpus < 1: + return None + base, per_cpu = budget + return base + per_cpu * (cpus - 1) + + def describe_exit_status(returncode: int | None) -> str | None: """Describe an OpenVMM exit status that a time ABI guest failure produced.""" if returncode is None or returncode not in STATUS_CLASSES: diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 166d8821e..4cb76ddef 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3749,6 +3749,46 @@ def restored(elapsed: int) -> bytes: }, ) + def test_time_abi_evidence_reports_cpu_time_against_the_budget(self): + def query(phase: str, cpus: int, cpu_us: int | None) -> str: + cpu = "" if cpu_us is None else f" cpu_us={cpu_us}" + generation = 0 if phase == "boot" else 1 + return ( + f"NVX-TIME-ABI: v=1 phase={phase} status=ok cpus={cpus} " + f"tsc_hz=2793437000 lapic_hz=1000000000 generation={generation} " + f"elapsed_us=25000{cpu}\r\n" + f"NVX-TIME-ABI: v=1 phase=runtime status=synchronized " + f"generation={generation} discontinuities={generation} offset_ns=0 " + "uncertainty_ns=900 rejected_samples=0 last_sample_error=none\r\n" + ) + + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + # Within KVM's budget (20 ms for a boot at 8 CPUs), over it (5 ms + # for a restore at 1 CPU), and an older image without cpu_us, which + # only reports elapsed_us. + (output_dir / "smp-8.log").write_text(query("boot", 8, 4600)) + (output_dir / "restore-1.log").write_text(query("restore", 1, 5500)) + (output_dir / "restore-8.log").write_text(query("restore", 8, 2000)) + (output_dir / "older.log").write_text(query("boot", 1, None)) + evidence = microvm_tests.time_abi_evidence(output_dir, "kvm") + unknown = microvm_tests.time_abi_evidence(output_dir) + # Each phase has its own budget: a larger restore budget absorbs + # the 1-CPU restore. + phased = {**time_abi.CHECK_CPU_BUDGET_US["kvm"], "restore": (6_000, 0)} + with patch.dict(time_abi.CHECK_CPU_BUDGET_US, {"kvm": phased}): + per_phase = microvm_tests.time_abi_evidence(output_dir, "kvm") + self.assertEqual(evidence["boot_markers"], "2") + self.assertEqual(evidence["boot_cpu_us"], "4600-4600") + self.assertEqual(evidence["boot_cpu_over_budget"], "0") + self.assertEqual(evidence["restore_cpu_us"], "2000-5500") + self.assertEqual(evidence["restore_cpu_over_budget"], "1") + self.assertEqual(per_phase["restore_cpu_over_budget"], "0") + self.assertEqual(per_phase["boot_cpu_over_budget"], "0") + # Without a backend there is no budget to compare with. + self.assertEqual(unknown["restore_cpu_us"], "2000-5500") + self.assertNotIn("restore_cpu_over_budget", unknown) + def test_time_abi_evidence_line_goes_to_the_log_and_the_job_summary(self): with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) / "out" diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 6619e27bf..d67d6da62 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -135,6 +135,22 @@ def test_names_the_catalog_profiles(self): ["intel.skylake-sp.v1", "intel.icelake-sp.v1", "intel.emeraldrapids.v1"], ) + def test_computes_the_checks_cpu_time_budget(self): + # The spec's bare-metal floors: a base plus an increment per additional + # CPU, one budget per backend and phase. + for backend in ("kvm", "mshv", "whp"): + self.assertEqual( + set(time_abi.CHECK_CPU_BUDGET_US[backend]), + {"boot", "capture", "restore"}, + ) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "boot", 1), 6_000) + self.assertEqual(time_abi.check_cpu_budget_us("mshv", "restore", 8), 5_000) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 2), 1_400) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 1), 20_000) + self.assertIsNone(time_abi.check_cpu_budget_us("hvf", "boot", 1)) + self.assertIsNone(time_abi.check_cpu_budget_us("kvm", "runtime", 1)) + self.assertIsNone(time_abi.check_cpu_budget_us("kvm", "boot", 0)) + class MonitorTests(unittest.TestCase): def test_classifies_commands(self): @@ -1540,7 +1556,9 @@ def boot(cpus: int) -> str: return " ALPINE-MICROVM-BOOT-OK: 3.22.1\n" + line + STATUS_OK smp = boot(8) + warp_output(pairs=28) * 4 + warp_output(pairs=28, offset=60) - single = boot(1) + warp_output(pairs=0, offset=0) + # A newer guest also reports the boot check's CPU time. + single = boot(1).replace("elapsed_us=1873", "elapsed_us=1873 cpu_us=1100") + single += warp_output(pairs=0, offset=0) with ( patch.object(doctor.os, "cpu_count", return_value=8), patch.object( @@ -1551,8 +1569,14 @@ def boot(cpus: int) -> str: ): result = doctor.check_guest_warp(context) self.assertTrue(result.passed, result.detail) - self.assertIn("vcpus=8 rounds=5 idle_gaps_s=0.1,1,5,1", result.detail) - self.assertIn("vcpus=1 rounds=1 idle_gaps_s=none", result.detail) + self.assertIn( + "vcpus=8 rounds=5 idle_gaps_s=0.1,1,5,1 boot_elapsed_us=1873;", + result.detail, + ) + self.assertIn( + "vcpus=1 rounds=1 idle_gaps_s=none boot_elapsed_us=1873 boot_cpu_us=1100", + result.detail, + ) self.assertEqual(context.facts["guest_warp_ns"], "60") (smp_call, single_call) = run.call_args_list for call, processors, gaps in ( From 74786bad6a33096cad3cd383e7762d92c6956921 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:11:33 -0700 Subject: [PATCH 121/188] doc: update the time ABI spec for the L3 fix, Q27, and MSHV's restore changes Bring the specification up to date with what has landed in OpenVMM at e06ed4d4a and with today's measurements and decisions. - CPUID leaf 4: state the VMM-owned cache sharing (one core per vCPU, L1 and L2 private, the L3 shared by every vCPU), and make a change to how OpenVMM computes its own leaves a snapshot compatibility break (E_CPU_SURFACE, recapture), as a non-goal and in the effective CPUID text. The KVM multi-vCPU cold-boot row gives the bisect result: the private L3 made cache-info initialization wait a tick, which the fix removes. - Q27: CONFIG_IRQ_TIME_ACCOUNTING stays off; it costs WHP cold boots 15 to 24 ms in first-touch exits and changes no cpu_us. cpu_us therefore includes interrupts and host preemption, covers the daemon's side of its fork at restore, and is checked per sample. - cpu_us budgets: a + b * (n - 1) at 512 MiB from the bare-metal floors, with KVM restore at 5 + 1.5 from azure-kvm-5; the Azure maxima for MSHV and WHP stay TBD. WHP's restore cost grows with guest memory. - MSHV: preflight leaves partition time frozen until the synchronized TSC set or the first VP run; CPUID results are registered only where VP 0 does not already present them, and every entry is read back; the host CPUID is read on a background thread; restores pay about 1.1 ms per first-touched 2 MiB chunk in the root driver. - The start-up budget cites the KVM measurement at 31a5a04f9, the boot check anchor gives the nested WHP margins, the migration notes reject by version only pre-time-ABI snapshots, FB_CLEAR's absence has its precise reason, and the gate notes prometheus32's idle clock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 93 ++++++++++++++++++++++++++++-------------- 1 file changed, 62 insertions(+), 31 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 43ae0e66b..cbc5637b3 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -47,6 +47,9 @@ Goals: Non-goals: - Restore across backends or across CPU generations. Both are rejected. +- Restore by an OpenVMM build that computes a different + [effective CPUID](#cpu-profiles) for the snapshot's profile and topology. + It is rejected with `E_CPU_SURFACE`. - TSC scaling and RDTSC trapping. - Paravirtual clocks: kvmclock, the Hyper-V reference TSC page and reference counter, synthetic timers, VMGenID, and VMClock. @@ -262,8 +265,8 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | -| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity and explicit zero leaves; preflight reads the identity leaves back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`); programming them takes 43 to 62 µs | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every entry of the effective CPUID; reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity leaves, while the explicit zero leaves already read zero in VP 0's view and get none (see the next row); preflight reads every entry back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`) | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 12 of 61 entries on prometheus30 (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 13 of 65 on azure-azlinux-5 (the same plus invariant TSC); registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -273,7 +276,7 @@ settled cell on the registered hosts. | Invariant TSC exposed | CPUID bit. KVM hides it from a guest with `"Hv#1"` until KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is set, so OpenVMM writes 1 to it host-side at vCPU creation | Forced by the profile through a CPUID intercept result: Azure's nested MSHV cannot offer the bit to its guests, although the host OS sees an invariant TSC, and without it each AP pays about 150 ms of calibration. Qualification measures the property instead (`H4`, `H6`) | Set by the CPUID override on every host: on Azure, WHP cannot offer it through the feature banks (bank 1 lacks `TscInvariant`), although the host OS sees an invariant TSC. There the guest TSCs stayed within 60 ns of each other at 8 vCPUs over 60 s and matched the declared rate against host QPC to 0.000 ppm (residual at most 0.07 µs) over 118 s; `H4` and `H6` measure both on every host | | No paravirtual or synthetic features | No KVM leaves; `KVM_CAP_ENFORCE_PV_FEATURE_CPUID`, so KVM's paravirtual MSRs raise #GP; KVM's in-kernel Hyper-V MSRs are unreachable behind the filter | No synthetic processor features | `--hv` stays rejected for the microVM | | Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | -| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time, write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | +| Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time (already frozen since preflight), write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | | Read-back before any VP runs | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | | Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | | VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | @@ -358,8 +361,13 @@ document in pretty form. VMM-owned bits are a code table, not profile data: `CPUID.1:EBX[31:16]` (logical count and initial APIC ID), x2APIC (`CPUID.1:ECX[21]`), the core and cache-sharing counts in `CPUID.4:EAX[31:14]`, and the topology leaves -`0xB` and `0x1F`, which profiles omit. Runtime-owned bits mirror control -state: `OSXSAVE`, `OSPKE`, and the XSAVE sizes for the current XCR0 and XSS. +`0xB` and `0x1F`, which profiles omit. The machine has one socket and one +core per vCPU ([configuration boundary](configuration-boundary.md)), so in +`CPUID.4` every cache reports the vCPU count less one as the cores per +socket (`EAX[31:26]`), and the L3 cache reports it as the logical processors +sharing the cache (`EAX[25:14]`) too; the L1 and L2 caches report 0 there, +one vCPU each. Runtime-owned bits mirror control state: `OSXSAVE`, `OSPKE`, +and the XSAVE sizes for the current XCR0 and XSS. The time ABI owns `0x40000000..=0x4fffffff`, `0x15`, and `0x16`, and every profile pins the [CPU time bits](#cpu-time-bits). Invariant TSC and ARAT are pinned set and, with the hypervisor bit, exempt from host-support @@ -384,6 +392,10 @@ APIC identity, and the runtime-owned bits have mask 0. OpenVMM computes it, records it as the manifest's packed binary record, and recomputes it on restore, so the record never depends on how a backend enumerates or caches CPUID, and one profile and topology give the same record on every backend. +The record also fixes OpenVMM's own leaves and VMM-owned bits, so changing +how OpenVMM computes them breaks snapshot compatibility: snapshots that +earlier builds captured fail restore with `E_CPU_SURFACE` and must be +recaptured, and the release notes say so. Its canonical JSON form (`openvmm-effective-cpuid/v1`) serves tools, never the start paths. At preflight, before any VP runs, each backend reports VP 0's CPUID for the governed leaves (KVM from the `KVM_SET_CPUID2` table it @@ -391,9 +403,8 @@ programmed, MSHV by reading VP 0, WHP from VP 0's register view), reading an entry without a subleaf at subleaf 0, and OpenVMM compares the report with the effective CPUID under its masks. MSHV and WHP also report VP 0's view at the reserved entries the host's CPUID enumerates (verification step 6). MSHV -reads every reported entry from the hypervisor, including those it -registered with full masks; only the identity range's explicit zero leaves -are reported as registered, and preflight instead requires VP 0 to read zero +reads every reported entry from the hypervisor, the explicit zero leaves of +the identity range included, and preflight also requires VP 0 to read zero at six sentinel leaves of the range (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`; `E_IDENTITY_ROUTING`). It batches the reads into rep @@ -545,7 +556,9 @@ development hosts, the shared profile drops what one backend alone offers: PKU, UMIP, and FDP_EXCPTN_ONLY (KVM only), FLUSH_L1D (not on WHP), and the AMD-alias speculation bits in `0x80000008` EBX (KVM only; Intel guests use the `7.0` EDX equivalents, which every backend presents). Without FLUSH_L1D, -and with `FB_CLEAR` clear because KVM does not present it on these hosts, +and with `FB_CLEAR` clear because KVM's feature MSR does not offer it on these +hosts (KVM would accept it in a host-initiated write, but v1 pins only what +every backend's surface offers), Linux cannot confirm the microcode that makes VERW clear the fill buffers. It therefore reports MMIO Stale Data as "Vulnerable: Clear CPU buffers attempted, no microcode" on every backend. Before the profiles, KVM @@ -764,8 +777,10 @@ In the worker, with every VP stopped: and effective CPUID (`E_PROFILE_UNSUPPORTED`, `E_CPU_SURFACE`, `E_CPU_UNLISTED`), identity routing (`E_IDENTITY_ROUTING`), the synchronized-set primitive (`E_TSC_SYNC_UNSUPPORTED`), and no scaling - (`E_TSC_SCALING_ACTIVE`). MSHV's preflight writes `TimeFreeze` to probe - the primitive and reads VP 0's CPUID at its reset state. + (`E_TSC_SCALING_ACTIVE`). MSHV's preflight freezes partition time, which + probes the primitive, and leaves it frozen, as a reset does: steps 9 to 12 + run with it frozen, and at cold boot the first VP run resumes it. It also + reads VP 0's CPUID at its reset state. 9. Read `F_d` and `L_d` and apply the [rate policy](#tsc-rate-policy-and-lapic-rate-rule) (`E_TSC_RATE_UNAVAILABLE`, `E_TSC_RATE_IMPLAUSIBLE`, @@ -1097,8 +1112,9 @@ starting 150 ms after its time ABI boot step (`nvx-time boot`) at [Snapshot agent](#snapshot-agent)). The boot step is the anchor because every mode reaches it, whereas a mode that execs an agent reports readiness later, on the agent's own protocol. In shell mode the boot step comes 15 to -44 ms before shell-ready on Alpine and 26 to 84 ms before it on Ubuntu at 1 to -8 vCPUs, so the checks start about 65 to 135 ms after it. Because they start +44 ms before shell-ready on Alpine on bare metal, up to 96 ms before it on +the nested Azure WHP runners, and 26 to 84 ms before it on Ubuntu at 1 to +8 vCPUs, so the checks start about 54 to 135 ms after it. Because they start at `SCHED_IDLE`, a start closer to shell-ready costs the boot path nothing. The checks start the daemon when they pass. Every check stays fail-fast: a failure powers the guest off with status 193, even if the @@ -1200,13 +1216,15 @@ latency and is not budgeted. `cpu_us` is the CPU time that the `nvx-time` processes spend on it (`CLOCK_PROCESS_CPUTIME_ID`, summed over their threads); other processes, such as the shell's children in steps 9 and 10, are not counted. At restore, `cpu_us` counts only step 13's part: the -daemon's preparation and the step 11 checks. Steps 6 to 12 are the readiness +daemon's preparation, including its side of the fork that runs the checks, +and the step 11 checks. Steps 6 to 12 are the readiness path, whose cost the restore latency gate already measures, so `cpu_us` is -the background cost to the workload. The NVX kernel does not charge -interrupt time to it (`CONFIG_IRQ_TIME_ACCOUNTING`), but it still scales with -the host processor's clock: a host idling at a low frequency runs the checks -more slowly. The fleet matrix driver checks `cpu_us` against the backend's -budget for the phase in +the background cost to the workload. It includes interrupts that arrive +while the checks run, and it scales with the host processor's clock: a host +idling at a low frequency runs the checks more slowly, and because the +Hyper-V identity gives the guest no steal-time accounting, host preemption is +charged to the checks too. The fleet matrix driver checks every `cpu_us` +sample against the backend's budget for the phase in [Performance expectations](#performance-expectations-and-acceptance-gate) during validation; CI reports it without gating on it. @@ -1684,7 +1702,10 @@ Routing CI jobs by generation labels is optional future work. **Gate.** Per backend and vCPU count, no p50 is worse than the base-branch median by more than `max(5%, 2 ms)`. The gate covers all 36 one-vCPU metrics and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. It is measured on -the CI Azure platforms; bare-metal results are informational. +the CI Azure platforms; bare-metal results are informational. prometheus32 +idles at 800 MHz (`intel_pstate` powersave without HWP), which inflates its +absolute restore latency (`restored_ms` p50 at one vCPU 39 ms, against 23 ms +at full clock) and its `cpu_us`; the arms of an A/B share the effect. Expected effects: @@ -1698,12 +1719,12 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are TBD(guest): the next guest build's maxima plus headroom. Provisionally they are 2.5 ms at one vCPU plus 0.75 ms per additional vCPU on KVM and MSHV and 5 ms plus 1.5 ms on WHP. Wiring v6 measured boot medians of 1.6 to 3.0 ms on MSHV, 3.6 to 5.0 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.3 to 6.6 ms on WHP at 1 to 8 vCPUs, with captures under 2 ms everywhere; WHP restores measured 22 to 30 ms on bare metal and 29 to 45 ms nested. The next guest build counts only step 13 at restore and runs with `CONFIG_IRQ_TIME_ACCOUNTING` | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are a + b·(n − 1) ms at n vCPUs, set to the larger of the bare-metal and Azure maxima of wiring v7 on the production kernel, plus headroom. The bare-metal floors, which cover every sample on prometheus28, 30, and 32 with 10 to 30% headroom, are boot 6 + 2, capture 1 + 0.4, and restore 4 + 0.75 on KVM, where nested azure-kvm-5 raises restore to 5 + 1.5 (medians 3.3 to 7.8 ms at 1 to 8 vCPUs and a maximum of 13.1 ms, from first-touch faults on the file-backed restored RAM); 3 + 0.5, 1 + 0.4, and 1.5 + 0.5 on MSHV; and 5 + 1, 1 + 0.4, and 20 + 2 on WHP. The Azure maxima are TBD(kvm, mshv, whp). The budgets are set at 512 MiB, the gate's memory size, which bounds the smaller validation guests: WHP's restore `cpu_us` grows with guest memory (medians 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure), because the checks' first touches fault in lazily registered RAM. Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated and not yet explained: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32, where `msr_init` waits about one 10 ms tick for CPU 1, and about +25 ms at 2 vCPUs on azure-kvm-5 | +| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32 and about +25 ms on azure-kvm-5. A component bisect blames no profile field: OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on most boots. With the L3 shared by the socket the wait is gone (0 of 40 boots), and prometheus32 cold boots change by −4.6 [−10.1, +0.2] ms at 2 vCPUs and −7.2 [−11.1, −2.6] ms at 8. The msr driver's per-CPU hotplug callback (`msr_init`) still waits one or two ticks on multi-vCPU boots, as it did before the profiles | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | -| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority | +| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | Expected wins are tracked separately and do not relax the gate. @@ -1711,7 +1732,11 @@ Expected wins are tracked separately and do not relax the gate. **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), measured on prometheus32 (KVM) with the kvm agent's attribution harness; the -flip requires it. Neither path encodes, decodes, or hashes a profile document +flip requires it. At `31a5a04f9`, up to the backend preflight at one vCPU, +a restore starts 0.46 ms and a cold boot 0.28 ms faster than `e7ec0ca6c`; +the only added work is the worker's time ABI CPU checks, 40 to 60 µs over +the pre-profile path, and every 90% upper bound is within +0.30 ms. +Neither path encodes, decodes, or hashes a profile document or an effective CPUID: - the pinned profiles are compile-time constants with precomputed canonical @@ -1742,7 +1767,10 @@ The backends add their own CPUID work: nested roots. Skipping the range took MSHV's from 96 to 49 µs on prometheus30 and from 305 to 220 µs nested, more than its share of the queries suggests. KVM takes its surface from `KVM_GET_SUPPORTED_CPUID` and - does not enumerate. + does not enumerate. MSHV enumerates on a thread that partition creation + starts, so the read leaves the start path; starting the thread costs about + 50 µs in a root partition, and keeping its channel until the partition + drops avoids freeing memory there, which costs 12 to 18 µs. - MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on prometheus30 and 51 µs on azure-azlinux-5, against 0.51 and 0.82 ms with one call per entry. WHP has no batched read: its report takes 57 to 60 @@ -1767,8 +1795,8 @@ after the acknowledgement, outside these phases. `nvx-time status` reports checks. On every cold boot and restore, the worker logs `time ABI CPU checks passed` with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. -Their sum per restore at one vCPU is about 0.19 ms on bare-metal MSHV and -0.14 ms nested, and 0.87 ms on bare-metal WHP and 1.6 to 1.9 ms nested, about +Their sum per restore at one vCPU is about 0.16 ms on bare-metal MSHV and +0.12 ms nested, and 0.87 ms on bare-metal WHP and 1.6 to 1.9 ms nested, about 95% of it WHP's per-leaf read-back. OpenVMM's comparison and unlisted-entry check take 40 to 66 µs of each. @@ -1893,8 +1921,10 @@ Removed from NVX: and needs ACPI CPPC data the microVM lacks); and `CONFIG_KVM_GUEST` with `CONFIG_PARAVIRT_CLOCK` and `CONFIG_HALTPOLL_CPUIDLE`, which are dormant without a KVM signature. `CONFIG_HYPERVISOR_GUEST` and `CONFIG_PARAVIRT` - stay on and `CONFIG_HYPERV` stays off. `CONFIG_IRQ_TIME_ACCOUNTING` is - turned on, so `cpu_us` does not charge interrupts to the checks. The CI + stay on and `CONFIG_HYPERV` stays off. `CONFIG_IRQ_TIME_ACCOUNTING` stays + off: it slows WHP cold boots by 15 to 24 ms on bare metal, because the guest + then first-touches about 400 more pages, each a memory-access exit on WHP, + and it changes no `cpu_us` measurably. The CI debug kernel adds `CONFIG_DEBUG_KERNEL` with the soft-lockup and hung-task detectors only. - Guest: RTC polling in `nvx-reseed`, and the restore packet v1 to v3 parser @@ -1912,8 +1942,9 @@ Removed from NVX: Migration impact: -- Every existing snapshot is rejected with `E_SNAPSHOT_VERSION`; templates - must be recaptured. +- Every snapshot from a release before the time ABI (manifest version 5 or + older) is rejected with `E_SNAPSHOT_VERSION`; templates must be + recaptured. - OpenVMM, the kernel, and the initramfs must be upgraded together. A new guest on an old OpenVMM fails the boot check (status 193); an old guest's snapshot agent cannot parse packet v4 and fails closed. From 669859dd69b27f73686b3e469e96460950524bb6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:17:55 -0700 Subject: [PATCH 122/188] doc: mark the time ABI spec Proposed until validation passes The time ABI is implemented on the time-abi-v1 integration branches, but it becomes the machine contract only when they pass p6 validation and merge. Mark the specification Proposed until then, as the design index defines the marker; a later commit removes it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index cbc5637b3..7cb06f5fa 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -2,9 +2,11 @@ [Design index](../design.md) -This document specifies NVX time ABI v1, the single guest-visible time, -timer, and clock contract of the microVM profile on KVM, MSHV, and WHP, as NVX -and its OpenVMM fork (`nanvix/openvmm`) implement it. It replaces the earlier +**Proposed.** This document specifies NVX time ABI v1, the single +guest-visible time, timer, and clock contract of the microVM profile on KVM, +MSHV, and WHP. NVX and its OpenVMM fork (`nanvix/openvmm`) implement it on +their `time-abi-v1` integration branches, and it becomes part of the current +machine contract when they pass validation and merge. It replaces the earlier time rules of [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) and the clock tokens of [Cold boot](cold-boot.md#effective-command-line). From b39141a1bbac4e8ca59fcc9b7c74c3fbae5186e1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:27:58 -0700 Subject: [PATCH 123/188] doc: add the effective CPUID to the template compatibility class An OpenVMM build that computes a different effective CPUID for a profile and topology, such as the L3 cache fix, rejects earlier templates with E_CPU_SURFACE. Say so where the design describes template compatibility, matching the time ABI specification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/snapshot-and-restore.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/doc/design/snapshot-and-restore.md b/doc/design/snapshot-and-restore.md index 82bce57b4..d0ed2e9ae 100644 --- a/doc/design/snapshot-and-restore.md +++ b/doc/design/snapshot-and-restore.md @@ -382,12 +382,15 @@ conversion and standalone NVX snapshot import are not supported. ## Template compatibility and placement A template represents a compatibility class, not an arbitrary fleet image. -The class includes the backend, CPU profile and generation, guest kernel and -agent build, effective command line, device roles and geometry, network -identity/policy, and consumed layer identities. Deployment must rebuild or -recertify templates on relevant rollouts; the VMM's actual compatibility and -saved-state checks, not a promise about every build with the same version -string, remain authoritative. +The class includes the backend, CPU profile and generation, the +[effective CPUID](time-abi.md#cpu-profiles) that the OpenVMM build computes +for that profile and topology, guest kernel and agent build, effective command +line, device roles and geometry, network identity/policy, and consumed layer +identities. Deployment must rebuild or recertify templates on relevant +rollouts: an OpenVMM build that computes a different effective CPUID, for +example a different cache topology, rejects earlier templates with +`E_CPU_SURFACE`. The VMM's actual compatibility and saved-state checks, not a +promise about every build with the same version string, remain authoritative. Processor capacity and captured RAM geometry are immutable, but the opt-in activation contracts below allow one template to serve several online CPU From ac3d81081d30cb21d21a66530f58d3cfe4aff61e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:48:21 -0700 Subject: [PATCH 124/188] doc: record the MSHV root-driver costs of a restore mshv traced an MSHV restore on prometheus30. Three root-driver costs, which the legacy path pays alike, make up 50 of 70 ms at one vCPU and 130 of 150 ms at eight: no-access RAM registration, the one-page deposit loop of MSHV_CREATE_VP for each later processor, and the copy-on-write first touch of each 2 MiB chunk after resume. Record them as host characteristics with their Azure figures and remedies; the ABI does not change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 7cb06f5fa..65d099e1c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1720,7 +1720,7 @@ Expected effects: | `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | -| MSHV VP creation | Serialized at about 14 ms per application processor on bare metal (27 and 85 ms at 4 and 8 vCPUs); the frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | +| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On prometheus30 (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first, about 14 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are a + b·(n − 1) ms at n vCPUs, set to the larger of the bare-metal and Azure maxima of wiring v7 on the production kernel, plus headroom. The bare-metal floors, which cover every sample on prometheus28, 30, and 32 with 10 to 30% headroom, are boot 6 + 2, capture 1 + 0.4, and restore 4 + 0.75 on KVM, where nested azure-kvm-5 raises restore to 5 + 1.5 (medians 3.3 to 7.8 ms at 1 to 8 vCPUs and a maximum of 13.1 ms, from first-touch faults on the file-backed restored RAM); 3 + 0.5, 1 + 0.4, and 1.5 + 0.5 on MSHV; and 5 + 1, 1 + 0.4, and 20 + 2 on WHP. The Azure maxima are TBD(kvm, mshv, whp). The budgets are set at 512 MiB, the gate's memory size, which bounds the smaller validation guests: WHP's restore `cpu_us` grows with guest memory (medians 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure), because the checks' first touches fault in lazily registered RAM. Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | From dcab8cd1e68ecfaca69026580c16c0ceb7217c8a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:17:02 -0700 Subject: [PATCH 125/188] openvmm: bump the time ABI v1 integration head Advance nanvix/openvmm nvx/time-abi-v1 from cae746f6f to e06ed4d4a, a fast-forward that adds the commits held back by the lockstep freeze: - virt_mshv: read the host CPUID on another thread, keep partition time frozen from the preflight to the TSC set, register only the CPUID results VP 0 does not already present, and read the partition capabilities' CPUID in one hypercall; - virt: type the x86 topology helpers for x86 topologies (the aarch64 build fix), build the time ABI effective CPUID in one place, and share the L3 cache across the socket in CPUID leaf 4; - cpu_profile: point the effective CPUID docs at virt's builder; - openvmm_entry, virt: trim the time ABI restore preflight. The leaf 4 change alters the L3 subleaf at two or more vCPUs, so multi-vCPU snapshots captured before it fail restore with E_CPU_SURFACE and must be recaptured. 1-vCPU snapshots are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index cae746f6f..e06ed4d4a 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit cae746f6f20e956ec0decc7d35971c08e3f12234 +Subproject commit e06ed4d4a0033200b0583a12b76a84dec2c511e1 From 8f062d8e9c7ceac735566b1aa2b2979f0c605c77 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:59:54 -0700 Subject: [PATCH 126/188] doc: say how the snapshot agent detects a failed capture A committed capture terminates the source, so the 0x605 write returns in the source only after a capture without a destination or one that failed before its commit point. The agent tells it from a restore by status bit 1 of 0xea: clear in the source, which keeps its generation, and set in a restored process, whose packet carries the next generation. It then thaws its barriers and runs nvx-time cancel-capture. Reads of 0x605 return 0xff. State this in the snapshot agent section, note that saved values left by an earlier request refuse a capture, add a failed-capture case to the restore matrix, and mention the release in the capture boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/snapshot-and-restore.md | 4 +++- doc/design/time-abi.md | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/doc/design/snapshot-and-restore.md b/doc/design/snapshot-and-restore.md index d0ed2e9ae..bf743c3ca 100644 --- a/doc/design/snapshot-and-restore.md +++ b/doc/design/snapshot-and-restore.md @@ -46,7 +46,9 @@ tier, clone/resume, and fresh/paired scratch combinations are rejected. For paired scratch, the guest snapshot helper first freezes the workload cgroup with a bounded wait, calls `sync`, and freezes the mounted filesystem. Its fresh-scratch mode instead requires scratch to be unmounted. A rejected capture -thaws every guest-owned barrier; failure to thaw terminates the VM. +thaws every guest-owned barrier and then releases the stall-detector +suppression ([snapshot agent](time-abi.md#snapshot-agent)); failure to thaw or +release terminates the VM. 1. gate host input and defer completion of the snapshot-port write; 2. stop the vCPU at the I/O boundary while completing the write, so saved state diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 65d099e1c..a66224829 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1308,15 +1308,23 @@ Before the capture request: `C10`), which do not wait and print nothing; they record `capture_status=ok` in the state file, which the snapshot carries. 2. Save `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` and write 1. + Values that an earlier request saved and never restored refuse the + capture. 3. Debug: save and zero `/proc/sys/kernel/soft_watchdog` and `/proc/sys/kernel/hung_task_timeout_secs`. 4. Apply the existing freezer and scratch barriers. 5. Write the capture request to `0x605`. -If the write returns without a restore (no destination or a rejected -capture), the agent removes the barriers and then restores the values saved -in steps 2 and 3. Otherwise, the restored process runs these steps. Steps 6 -to 10 and 12 are the readiness path: nothing else runs before the +A committed capture terminates the source, so the write returns in the +source VM only when the capture had no destination or failed before its +commit point and OpenVMM resumed the guest. Step 6's status read tells the +cases apart: bit 1 is clear in the source, which has no restore packet and +keeps its generation, and set in a restored process, whose packet carries +the next `g`. Reads of `0x605` return `0xff`; a capture reports no status. +In the source, the agent removes the barriers and then restores the values +saved in steps 2 and 3 (`nvx-time cancel-capture`); if any of these fails, +it powers the guest off with status 1. A restored process runs these steps. +Steps 6 to 10 and 12 are the readiness path: nothing else runs before the acknowledgement. 6. Read the status from `0xea`. Bit 1 is always set after a restore. @@ -1854,6 +1862,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | +| Failed capture: a request without a destination; then, in one VM, a request whose destination the host created after launch (a rejected preflight), the destination's removal, and a second request | One host per backend; CI's `snapshot-core` covers the request without a destination | Each failed request returns in the same VM with status bit 1 clear, `rcu_cpu_stall_suppress` back to its saved value, and `nvx-time status` passing at `generation=0`, with no violation; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From feb5659df375ed123cd59f009177a4538de25b69 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 07:08:41 -0700 Subject: [PATCH 127/188] doc: make the failed-capture case fail after quiesce The orchestrator's p6 case fails the capture after quiesce: the host makes the destination's parent unwritable after launch, so creating the staging directory fails, the error is rollback-safe, and every state unit restarts. Run it on each bare-metal host at 1 and 8 vCPUs, three times each, and follow it with a normal capture after 30 s idle. A destination that already exists is rejected before quiesce, and CI's snapshot-core covers a request without one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index a66224829..6207bb8a5 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1862,7 +1862,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: a request without a destination; then, in one VM, a request whose destination the host created after launch (a rejected preflight), the destination's removal, and a second request | One host per backend; CI's `snapshot-core` covers the request without a destination | Each failed request returns in the same VM with status bit 1 clear, `rcu_cpu_stall_suppress` back to its saved value, and `nvx-time status` passing at `generation=0`, with no violation; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; CI's `snapshot-core` covers a request without a destination, which preflight rejects before quiesce | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From f219cf485ab11803605da8509dd09b0aab2838dd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 07:34:59 -0700 Subject: [PATCH 128/188] doc: say what CI's snapshot-core asserts for the failed-capture case snapshot-core exercises a capture request without a destination, which OpenVMM releases before preflight rather than rejecting it there, and asserts only that the guest continues once and exits 0. Say so instead of claiming the matrix case's full checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6207bb8a5..84d19e583 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1862,7 +1862,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; CI's `snapshot-core` covers a request without a destination, which preflight rejects before quiesce | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; CI's `snapshot-core` exercises a request without a destination, which OpenVMM releases before preflight, and asserts only that the guest continues once and exits 0 | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From b7a38eb1cde9f63590b322db47fab32563b7097a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 07:38:24 -0700 Subject: [PATCH 129/188] doc: correct the L3 fix's cold-boot effect on KVM kvm's A/B at the integration head, on prometheus32 with wiring v7m, shows the L3 fix removing the one-tick slow mode of 2-vCPU cold boots: boots more than 8 ms over the earlier median fall from 9 of 20 to 2 of 20 (mean -6.4 ms, equal medians). 8-vCPU cold boots show no measurable change. The bisect's -7.2 ms at 8 vCPUs came from boots with initcall_debug and an older guest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 84d19e583..af3346064 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1732,7 +1732,7 @@ Expected effects: | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are a + b·(n − 1) ms at n vCPUs, set to the larger of the bare-metal and Azure maxima of wiring v7 on the production kernel, plus headroom. The bare-metal floors, which cover every sample on prometheus28, 30, and 32 with 10 to 30% headroom, are boot 6 + 2, capture 1 + 0.4, and restore 4 + 0.75 on KVM, where nested azure-kvm-5 raises restore to 5 + 1.5 (medians 3.3 to 7.8 ms at 1 to 8 vCPUs and a maximum of 13.1 ms, from first-touch faults on the file-backed restored RAM); 3 + 0.5, 1 + 0.4, and 1.5 + 0.5 on MSHV; and 5 + 1, 1 + 0.4, and 20 + 2 on WHP. The Azure maxima are TBD(kvm, mshv, whp). The budgets are set at 512 MiB, the gate's memory size, which bounds the smaller validation guests: WHP's restore `cpu_us` grows with guest memory (medians 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure), because the checks' first touches fault in lazily registered RAM. Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32 and about +25 ms on azure-kvm-5. A component bisect blames no profile field: OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on most boots. With the L3 shared by the socket the wait is gone (0 of 40 boots), and prometheus32 cold boots change by −4.6 [−10.1, +0.2] ms at 2 vCPUs and −7.2 [−11.1, −2.6] ms at 8. The msr driver's per-CPU hotplug callback (`msr_init`) still waits one or two ticks on multi-vCPU boots, as it did before the profiles | +| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32 and about +25 ms on azure-kvm-5. A component bisect blames no profile field: OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket the wait is gone (0 of 40 boots). At the integration head on prometheus32 with wiring v7m, 2-vCPU cold boots more than 8 ms over the earlier median fall from 9 of 20 to 2 of 20 (mean −6.4 [−11.9, −1.0] ms; the medians tie, because fewer than half of the earlier boots waited), and 8-vCPU cold boots show no measurable change (+3.0 [−1.6, +7.5] ms). The bisect's −7.2 ms at 8 vCPUs came from boots with `initcall_debug` and an older guest. The msr driver's per-CPU hotplug callback (`msr_init`) still waits one or two ticks on multi-vCPU boots, as it did before the profiles | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | From 725985b42970789aa462a3948225b37fa6a80a99 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 07:43:18 -0700 Subject: [PATCH 130/188] ci: check that a released snapshot request restores the guest's state snapshot-core's first VM has no snapshot destination, so OpenVMM releases its snapshot request and the request returns in the source. Before the request, the guest's snapshot agent saves and overrides the stall detectors' settings, and when the request returns it must restore them (nvx-time cancel-capture). The scenario only checked that the guest continued once and that OpenVMM exited 0. After the request, it now also requires that nvx-time status exits 0 with a passing boot line at generation 0 and no restore line, and that rcu_cpu_stall_suppress reads 0. These are the source-side expectations of the spec's failed-capture case in the restore matrix, which covers a capture that fails after quiesce. doc/ci.md describes the check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 8 ++ scripts/nvx_tools/microvm_tests.py | 61 +++++++++++++++ scripts/test_microvm_tests.py | 115 ++++++++++++++++++++++++++++- 3 files changed, 181 insertions(+), 3 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index e13c83908..9b464434e 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -106,6 +106,14 @@ every CPU, a summary with `status=ok`, the requested CPU count, and no failures, and exit status 0; a failure lists each failing check with the guest's detail. The guest command fits on one console line, so the console's echo of it ends before the check prints. +The `snapshot-core` scenario first sends a snapshot request to a guest that +OpenVMM launched without a snapshot destination. OpenVMM releases the request, +so it returns in the source, which must continue exactly once. Before the +request, the guest's snapshot agent saved and overrode the stall detectors' +settings, and it must restore them when the request returns +(`nvx-time cancel-capture`). Afterwards, `nvx-time status` must exit 0 with a +passing boot line at generation 0 and no restore line, and +`/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` must read 0. At the end of a passing run, `test-microvm` prints one `NVX-TIME-ABI-EVIDENCE:` line and adds it to the GitHub job summary, because CI keeps the guest logs only diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index a5640982f..574575293 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -62,6 +62,7 @@ from .time_abi import ( ABI_VERSION, CHECK_CPU_BUDGET_US, + STATUS_TIMEOUT_SECONDS, WARP_PROBE_COMPLETION_MARKER, WARP_PROBE_FAILURE_MARKER, WARP_SUMMARY_PREFIX, @@ -3050,6 +3051,60 @@ def run_restore_memory( ) +CANCELED_CAPTURE_RCU_PREFIX = b"NVX-CANCELED-CAPTURE-RCU " +CANCELED_CAPTURE_DONE_MARKER = b"NVX-CANCELED-CAPTURE-DONE" +RCU_STALL_SUPPRESS_PATH = "/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress" + + +def canceled_capture_checks() -> str: + """Return the guest checks after a snapshot request that OpenVMM released: + the time ABI status, then whether RCU stall detection is still suppressed.""" + rcu = CANCELED_CAPTURE_RCU_PREFIX.decode() + done = CANCELED_CAPTURE_DONE_MARKER.decode() + # A quote pair splits each marker, so the console's echo of the command, + # which the tty may wrap, never contains it. + return ( + status_script() + + f'echo "{rcu[:4]}""{rcu[4:]}$(cat {RCU_STALL_SUPPRESS_PATH})"; ' + + f'echo "{done[:4]}""{done[4:]}"\n' + ) + + +def check_canceled_capture(output: bytes, command: Sequence[str]) -> None: + """Check that a released snapshot request left the source as it was. + + The snapshot agent saves and overrides the stall detectors' settings + before the request; when the request returns in the source, it removes its + barriers and restores the saved values (nvx-time cancel-capture). The + source then reports a passing status at generation 0 with no restore, and + RCU stall detection is no longer suppressed. + """ + context = "after the released snapshot request" + monitor = TimeAbiMonitor(command) + try: + monitor.feed(output) + monitor.finish() + monitor.require_status("its checks finished") + monitor.require_boot("its checks finished") + except TimeAbiFailure as error: + raise RuntimeError(f"{context}: {error}") from error + if monitor.restores: + raise RuntimeError( + f"{context}: nvx-time status reported a restore at " + f"generation={monitor.restores[-1].get('generation')}, but the " + "request returned in the source" + ) + try: + value = _single_marker_value(output, CANCELED_CAPTURE_RCU_PREFIX) + except RuntimeError as error: + raise RuntimeError(f"{context}: {error}") from error + if value != b"0": + raise RuntimeError( + f"{context}: rcu_cpu_stall_suppress is {value.decode(errors='replace')!r}, " + "not 0: the snapshot agent did not restore the value it saved" + ) + + def run_snapshot_core( executable: Path, kernel: Path, @@ -3076,6 +3131,12 @@ def run_snapshot_core( process.wait_for(BOOT_MARKER, timeout) process.send_line("nvx-snapshot; echo NVX-SNAPSHOT-NO-DESTINATION-OK") process.wait_for_line(no_destination_marker, timeout) + checks_start = len(process.output) + process.send_bytes(canceled_capture_checks().encode()) + process.wait_for_line( + CANCELED_CAPTURE_DONE_MARKER, max(timeout, STATUS_TIMEOUT_SECONDS) + ) + check_canceled_capture(process.output[checks_start:], no_destination_command) process.send_line("nvx-exit 0") result = process.wait(timeout) if result.returncode != 0: diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 4cb76ddef..3f4b5c5c0 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -121,6 +121,39 @@ def _restore_status_output( ).encode() +def _canceled_capture_output( + backend: str, + *, + rcu: str | None = "0", + restored: bool = False, + generation: int = 0, + status: int | None = 0, +) -> bytes: + """Return what the guest prints for the checks after a released snapshot + request: the status query's lines, then the RCU suppression flag.""" + rates = f"tsc_hz=2793437000 lapic_hz={time_abi.LAPIC_HZ[backend]}" + lines = [ + f"NVX-TIME-ABI: v=1 phase=boot status=ok cpus=1 {rates} " + f"generation={generation} elapsed_us=2390" + ] + if restored: + lines.append( + f"NVX-TIME-ABI: v=1 phase=restore status=ok cpus=1 {rates} " + "generation=1 elapsed_us=310" + ) + lines.append( + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=0 " + "discontinuities=0 offset_ns=-1200 uncertainty_ns=900 rejected_samples=0 " + "last_sample_error=none" + ) + if status is not None: + lines.append(f"NVX-TIME-STATUS-EXIT status={status}") + if rcu is not None: + lines.append(f"NVX-CANCELED-CAPTURE-RCU {rcu}") + lines.append("NVX-CANCELED-CAPTURE-DONE") + return "".join(f"{line}\r\n" for line in lines).encode() + + def _restore_processors_measure( backend: str, *, @@ -2030,11 +2063,15 @@ def test_smp_worker_requires_bounded_local_timer_progress(self): def test_snapshot_core_waits_for_no_destination_marker_line_before_exit(self): events: list[tuple[str, bytes | str | None]] = [] marker = b"NVX-SNAPSHOT-NO-DESTINATION-OK" + command = ["openvmm", "--hypervisor", "whp", "--kernel", "vmlinux"] class StopAfterNoDestination(Exception): pass class FakeProcess: + def __init__(self) -> None: + self.console = bytearray(b"NVX-SNAPSHOT-NO-DESTINATION-OK\r\n") + def __enter__(self): return self @@ -2046,6 +2083,10 @@ def __exit__( ) -> None: return None + @property + def output(self) -> bytes: + return bytes(self.console) + def wait_for(self, expected: bytes, _timeout: float) -> None: events.append(("wait_for", expected)) @@ -2055,14 +2096,16 @@ def wait_for_line(self, expected: bytes, _timeout: float) -> None: def send_line(self, line: str) -> None: events.append(("send_line", line)) + def send_bytes(self, data: bytes) -> None: + events.append(("send_bytes", data.decode())) + self.console.extend(_canceled_capture_output("whp")) + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: events.append(("wait", None)) return openvmm_process.OpenvmmProcessResult(0, marker + b"\n") with ( - patch.object( - microvm_tests, "workload_boot_command", return_value=["openvmm"] - ), + patch.object(microvm_tests, "workload_boot_command", return_value=command), patch.object(microvm_tests, "OpenvmmProcess", return_value=FakeProcess()), patch.object( microvm_tests.tempfile, @@ -2087,11 +2130,77 @@ def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: ("wait_for", microvm_tests.BOOT_MARKER), ("send_line", "nvx-snapshot; echo NVX-SNAPSHOT-NO-DESTINATION-OK"), ("wait_for_line", marker), + ("send_bytes", microvm_tests.canceled_capture_checks()), + ("wait_for_line", microvm_tests.CANCELED_CAPTURE_DONE_MARKER), ("send_line", "nvx-exit 0"), ("wait", None), ], ) + def test_canceled_capture_checks_query_status_and_rcu_suppression(self): + script = microvm_tests.canceled_capture_checks() + self.assertTrue(script.startswith(time_abi.status_script())) + self.assertIn( + "$(cat /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress)", script + ) + # The command never contains a whole marker, so the console's echo of + # it, wrapped or not, can't stand in for the guest's answers. + for marker in ( + microvm_tests.CANCELED_CAPTURE_RCU_PREFIX, + microvm_tests.CANCELED_CAPTURE_DONE_MARKER, + ): + self.assertNotIn(marker.decode().strip(), script) + # The shell prints both markers whole. + shell = shutil.which("sh") + if shell is None: + self.skipTest("no POSIX shell") + answers = subprocess.run( + [shell, "-c", script.removeprefix(time_abi.status_script())], + capture_output=True, + check=True, + ).stdout + lines = answers.splitlines() + self.assertEqual(lines[-1], microvm_tests.CANCELED_CAPTURE_DONE_MARKER) + self.assertTrue(lines[0].startswith(microvm_tests.CANCELED_CAPTURE_RCU_PREFIX)) + + def test_canceled_capture_requires_generation_zero_and_no_suppression(self): + command = ["openvmm", "--hypervisor", "kvm", "--kernel", "vmlinux"] + echoed = ( + b'~ # echo "NVX-""CANCELED-CAPTURE-RCU $(cat /sys/module/rcupdate/' + b'parameters/rcu_cpu_stall_suppress)"; echo "NVX-""CANCELED-CAPTURE-DONE"\r\n' + ) + microvm_tests.check_canceled_capture( + echoed + _canceled_capture_output("kvm"), command + ) + for output, message in ( + (_canceled_capture_output("kvm", rcu="1"), "rcu_cpu_stall_suppress is '1'"), + ( + _canceled_capture_output("kvm", restored=True), + "reported a restore at generation=1", + ), + ( + _canceled_capture_output("kvm", generation=1), + "generation=1 is not 0 at cold boot", + ), + (_canceled_capture_output("kvm", status=1), "nvx-time status exited 1"), + ( + _canceled_capture_output("kvm", status=None), + "nvx-time status did not finish", + ), + ( + _canceled_capture_output("kvm", rcu=None), + "expected exactly one b'NVX-CANCELED-CAPTURE-RCU ' marker, found 0", + ), + ): + with self.subTest(message=message): + with self.assertRaisesRegex(RuntimeError, re.escape(message)) as raised: + microvm_tests.check_canceled_capture(output, command) + self.assertTrue( + str(raised.exception).startswith( + "after the released snapshot request: " + ) + ) + def test_snapshot_marker_parsers_require_single_well_formed_values(self): output = b"PREFIX-12\r\nPAIR-4-5\n" self.assertEqual( From 6bc6abc85384f9b2fb9518fb003724a7d0833d56 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 08:12:33 -0700 Subject: [PATCH 131/188] doc: widen what CI's snapshot-core asserts for a released capture ci's 04354c44 adds the cancel-path checks to snapshot-core: after the released request, nvx-time status passes at generation 0 with no restore, and rcu_cpu_stall_suppress reads 0. Say so in the failed-capture row, and link to the CI guide's description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index af3346064..57fde5dda 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1862,7 +1862,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; CI's `snapshot-core` exercises a request without a destination, which OpenVMM releases before preflight, and asserts only that the guest continues once and exits 0 | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From 91849f78df74d18254e3c1f2d0ec6fe7c4feba16 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 08:32:20 -0700 Subject: [PATCH 132/188] doc: publish the final cpu_us budgets Set each backend's per-phase budgets from wiring v7m on vmlinux-lockstep: the larger of the bare-metal and Azure maxima, with at least 20% headroom. KVM is 6 + 2, 1 + 0.4, and 5 + 1.75 ms; MSHV is 3 + 0.75, 1 + 0.4, and 2.5 + 0.5; WHP is 20 + 1.5, 1 + 0.4, and 35 + 6, at boot, capture, and restore. WHP's come from 512 MiB guests, because its restore cost grows with memory, and KVM's and MSHV's from the 128 MiB guests that CI and the fleet matrices run. Move the budgets out of the performance table into their own paragraph, with the host stalls that each backend's checks pay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 57fde5dda..51a210e7c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1729,7 +1729,7 @@ Expected effects: | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | | MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On prometheus30 (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first, about 14 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`). Each backend has a `cpu_us` budget per phase (boot, capture, and restore), because WHP's restore checks also pay the first-touch faults of its lazily registered copy-on-write RAM, which the workload would otherwise pay. The fleet matrix driver enforces the budgets during validation; CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The budgets are a + b·(n − 1) ms at n vCPUs, set to the larger of the bare-metal and Azure maxima of wiring v7 on the production kernel, plus headroom. The bare-metal floors, which cover every sample on prometheus28, 30, and 32 with 10 to 30% headroom, are boot 6 + 2, capture 1 + 0.4, and restore 4 + 0.75 on KVM, where nested azure-kvm-5 raises restore to 5 + 1.5 (medians 3.3 to 7.8 ms at 1 to 8 vCPUs and a maximum of 13.1 ms, from first-touch faults on the file-backed restored RAM); 3 + 0.5, 1 + 0.4, and 1.5 + 0.5 on MSHV; and 5 + 1, 1 + 0.4, and 20 + 2 on WHP. The Azure maxima are TBD(kvm, mshv, whp). The budgets are set at 512 MiB, the gate's memory size, which bounds the smaller validation guests: WHP's restore `cpu_us` grows with guest memory (medians 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure), because the checks' first touches fault in lazily registered RAM. Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | | The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32 and about +25 ms on azure-kvm-5. A component bisect blames no profile field: OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket the wait is gone (0 of 40 boots). At the integration head on prometheus32 with wiring v7m, 2-vCPU cold boots more than 8 ms over the earlier median fall from 9 of 20 to 2 of 20 (mean −6.4 [−11.9, −1.0] ms; the medians tie, because fewer than half of the earlier boots waited), and 8-vCPU cold boots show no measurable change (+3.0 [−1.6, +7.5] ms). The bisect's −7.2 ms at 8 vCPUs came from boots with `initcall_debug` and an older guest. The msr driver's per-CPU hotplug callback (`msr_init`) still waits one or two ticks on multi-vCPU boots, as it did before the profiles | @@ -1739,6 +1739,39 @@ Expected effects: Expected wins are tracked separately and do not relax the gate. +**`cpu_us` budgets.** Each backend budgets the CPU time of each phase's +checks (boot, capture, and restore) at a + b·(n − 1) ms for n online vCPUs, +per sample. A budget is the larger of the bare-metal and Azure maxima of +wiring v7m on the production kernel (`vmlinux-lockstep`), with at least +20% headroom: + +| Backend | Boot | Capture | Restore | Measured on | +| --- | --- | --- | --- | --- | +| KVM | 6 + 2 | 1 + 0.4 | 5 + 1.75 | prometheus32 and `azure-kvm-5`, 128 MiB | +| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 MiB | +| WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | prometheus28 and the 8370C and 8573C runners, 512 MiB | + +The fleet matrix driver enforces the budgets during validation. CI reports +each phase against its budget (`_cpu_over_budget`) but does not gate +on it, because the A/B gate above covers latency. The checks' CPU time also +counts the host stalls of their first touches of restored RAM: + +- WHP's restore checks fault in its lazily registered copy-on-write RAM, + which the workload would otherwise do. Their cost grows with guest memory: + medians are 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and + 8 vCPUs on Azure. So WHP's budgets come from 512 MiB guests, which bound + smaller ones. Its Azure maxima are 17.03 ms at boot with 2 vCPUs and + 42.20 ms at restore with 4. +- On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the + toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The + restore budget leaves room for one more stall above every maximum. +- On `azure-kvm-5`, KVM's restore checks fault in the file-backed restored + RAM 4 KiB at a time: medians of 3.3 to 7.8 ms at 1 to 8 vCPUs, and a + maximum of 13.14 ms. + +KVM's and MSHV's budgets come from 128 MiB guests, the size that CI and the +fleet matrices run. + **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), measured on prometheus32 (KVM) with the kvm agent's attribution harness; the From 57128b353dd39563f990f1baa3c6bb09c48c6557 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 08:34:35 -0700 Subject: [PATCH 133/188] ci: set the time ABI checks' CPU budgets to the spec's final values The spec published its final cpu_us budgets (doc/design/time-abi.md, "cpu_us budgets"): the larger of the bare-metal and Azure maxima of v7m on vmlinux-lockstep, with at least 20% headroom, held per sample at a + b * (n - 1) for n online vCPUs. CHECK_CPU_BUDGET_US now holds them, in microseconds: - KVM: boot (6000, 2000), capture (1000, 400), restore (5000, 1750); - MSHV: boot (3000, 750), capture (1000, 400), restore (2500, 500); - WHP: boot (20000, 1500), capture (1000, 400), restore (35000, 6000). Every budget is at least the bare-metal floor it replaces. The evidence line's over-budget counts stay report-only: CI gates on neither cpu_us nor elapsed_us. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/nvx_tools/time_abi.py | 24 ++++++++++++------------ scripts/test_time_abi.py | 12 +++++++++--- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 48b28d71a..09196d09e 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -33,21 +33,21 @@ } MIN_TSC_HZ = 500_000_000 MAX_TSC_HZ = 10_000_000_000 -# doc/design/time-abi.md, "Performance": each backend has a CPU-time -# (cpu_us) budget per phase (boot, capture, and restore): a base plus an -# increment per additional online CPU, in microseconds. These are the spec's -# bare-metal floors from the guest's v7 measurements; its final values are the -# larger of those and the Azure maxima. This table is the one place the -# harness reads them. The checks' wall time (elapsed_us) has no budget. CI -# reports both and gates on neither: the performance gate is the A/B -# comparison outside CI. +# doc/design/time-abi.md, "Performance expectations and acceptance gate", +# "cpu_us budgets": each backend has a CPU-time (cpu_us) budget per phase +# (boot, capture, and restore): a base plus an increment per additional online +# CPU, in microseconds, held per sample. These are the spec's final values: the +# larger of the bare-metal and Azure maxima of v7m on vmlinux-lockstep, with at +# least 20% headroom. This table is the one place the harness reads them. The +# checks' wall time (elapsed_us) has no budget. CI reports both and gates on +# neither: the performance gate is the A/B comparison outside CI. CHECK_CPU_BUDGET_US: Mapping[str, Mapping[str, tuple[int, int]]] = { - "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (5_000, 1_500)}, - "mshv": {"boot": (3_000, 500), "capture": (1_000, 400), "restore": (1_500, 500)}, + "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (5_000, 1_750)}, + "mshv": {"boot": (3_000, 750), "capture": (1_000, 400), "restore": (2_500, 500)}, "whp": { - "boot": (5_000, 1_000), + "boot": (20_000, 1_500), "capture": (1_000, 400), - "restore": (20_000, 2_000), + "restore": (35_000, 6_000), }, } WARP_BOUND_NS = 1000 diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index d67d6da62..ba17c79e6 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -136,7 +136,7 @@ def test_names_the_catalog_profiles(self): ) def test_computes_the_checks_cpu_time_budget(self): - # The spec's bare-metal floors: a base plus an increment per additional + # The spec's final budgets: a base plus an increment per additional # CPU, one budget per backend and phase. for backend in ("kvm", "mshv", "whp"): self.assertEqual( @@ -144,9 +144,15 @@ def test_computes_the_checks_cpu_time_budget(self): {"boot", "capture", "restore"}, ) self.assertEqual(time_abi.check_cpu_budget_us("kvm", "boot", 1), 6_000) - self.assertEqual(time_abi.check_cpu_budget_us("mshv", "restore", 8), 5_000) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 8), 17_250) + self.assertEqual(time_abi.check_cpu_budget_us("mshv", "boot", 4), 5_250) + self.assertEqual(time_abi.check_cpu_budget_us("mshv", "restore", 8), 6_000) self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 2), 1_400) - self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 1), 20_000) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 1), 35_000) + # The spec's binding WHP cells: boot at 2 vCPUs (17.03 ms measured) + # and restore at 4 (42.20 ms). + self.assertEqual(time_abi.check_cpu_budget_us("whp", "boot", 2), 21_500) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 4), 53_000) self.assertIsNone(time_abi.check_cpu_budget_us("hvf", "boot", 1)) self.assertIsNone(time_abi.check_cpu_budget_us("kvm", "runtime", 1)) self.assertIsNone(time_abi.check_cpu_budget_us("kvm", "boot", 0)) From 92b48b5af8cebc2aa4a2c70467e4636e489f8635 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 08:57:41 -0700 Subject: [PATCH 134/188] doc: correct MSHV's registered CPUID count after the L3 fix With the L3 shared by the socket, VP 0's own view on MSHV already presents leaf 4's L3 entry, so selective registration leaves it native: MSHV registers 11 of 61 entries on prometheus30 and 12 of 65 on azure-azlinux-5 at every vCPU count, the entries the row lists. The earlier 12 and 13 were the pre-fix counts at two or more vCPUs, which registered the private L3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 51a210e7c..3ef00a22a 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -268,7 +268,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity leaves, while the explicit zero leaves already read zero in VP 0's view and get none (see the next row); preflight reads every entry back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`) | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 12 of 61 entries on prometheus30 (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 13 of 65 on azure-azlinux-5 (the same plus invariant TSC); registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 11 of 61 entries on prometheus30 (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 12 of 65 on azure-azlinux-5 (the same plus invariant TSC), at every vCPU count, because VP 0's own view already presents leaf 4's shared L3; registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | From 5d8281f59c7d8222f9f136094191969abab266aa Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:07:09 -0700 Subject: [PATCH 135/188] doc: correct the bare-metal MSHV CPU check time mshv's logs sum the time ABI CPU checks of a one-vCPU restore on prometheus30 to about 153 us (recorded 29.5, presented 92, support 31), so about 0.15 ms rather than 0.16 ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 3ef00a22a..60617113d 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1838,7 +1838,7 @@ after the acknowledgement, outside these phases. `nvx-time status` reports checks. On every cold boot and restore, the worker logs `time ABI CPU checks passed` with `recorded_cpuid_us`, `presented_cpuid_us`, and `profile_support_us`. -Their sum per restore at one vCPU is about 0.16 ms on bare-metal MSHV and +Their sum per restore at one vCPU is about 0.15 ms on bare-metal MSHV and 0.12 ms nested, and 0.87 ms on bare-metal WHP and 1.6 to 1.9 ms nested, about 95% of it WHP's per-leaf read-back. OpenVMM's comparison and unlisted-entry check take 40 to 66 µs of each. From fb934576ca9357bae880aa66ed3985098c9a04b5 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:16:58 -0700 Subject: [PATCH 136/188] doc: record that the L3 fix ends the KVM multi-vCPU cold-boot regression kvm's finished bisect on prometheus32 compares the head with the L3 fix against the pre-profile head: -0.3, -0.0, and -1.0 ms at 2, 4, and 8 vCPUs, all within noise, against +9.2 ms at 2 vCPUs before the fix. So the private L3 was the whole regression. msr_init's tick wait predates the profiles: it competes with the asynchronous initramfs unpack, off the critical path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 60617113d..c30f9b2bb 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1732,7 +1732,7 @@ Expected effects: | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | Slower multi-vCPU cold boots, not gated: +8.0 [4.5, 15.5] ms at 2 vCPUs on prometheus32 and about +25 ms on azure-kvm-5. A component bisect blames no profile field: OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket the wait is gone (0 of 40 boots). At the integration head on prometheus32 with wiring v7m, 2-vCPU cold boots more than 8 ms over the earlier median fall from 9 of 20 to 2 of 20 (mean −6.4 [−11.9, −1.0] ms; the medians tie, because fewer than half of the earlier boots waited), and 8-vCPU cold boots show no measurable change (+3.0 [−1.6, +7.5] ms). The bisect's −7.2 ms at 8 vCPUs came from boots with `initcall_debug` and an older guest. The msr driver's per-CPU hotplug callback (`msr_init`) still waits one or two ticks on multi-vCPU boots, as it did before the profiles | +| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on prometheus32 against the pre-profile head, and about +25 ms on azure-kvm-5. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, prometheus32 cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate measures azure-kvm-5. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | From 32849acb73dcdc1a1a1ea9d1062e30fba5f272e1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:37:10 -0700 Subject: [PATCH 137/188] doc: raise KVM's restore cpu_us budget to 6.5 + 1.5 kvm's pooled azure-kvm-5 data (v7m on vmlinux-lockstep, 72 restores) has a 1-vCPU restore at 5.15 ms, over 5 + 1.75. Set KVM's restore budget to 6.5 + 1.5 ms (6.5, 8, 11, and 17 ms at 1, 2, 4, and 8 vCPUs), which keeps at least 26% headroom over every maximum, and update the azure-kvm-5 medians and maxima. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c30f9b2bb..9da049982 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1747,7 +1747,7 @@ wiring v7m on the production kernel (`vmlinux-lockstep`), with at least | Backend | Boot | Capture | Restore | Measured on | | --- | --- | --- | --- | --- | -| KVM | 6 + 2 | 1 + 0.4 | 5 + 1.75 | prometheus32 and `azure-kvm-5`, 128 MiB | +| KVM | 6 + 2 | 1 + 0.4 | 6.5 + 1.5 | prometheus32 and `azure-kvm-5`, 128 MiB | | MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 MiB | | WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | prometheus28 and the 8370C and 8573C runners, 512 MiB | @@ -1766,8 +1766,8 @@ counts the host stalls of their first touches of restored RAM: toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The restore budget leaves room for one more stall above every maximum. - On `azure-kvm-5`, KVM's restore checks fault in the file-backed restored - RAM 4 KiB at a time: medians of 3.3 to 7.8 ms at 1 to 8 vCPUs, and a - maximum of 13.14 ms. + RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs, and maxima + of 5.15 ms at 1 vCPU and 13.14 ms at 8, over 72 restores. KVM's and MSHV's budgets come from 128 MiB guests, the size that CI and the fleet matrices run. From c4f04995dfa339bf679140ac996288cb46b59e94 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:39:26 -0700 Subject: [PATCH 138/188] ci: raise KVM's restore CPU budget to the spec's corrected 6.5 + 1.5 ms The spec corrected one cell of its final cpu_us budgets: KVM restore is 6.5 + 1.5 ms (6.5, 8, 11, and 17 ms at 1, 2, 4, and 8 vCPUs). azure-kvm-5's pooled v7m data has a 1-vCPU restore at 5.15 ms, over the published 5 + 1.75, and the new value keeps at least 26% headroom over every maximum. CHECK_CPU_BUDGET_US's KVM restore entry is now (6500, 1500); the other budgets are unchanged. The evidence test took its over-budget restore sample from the old 5 ms budget; it now derives the sample and the larger per-phase budget from the table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/nvx_tools/time_abi.py | 2 +- scripts/test_microvm_tests.py | 20 +++++++++++++------- scripts/test_time_abi.py | 3 ++- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 09196d09e..d8d6bd32e 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -42,7 +42,7 @@ # checks' wall time (elapsed_us) has no budget. CI reports both and gates on # neither: the performance gate is the A/B comparison outside CI. CHECK_CPU_BUDGET_US: Mapping[str, Mapping[str, tuple[int, int]]] = { - "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (5_000, 1_750)}, + "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (6_500, 1_500)}, "mshv": {"boot": (3_000, 750), "capture": (1_000, 400), "restore": (2_500, 500)}, "whp": { "boot": (20_000, 1_500), diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 3f4b5c5c0..58c7b9841 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3873,29 +3873,35 @@ def query(phase: str, cpus: int, cpu_us: int | None) -> str: with tempfile.TemporaryDirectory() as temporary: output_dir = Path(temporary) - # Within KVM's budget (20 ms for a boot at 8 CPUs), over it (5 ms - # for a restore at 1 CPU), and an older image without cpu_us, which - # only reports elapsed_us. + # KVM's restore budget at 1 CPU is its base. + restore_budget = time_abi.CHECK_CPU_BUDGET_US["kvm"]["restore"][0] + over = restore_budget + 500 + # Within KVM's budget (20 ms for a boot at 8 CPUs), over it for a + # restore at 1 CPU, and an older image without cpu_us, which only + # reports elapsed_us. (output_dir / "smp-8.log").write_text(query("boot", 8, 4600)) - (output_dir / "restore-1.log").write_text(query("restore", 1, 5500)) + (output_dir / "restore-1.log").write_text(query("restore", 1, over)) (output_dir / "restore-8.log").write_text(query("restore", 8, 2000)) (output_dir / "older.log").write_text(query("boot", 1, None)) evidence = microvm_tests.time_abi_evidence(output_dir, "kvm") unknown = microvm_tests.time_abi_evidence(output_dir) # Each phase has its own budget: a larger restore budget absorbs # the 1-CPU restore. - phased = {**time_abi.CHECK_CPU_BUDGET_US["kvm"], "restore": (6_000, 0)} + phased = { + **time_abi.CHECK_CPU_BUDGET_US["kvm"], + "restore": (over + 500, 0), + } with patch.dict(time_abi.CHECK_CPU_BUDGET_US, {"kvm": phased}): per_phase = microvm_tests.time_abi_evidence(output_dir, "kvm") self.assertEqual(evidence["boot_markers"], "2") self.assertEqual(evidence["boot_cpu_us"], "4600-4600") self.assertEqual(evidence["boot_cpu_over_budget"], "0") - self.assertEqual(evidence["restore_cpu_us"], "2000-5500") + self.assertEqual(evidence["restore_cpu_us"], f"2000-{over}") self.assertEqual(evidence["restore_cpu_over_budget"], "1") self.assertEqual(per_phase["restore_cpu_over_budget"], "0") self.assertEqual(per_phase["boot_cpu_over_budget"], "0") # Without a backend there is no budget to compare with. - self.assertEqual(unknown["restore_cpu_us"], "2000-5500") + self.assertEqual(unknown["restore_cpu_us"], f"2000-{over}") self.assertNotIn("restore_cpu_over_budget", unknown) def test_time_abi_evidence_line_goes_to_the_log_and_the_job_summary(self): diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index ba17c79e6..9cb6d9d87 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -144,7 +144,8 @@ def test_computes_the_checks_cpu_time_budget(self): {"boot", "capture", "restore"}, ) self.assertEqual(time_abi.check_cpu_budget_us("kvm", "boot", 1), 6_000) - self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 8), 17_250) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 1), 6_500) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 8), 17_000) self.assertEqual(time_abi.check_cpu_budget_us("mshv", "boot", 4), 5_250) self.assertEqual(time_abi.check_cpu_budget_us("mshv", "restore", 8), 6_000) self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 2), 1_400) From 9285f2575c7b3f87cd4ad80bad4aa96e38dc5aa0 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:45:02 -0700 Subject: [PATCH 139/188] doc: say how the wall-clock discipline converges whp's p6 run on an Azure WHP runner shows the guest settling 1.76 ms behind host UTC after a restore while the PLL's frequency integrator absorbs the host UTC clock's 6.88 ppm rate against its TSC, which takes about an hour at the 64 s cadence. Restore repair corrects only the TSC's rate deviation. State this, with the steady offset of about 256 s times the rate, which stays far below the 128 ms step threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 9da049982..82bbc3c4b 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1451,6 +1451,14 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: `esterror = ceil(epsilon / 1000)` µs. - **Bounds.** The kernel limits the frequency to ±500 ppm; the declared rate tolerance uses at most 250 ppm of it. +- **Convergence.** The boot check and restore repair step the clock, and + restore repair corrects the TSC's rate deviation. Neither corrects the rate + at which the host's own UTC clock runs against its TSC, which host time + synchronization slews by a few ppm. The PLL's frequency integrator absorbs + that rate over about an hour at the 64 s cadence. Until then the offset + settles near 256 s times the rate: 1.76 ms behind host UTC at the 6.88 ppm + of an Azure WHP runner, and an expected 4.2 ms at prometheus28's 16.5 ppm. + Both are far below the step threshold. The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. From 21503eebf73cda4854f6ea5d215de29b3845994c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 09:47:51 -0700 Subject: [PATCH 140/188] doc: say that MSHV's cpu_us budgets hold at 512 MiB too mshv's 512 MiB runs on prometheus30 and azure-azlinux-5 (v7m on vmlinux-lockstep) fit MSHV's row with room to spare: the tightest is boot at 4 vCPUs, 3.65 of 5.25 ms. MSHV registers all guest RAM before the restore, and the checks pay only for the 2 MiB chunks they first touch, so unlike WHP the cost doesn't grow with memory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 82bbc3c4b..502cd6431 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1756,7 +1756,7 @@ wiring v7m on the production kernel (`vmlinux-lockstep`), with at least | Backend | Boot | Capture | Restore | Measured on | | --- | --- | --- | --- | --- | | KVM | 6 + 2 | 1 + 0.4 | 6.5 + 1.5 | prometheus32 and `azure-kvm-5`, 128 MiB | -| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 MiB | +| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 and 512 MiB | | WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | prometheus28 and the 8370C and 8573C runners, 512 MiB | The fleet matrix driver enforces the budgets during validation. CI reports @@ -1777,8 +1777,10 @@ counts the host stalls of their first touches of restored RAM: RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs, and maxima of 5.15 ms at 1 vCPU and 13.14 ms at 8, over 72 restores. -KVM's and MSHV's budgets come from 128 MiB guests, the size that CI and the -fleet matrices run. +KVM's budgets come from 128 MiB guests, the size that CI and the fleet +matrices run. MSHV's hold at 128 and 512 MiB alike: MSHV registers all +guest RAM before the restore, and the checks pay only for the chunks they +touch, which doesn't depend on the guest's memory size. **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), From 3d6f17dc559bb5c6cf0a7485ebd9d863d62d7004 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 10:04:45 -0700 Subject: [PATCH 141/188] doc: quote the start-up budget at the integration head kvm's attribution run on prometheus32 compares e06ed4d4a with the pre-profile e7ec0ca6c: up to the backend preflight at one vCPU, a restore starts 0.32 ms and a cold boot 0.30 ms sooner. The CPU checks add about 56 us, and every 90% upper bound is within +0.31 ms. Quote these instead of the 31a5a04f9 figures, and note that process start moves milestones by about 0.25 ms between runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 502cd6431..2c6ea91e7 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1785,10 +1785,12 @@ touch, which doesn't depend on the guest's memory size. **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), measured on prometheus32 (KVM) with the kvm agent's attribution harness; the -flip requires it. At `31a5a04f9`, up to the backend preflight at one vCPU, -a restore starts 0.46 ms and a cold boot 0.28 ms faster than `e7ec0ca6c`; -the only added work is the worker's time ABI CPU checks, 40 to 60 µs over -the pre-profile path, and every 90% upper bound is within +0.30 ms. +flip requires it. At the integration head (`e06ed4d4a`), up to the backend +preflight at one vCPU, a restore starts 0.32 ms and a cold boot 0.30 ms +faster than `e7ec0ca6c`. The only added work is the worker's time ABI CPU +checks, about 56 µs over the pre-profile path, and every 90% upper bound is +within +0.31 ms. Process start moves each binary's milestones by about +0.25 ms from run to run, so only differences within one run count. Neither path encodes, decodes, or hashes a profile document or an effective CPUID: From 22b838eabd9dc5644a2c577de95096877f6551e3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 10:06:32 -0700 Subject: [PATCH 142/188] doc: say that fleet validation checks every recorded cpu_us sample The orchestrator's p6 plan records cpu_us on every sample and applies the budgets to the recorded data afterwards, so the matrix driver is report-only during the runs. Describe the check as fleet validation checking every recorded sample against the budgets, instead of the driver enforcing them while it runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 2c6ea91e7..72a3459db 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1759,10 +1759,11 @@ wiring v7m on the production kernel (`vmlinux-lockstep`), with at least | MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 and 512 MiB | | WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | prometheus28 and the 8370C and 8573C runners, 512 MiB | -The fleet matrix driver enforces the budgets during validation. CI reports -each phase against its budget (`_cpu_over_budget`) but does not gate -on it, because the A/B gate above covers latency. The checks' CPU time also -counts the host stalls of their first touches of restored RAM: +Fleet validation records every sample's `cpu_us` and checks each one +against these budgets. CI reports each phase against its budget +(`_cpu_over_budget`) but does not gate on it, because the A/B gate +above covers latency. The checks' CPU time also counts the host stalls of +their first touches of restored RAM: - WHP's restore checks fault in its lazily registered copy-on-write RAM, which the workload would otherwise do. Their cost grows with guest memory: @@ -1874,9 +1875,10 @@ matrix driver run `/sbin/nvx-time status` over the console after shell-ready and after every restore. They require exit status 0 and, after a restore, a `phase=restore` line with `status=ok`, the restored `generation`, and `cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with -`generation=0`. During fleet validation (`p6`), the matrix driver also -requires each phase's `cpu_us` to be within the backend's budget for that -phase in [Performance expectations](#performance-expectations-and-acceptance-gate); +`generation=0`. During fleet validation (`p6`), the matrix driver records +each phase's `cpu_us`, and every recorded sample must be within the +backend's budget for that phase in +[Performance expectations](#performance-expectations-and-acceptance-gate); CI reports it without gating on it. The fleet runs them on the hosts our SSH account can use, which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 From d365de0ea62595734e88f7890f273529ccb6dfe7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 10:52:20 -0700 Subject: [PATCH 143/188] doc: give MSHV's per-processor creation cost precisely mshv's VP-creation trace at 8 vCPUs shows the 90-page pre-deposit covering the BSP and the first application processor. The second needs about 33 more one-page deposits (about 6 ms), and each later one 82 to 84 (14 to 15 ms). Say so instead of about 14 ms for every processor after the first; the 21 and 78 ms totals at 4 and 8 vCPUs are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 72a3459db..4d45eee36 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1736,7 +1736,7 @@ Expected effects: | `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | -| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On prometheus30 (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first, about 14 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | +| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On prometheus30 (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first: the second takes about 6 ms and each later one 14 to 15 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | From 81d77f470bc651660acd8fa7a32f7a7999cd22e9 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 11:37:43 -0700 Subject: [PATCH 144/188] doc: record the guest's RTC write and correct the packet's exit count perf's p6 measurements show that the discipline's clear STA_UNSYNC makes the guest kernel write the CMOS RTC within a second of each clock step and every 11 minutes, 26 port exits per write. v1 keeps the clock reported as synchronized, so record the write as an accepted cost. They also show that the gate's base reads no restore packet in its untiered restores, so there packet v4 adds 25 port exits instead of saving any; say so. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 4d45eee36..09c196cba 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1448,7 +1448,11 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: nanoseconds, `status = STA_PLL | STA_NANO` (clearing `STA_UNSYNC`), `constant = 4` at the 16 s cadence or 6 at the 64 s cadence, `maxerror = ceil((|theta| + epsilon) / 1000)` µs, and - `esterror = ceil(epsilon / 1000)` µs. + `esterror = ceil(epsilon / 1000)` µs. A clear `STA_UNSYNC` reports the + clock to applications as synchronized, with these error bounds. It also + makes the kernel write the CMOS RTC within a second and then every + 11 minutes, which v1 accepts (see + [Performance expectations](#performance-expectations-and-acceptance-gate)). - **Bounds.** The kernel limits the frequency to ±500 ppm; the declared rate tolerance uses at most 250 ppm of it. - **Convergence.** The boot check and restore repair step the clock, and @@ -1730,8 +1734,9 @@ Expected effects: | Change | Expected effect | | --- | --- | | WHP restored SMP without RDTSC emulation | Removes the 1-to-2 vCPU restore jump (120.7 ms to 165.9 ms p50): SMP restores measured 8 to 18 ms faster in the spike and 14 to 31 ms faster on 8370C runners at 2 to 8 vCPUs. A guest timestamp read after an SMP restore drops from 41 to 73 µs to 10 to 16 ns | -| Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits, most visible on WHP | -| Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore | +| Packet v4 with four-byte reads instead of 83 or more byte reads | Fewer restore exits than reading packets v1 to v3 a byte at a time. The gate's base reads no packet in its untiered restores, so there the time ABI adds 25 port exits per restore (59 against 34 on KVM): the selector write and 24 four-byte reads | +| Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore; the gate's base skips the RTC reads in its untiered restores | +| The kernel's RTC write | Because the discipline clears `STA_UNSYNC`, the guest kernel writes the CMOS RTC within a second of each clock step at boot and restore and then every 11 minutes: 26 port exits per write, about 0.1 ms on KVM and more where port exits cost more. Keeping `STA_UNSYNC` set would avoid it but report `TIME_ERROR` to applications | | No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | | `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | From b459e5654a080e31fb382a5061818ea0343cb520 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 11:40:21 -0700 Subject: [PATCH 145/188] doc: say which failed captures carry a time ABI code Only a capture that a time ABI check rejects logs a time ABI code. A capture that fails for another reason before its commit point, such as a failed publication, also rolls back but logs no code, which the matrix's failed-capture case relies on. State this, name the rollback log lines in that case, and list the debug kernel's watchdog values among the saved values it compares. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 09c196cba..c51db2ce1 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1510,8 +1510,11 @@ the process's fatal-error line leads with the first code in the chain, `fatal error: [E_TSC_RATE_TOLERANCE] `, followed by the full cause chain. A rejected cold boot or restore exits the OpenVMM process with status 1, the existing fatal-error status; tests and orchestrators match -the bracketed code at the start of that line. A rejected capture is -rollback-safe: the guest continues, and OpenVMM logs the code. +the bracketed code at the start of that line. A capture that a time ABI +check rejects is rollback-safe: the guest continues, and OpenVMM logs the +code. A capture that fails for another reason before its commit point, such +as a failed publication, also rolls back, and its log carries no time ABI +code. | VMM code | Condition | Detected at | | --- | --- | --- | @@ -1914,7 +1917,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values restored (`rcu_cpu_stall_suppress` 0 and no saved-values file), and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From facd91e101dc0fdd4a617c08221226e9305dc767 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 17:38:39 -0700 Subject: [PATCH 146/188] guest: make nvx-exit a single static exec nvx-exit stops the VM by writing its exit code to OpenVMM's control port 0x604. It was a shell script that ran sh, a command substitution, and a printf | dd pipeline. After a restore, the guest has started no process before it reports ready, so its first command pays a cold first touch for every fork, exec and exit. That first command is usually nvx-exit, and the cost showed in the WHP restore's guest-exit teardown. A single exec measured 6.3 to 6.7 ms less there. It offsets the cost rather than removing it: whatever runs first after a restore still pays its own first touches. guest/common/nvx-exit.c replaces the script, with the same CLI and semantics in one exec: - nvx-exit [CODE]: CODE defaults to 1, and any CODE that isn't a decimal number from 0 to 255 becomes 1; - one pwrite() of one byte to /dev/port at offset 0x604; - no output; if the write fails, it exits with status 1, as the script's dd did with its stderr suppressed. The one change is that a leading zero no longer makes the shell's printf read the code as octal: 010 now sends 10, not 8. No caller passes one. It's built like nvx-time, as a static musl helper (musl-gcc -static -Os -s), and the binary is 13,376 bytes. Every caller runs it as a command with one argument: init, nvx-init-agent, nvx-snapshot, nvx-virtio-restore-probe, and the benchmark and microVM test scripts. nvx-managed-agent.c already exec()s /sbin/nvx-exit. So none depends on it being a script. The shellcheck and shfmt lists in check-quality, doc/setup.md and the code-improvement workflows drop the script. The four compiled lock files change only in those lists and their frontmatter hash. A new test builds the source against a regular file in place of /dev/port. It checks the byte, the offset, the argument handling, and the silent failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/check-quality/action.yml | 4 +- .github/workflows/code-deduplication.lock.yml | 8 +- .github/workflows/code-documentation.lock.yml | 8 +- .github/workflows/code-quality.lock.yml | 8 +- .github/workflows/code-reusability.lock.yml | 8 +- .github/workflows/shared/code-improvement.md | 2 - doc/guests/ubuntu-guest.md | 1 - doc/setup.md | 6 +- docker/Dockerfile | 5 +- guest/common/nvx-exit | 11 --- guest/common/nvx-exit.c | 46 ++++++++++ scripts/nvx_tools/build.py | 1 - scripts/nvx_tools/build_constants.py | 7 +- scripts/test_nvx_tools.py | 86 ++++++++++++++++++- 14 files changed, 159 insertions(+), 42 deletions(-) delete mode 100644 guest/common/nvx-exit create mode 100644 guest/common/nvx-exit.c diff --git a/.github/actions/check-quality/action.yml b/.github/actions/check-quality/action.yml index 0c1421913..9f735e8eb 100644 --- a/.github/actions/check-quality/action.yml +++ b/.github/actions/check-quality/action.yml @@ -33,7 +33,7 @@ runs: python -m compileall -q .github/specula ${SHELLCHECK} --shell=sh \ guest/common/init guest/alpine/nvx-container-enter \ - guest/alpine/nvx-container-launch guest/common/nvx-exit \ + guest/alpine/nvx-container-launch \ guest/common/nvx-hostmount guest/common/nvx-identity-probe \ guest/common/nvx-init-agent \ guest/common/nvx-container-enter-azurelinux \ @@ -49,7 +49,7 @@ runs: python -m ruff format --check scripts .github/specula ${SHFMT} -d -ln posix -i 4 -ci \ guest/common/init guest/alpine/nvx-container-enter \ - guest/alpine/nvx-container-launch guest/common/nvx-exit \ + guest/alpine/nvx-container-launch \ guest/common/nvx-hostmount guest/common/nvx-identity-probe \ guest/common/nvx-init-agent \ guest/common/nvx-container-enter-azurelinux \ diff --git a/.github/workflows/code-deduplication.lock.yml b/.github/workflows/code-deduplication.lock.yml index 9a5140213..dfa897eb7 100644 --- a/.github/workflows/code-deduplication.lock.yml +++ b/.github/workflows/code-deduplication.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"1d4d51a0229c7025e6eb447470e099548d6ad6c829905d4bee2937037c481e6c","body_hash":"e0b1c993ac7d18e879c5e30e28ededa5778e24274799365a492379c0e0b68a1a","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"51f23ce614b5aeb020af1081a771081a0b7dada8264ef9a10925b845b54159ff","body_hash":"e0b1c993ac7d18e879c5e30e28ededa5778e24274799365a492379c0e0b68a1a","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_CI_TRIGGER_TOKEN","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/setup-python","sha":"ece7cb06caefa5fff74198d8649806c4678c61a1","version":"v6"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"5e508589e03a7757a7e05b26e834292f5445bfb6","version":"v0.88.7"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14","digest":"sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14@sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"safeoutputs","tools":["create_pull_request","missing_data","missing_tool","noop"]}]} # This file was automatically generated by gh-aw (v0.88.7). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -519,7 +519,7 @@ jobs: REPO: ${{ github.repository }} WORKFLOW_ID: code-deduplication name: Validate the baseline and prepare bounded context - run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-exit\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" + run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" - name: Configure Git credentials env: @@ -588,7 +588,7 @@ jobs: env: GH_AW_FILE_ROOT: "${{ runner.temp }}/gh-aw" GH_AW_FILE_CONFIG: "{\"files\":[{\"path\":\"safeoutputs/config.json\",\"content_env\":\"GH_AW_SAFE_OUTPUTS_CONFIG\"}]}" - GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-deduplication/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-deduplication] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-deduplication/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-deduplication] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" with: script: | const path = require('path'); @@ -2337,7 +2337,7 @@ jobs: GH_AW_ALLOWED_DOMAINS: "*.docker.com,*.docker.io,*.githubusercontent.com,*.pythonhosted.org,anaconda.org,api.snapcraft.io,archive.ubuntu.com,auth.docker.io,azure.archive.ubuntu.com,binstar.org,bootstrap.pypa.io,codeload.github.com,conda.anaconda.org,conda.binstar.org,crates.io,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,dl.k8s.io,docs.github.com,files.pythonhosted.org,gcr.io,ghcr.io,github-cloud.githubusercontent.com,github-cloud.s3.amazonaws.com,github.blog,github.com,github.githubassets.com,index.crates.io,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,lfs.github.com,mcr.microsoft.com,objects.githubusercontent.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,patch-diff.githubusercontent.com,patchdiff.githubusercontent.com,pip.pypa.io,pkgs.k8s.io,ppa.launchpad.net,production.cloudflare.docker.com,pypi.org,pypi.python.org,quay.io,raw.githubusercontent.com,registry.hub.docker.com,repo.anaconda.com,repo.continuum.io,s.symcb.com,s.symcd.com,security.ubuntu.com,sh.rustup.rs,static.crates.io,static.rust-lang.org,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-deduplication/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-deduplication] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-deduplication/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-deduplication] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" GH_AW_CI_TRIGGER_TOKEN: ${{ secrets.GH_AW_CI_TRIGGER_TOKEN }} with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/code-documentation.lock.yml b/.github/workflows/code-documentation.lock.yml index 790586bc1..ec2e4b859 100644 --- a/.github/workflows/code-documentation.lock.yml +++ b/.github/workflows/code-documentation.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a0d970a08c9551be2b095d100604dfa0890fe6108cb6e17ddde0f7ca01fa1a5d","body_hash":"a595285ca0a68ff789b1bdeba56c49c74dff904695ca3e0f40b810e3d35515d6","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"f2a4c2f06ce008e11467cad0e3dc09081a0b1e3abf4356267b15acf6e917590a","body_hash":"a595285ca0a68ff789b1bdeba56c49c74dff904695ca3e0f40b810e3d35515d6","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_CI_TRIGGER_TOKEN","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/setup-python","sha":"ece7cb06caefa5fff74198d8649806c4678c61a1","version":"v6"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"5e508589e03a7757a7e05b26e834292f5445bfb6","version":"v0.88.7"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14","digest":"sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14@sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"safeoutputs","tools":["create_pull_request","missing_data","missing_tool","noop"]}]} # This file was automatically generated by gh-aw (v0.88.7). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -519,7 +519,7 @@ jobs: REPO: ${{ github.repository }} WORKFLOW_ID: code-documentation name: Validate the baseline and prepare bounded context - run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-exit\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" + run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" - name: Configure Git credentials env: @@ -588,7 +588,7 @@ jobs: env: GH_AW_FILE_ROOT: "${{ runner.temp }}/gh-aw" GH_AW_FILE_CONFIG: "{\"files\":[{\"path\":\"safeoutputs/config.json\",\"content_env\":\"GH_AW_SAFE_OUTPUTS_CONFIG\"}]}" - GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-documentation/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-documentation] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-documentation/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-documentation] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" with: script: | const path = require('path'); @@ -2337,7 +2337,7 @@ jobs: GH_AW_ALLOWED_DOMAINS: "*.docker.com,*.docker.io,*.githubusercontent.com,*.pythonhosted.org,anaconda.org,api.snapcraft.io,archive.ubuntu.com,auth.docker.io,azure.archive.ubuntu.com,binstar.org,bootstrap.pypa.io,codeload.github.com,conda.anaconda.org,conda.binstar.org,crates.io,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,dl.k8s.io,docs.github.com,files.pythonhosted.org,gcr.io,ghcr.io,github-cloud.githubusercontent.com,github-cloud.s3.amazonaws.com,github.blog,github.com,github.githubassets.com,index.crates.io,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,lfs.github.com,mcr.microsoft.com,objects.githubusercontent.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,patch-diff.githubusercontent.com,patchdiff.githubusercontent.com,pip.pypa.io,pkgs.k8s.io,ppa.launchpad.net,production.cloudflare.docker.com,pypi.org,pypi.python.org,quay.io,raw.githubusercontent.com,registry.hub.docker.com,repo.anaconda.com,repo.continuum.io,s.symcb.com,s.symcd.com,security.ubuntu.com,sh.rustup.rs,static.crates.io,static.rust-lang.org,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-documentation/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-documentation] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-documentation/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-documentation] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" GH_AW_CI_TRIGGER_TOKEN: ${{ secrets.GH_AW_CI_TRIGGER_TOKEN }} with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/code-quality.lock.yml b/.github/workflows/code-quality.lock.yml index ec88bf69a..9872a4567 100644 --- a/.github/workflows/code-quality.lock.yml +++ b/.github/workflows/code-quality.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"fb5a451e16c429e1f2168aa26a90632206357c4b28bd7cdf02c1b7719ea74e6d","body_hash":"bc3862e5dda314bb19f655be63479502a0694f3d33196063eba676a1b7760caa","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"b4667cecc2cfd7b4389c63578df63addc3f1f2cee256b08fa58f22ef308168bc","body_hash":"bc3862e5dda314bb19f655be63479502a0694f3d33196063eba676a1b7760caa","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_CI_TRIGGER_TOKEN","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/setup-python","sha":"ece7cb06caefa5fff74198d8649806c4678c61a1","version":"v6"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"5e508589e03a7757a7e05b26e834292f5445bfb6","version":"v0.88.7"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14","digest":"sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14@sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"safeoutputs","tools":["create_pull_request","missing_data","missing_tool","noop"]}]} # This file was automatically generated by gh-aw (v0.88.7). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -519,7 +519,7 @@ jobs: REPO: ${{ github.repository }} WORKFLOW_ID: code-quality name: Validate the baseline and prepare bounded context - run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-exit\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" + run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" - name: Configure Git credentials env: @@ -588,7 +588,7 @@ jobs: env: GH_AW_FILE_ROOT: "${{ runner.temp }}/gh-aw" GH_AW_FILE_CONFIG: "{\"files\":[{\"path\":\"safeoutputs/config.json\",\"content_env\":\"GH_AW_SAFE_OUTPUTS_CONFIG\"}]}" - GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-quality/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-quality] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-quality/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-quality] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" with: script: | const path = require('path'); @@ -2337,7 +2337,7 @@ jobs: GH_AW_ALLOWED_DOMAINS: "*.docker.com,*.docker.io,*.githubusercontent.com,*.pythonhosted.org,anaconda.org,api.snapcraft.io,archive.ubuntu.com,auth.docker.io,azure.archive.ubuntu.com,binstar.org,bootstrap.pypa.io,codeload.github.com,conda.anaconda.org,conda.binstar.org,crates.io,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,dl.k8s.io,docs.github.com,files.pythonhosted.org,gcr.io,ghcr.io,github-cloud.githubusercontent.com,github-cloud.s3.amazonaws.com,github.blog,github.com,github.githubassets.com,index.crates.io,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,lfs.github.com,mcr.microsoft.com,objects.githubusercontent.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,patch-diff.githubusercontent.com,patchdiff.githubusercontent.com,pip.pypa.io,pkgs.k8s.io,ppa.launchpad.net,production.cloudflare.docker.com,pypi.org,pypi.python.org,quay.io,raw.githubusercontent.com,registry.hub.docker.com,repo.anaconda.com,repo.continuum.io,s.symcb.com,s.symcd.com,security.ubuntu.com,sh.rustup.rs,static.crates.io,static.rust-lang.org,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-quality/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-quality] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-quality/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-quality] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" GH_AW_CI_TRIGGER_TOKEN: ${{ secrets.GH_AW_CI_TRIGGER_TOKEN }} with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/code-reusability.lock.yml b/.github/workflows/code-reusability.lock.yml index 259bca2d5..ba4ec5f67 100644 --- a/.github/workflows/code-reusability.lock.yml +++ b/.github/workflows/code-reusability.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"7e13dd40de6a047359e07ae36b1e2a2920f6f4d972e9c8723b4785dddc5e5f34","body_hash":"ae50c16847b63430a9dd92c5e35f9777feccbf32bfd75e527ec9c3c804165aec","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"ff9d68a851c3f55586297444ff4d31ea01ac267dd258f8decf78b8f2948628ed","body_hash":"ae50c16847b63430a9dd92c5e35f9777feccbf32bfd75e527ec9c3c804165aec","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol-fast","engine_versions":{"copilot":"1.0.86"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_CI_TRIGGER_TOKEN","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/setup-python","sha":"ece7cb06caefa5fff74198d8649806c4678c61a1","version":"v6"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"5e508589e03a7757a7e05b26e834292f5445bfb6","version":"v0.88.7"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14","digest":"sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34","pinned_image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.28.14@sha256:3a379c5e96e29499c815e9dd2a71334d01c326a9b73991c76544fda9cae35c34"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"safeoutputs","tools":["create_pull_request","missing_data","missing_tool","noop"]}]} # This file was automatically generated by gh-aw (v0.88.7). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -519,7 +519,7 @@ jobs: REPO: ${{ github.repository }} WORKFLOW_ID: code-reusability name: Validate the baseline and prepare bounded context - run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-exit\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" + run: "set -euo pipefail\nmkdir -p /tmp/gh-aw/agent/baseline-logs\ncat > /tmp/gh-aw/agent/validate_baseline.py <<'PY'\nfrom __future__ import annotations\n\nimport datetime\nimport json\nimport os\nimport subprocess\nfrom pathlib import Path\n\nroot = Path.cwd().resolve()\noutput_dir = Path(\"/tmp/gh-aw/agent\")\nlog_dir = output_dir / \"baseline-logs\"\n\nposix_shell_files = [\n \"guest/common/init\",\n \"guest/alpine/nvx-container-enter\",\n \"guest/alpine/nvx-container-launch\",\n \"guest/common/nvx-hostmount\",\n \"guest/common/nvx-identity-probe\",\n \"guest/common/nvx-init-agent\",\n \"guest/common/nvx-sandbox-smoke\",\n \"guest/common/nvx-snapshot\",\n \"guest/common/nvx-virtio-restore-probe\",\n \"scripts/setup/setup-linux-mshv.sh\",\n \"scripts/setup/setup-linux-runner.sh\",\n]\nbash_shell_files = [\n \".github/specula/setup-runner.sh\",\n \"guest/ubuntu/nvx-bashrc\",\n]\nunit_tests = [\n \"scripts/test_performance.py\",\n \"scripts/test_nvx_tools.py\",\n \"scripts/test_microvm_tests.py\",\n \"scripts/test_development_release.py\",\n]\nshellcheck_image = (\n \"koalaman/shellcheck-alpine@sha256:\"\n \"9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002\"\n)\nshfmt_image = (\n \"mvdan/shfmt@sha256:\"\n \"307d265ffd25ce832899ae17c93ed5062fc3375c514bba8f52cbf52792735c4d\"\n)\npowershell_syntax_check = r\"\"\"\n$tokens = $null\n$errors = $null\n[void][System.Management.Automation.Language.Parser]::ParseFile(\n (Resolve-Path \"scripts/setup/setup-windows-whp.ps1\"),\n [ref]$tokens,\n [ref]$errors\n)\nforeach ($error in $errors) {\n Write-Error $error -ErrorAction Continue\n}\nif ($errors.Count -ne 0) {\n exit 1\n}\n\"\"\"\n# Mirror .github/actions/check-aci-edge-sandboxes/action.yml. The manifest\n# path replaces its working directory, and a rustdoc flag in the command\n# line replaces its RUSTDOCFLAGS variable, so that each command is\n# self-contained and the agent can repeat it unchanged. The Windows type\n# check is an addition: CI runs the crate natively on Windows, which this\n# workflow cannot do. A missing toolchain variable fails these commands\n# without stopping the rest of the baseline.\naci_toolchain = \"+\" + os.environ.get(\"ACI_RUST_TOOLCHAIN\", \"unavailable\")\naci_msrv = \"+\" + os.environ.get(\"ACI_RUST_MSRV\", \"unavailable\")\naci_manifest = [\"--manifest-path\", \"aci_edge_sandboxes/Cargo.toml\"]\naci_locked = [*aci_manifest, \"--locked\"]\naci_deny_warnings = [\"--\", \"-D\", \"warnings\"]\naci_check = [\"check\", *aci_locked, \"--all-targets\", \"--all-features\"]\n\nchecks: list[tuple[str, list[str]]] = [\n (\"python-compile\", [\"python\", \"-m\", \"compileall\", \"-q\", \"scripts\"]),\n (\n \"python-unit-tests\",\n [\"python\", \"-m\", \"unittest\", *unit_tests, \"-v\"],\n ),\n (\n \"host-config-tests\",\n [\n \"python\",\n \".github/skills/nvx-host-connect/scripts/test_hosts.py\",\n \"-v\",\n ],\n ),\n (\"cli-help\", [\"python\", \"scripts/nvx.py\", \"--help\"]),\n (\n \"cli-openvmm-unit-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm-unit\", \"--help\"],\n ),\n (\n \"cli-openvmm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-openvmm\", \"--help\"],\n ),\n (\n \"cli-microvm-help\",\n [\"python\", \"scripts/nvx.py\", \"test-microvm\", \"--help\"],\n ),\n (\n \"cli-benchmark-help\",\n [\"python\", \"scripts/nvx.py\", \"benchmark\", \"--help\"],\n ),\n (\"ruff-check\", [\"python\", \"-m\", \"ruff\", \"check\", \"scripts\"]),\n (\n \"pyright-linux\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Linux\"],\n ),\n (\n \"pyright-windows\",\n [\"python\", \"-m\", \"pyright\", \"--pythonplatform\", \"Windows\"],\n ),\n (\n \"ruff-format\",\n [\"python\", \"-m\", \"ruff\", \"format\", \"--check\", \"scripts\"],\n ),\n (\n \"shellcheck\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=sh\",\n *posix_shell_files,\n ],\n ),\n (\n \"shellcheck-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shellcheck_image,\n \"shellcheck\",\n \"--shell=bash\",\n *bash_shell_files,\n ],\n ),\n (\n \"shfmt\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"posix\",\n \"-i\",\n \"4\",\n \"-ci\",\n *posix_shell_files,\n ],\n ),\n (\n \"shfmt-bash\",\n [\n \"docker\",\n \"run\",\n \"--rm\",\n \"--volume\",\n f\"{root}:/workspace:ro\",\n \"--workdir\",\n \"/workspace\",\n shfmt_image,\n \"-d\",\n \"-ln\",\n \"bash\",\n \"-i\",\n \"4\",\n \"-ci\",\n *bash_shell_files,\n ],\n ),\n (\n \"powershell-syntax\",\n [\n \"pwsh\",\n \"-NoProfile\",\n \"-NonInteractive\",\n \"-Command\",\n powershell_syntax_check,\n ],\n ),\n (\n \"aci-fmt\",\n [\"cargo\", aci_toolchain, \"fmt\", *aci_manifest, \"--check\"],\n ),\n (\n \"aci-clippy-all-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--all-targets\",\n \"--all-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-clippy-no-default-features\",\n [\n \"cargo\",\n aci_toolchain,\n \"clippy\",\n *aci_locked,\n \"--lib\",\n \"--no-default-features\",\n *aci_deny_warnings,\n ],\n ),\n (\n \"aci-test-all-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked, \"--all-features\"],\n ),\n (\n \"aci-test-default-features\",\n [\"cargo\", aci_toolchain, \"test\", *aci_locked],\n ),\n (\n \"aci-doc\",\n [\n \"cargo\",\n aci_toolchain,\n \"--config\",\n 'build.rustdocflags=[\"-D\", \"warnings\"]',\n \"doc\",\n *aci_locked,\n \"--no-deps\",\n \"--all-features\",\n ],\n ),\n (\"aci-msrv\", [\"cargo\", aci_msrv, *aci_check]),\n (\n \"aci-check-macos\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"aarch64-apple-darwin\",\n ],\n ),\n (\n \"aci-check-windows\",\n [\n \"cargo\",\n aci_toolchain,\n *aci_check,\n \"--target\",\n \"x86_64-pc-windows-msvc\",\n ],\n ),\n (\"git-diff-check\", [\"git\", \"diff\", \"--check\"]),\n]\n\ncommand_timeout_seconds = 300\nresults: list[dict[str, object]] = []\nfor name, command in checks:\n log_path = log_dir / f\"{name}.log\"\n try:\n completed = subprocess.run(\n command,\n cwd=root,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n errors=\"replace\",\n check=False,\n timeout=command_timeout_seconds,\n )\n text = completed.stdout\n status = \"passed\" if completed.returncode == 0 else \"failed\"\n return_code = completed.returncode\n except subprocess.TimeoutExpired as error:\n if isinstance(error.stdout, bytes):\n partial_output = error.stdout.decode(\"utf-8\", errors=\"replace\")\n else:\n partial_output = error.stdout or \"\"\n text = (\n f\"{partial_output}\\n\"\n f\"Timed out after {command_timeout_seconds} seconds.\\n\"\n )\n status = \"failed\"\n return_code = None\n except OSError as error:\n text = f\"{type(error).__name__}: {error}\\n\"\n status = \"failed\"\n return_code = None\n log_path.write_text(text, encoding=\"utf-8\")\n results.append(\n {\n \"name\": name,\n \"status\": status,\n \"return_code\": return_code,\n \"command\": command,\n \"log\": str(log_path),\n \"timeout_seconds\": command_timeout_seconds,\n }\n )\n\ncontext = {\n \"workflow_id\": os.environ[\"WORKFLOW_ID\"],\n \"validated_at\": datetime.datetime.now(datetime.timezone.utc).isoformat(),\n \"run_id\": os.environ.get(\"GITHUB_RUN_ID\", \"\"),\n \"head_sha\": os.environ.get(\"GITHUB_SHA\", \"\"),\n \"default_branch\": os.environ.get(\"DEFAULT_BRANCH\", \"dev\"),\n \"baseline\": results,\n \"baseline_passed\": all(item[\"status\"] == \"passed\" for item in results),\n \"not_run\": [\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"verify\"],\n \"reason\": (\n \"Requires the private openvmm submodule; this workflow must \"\n \"not initialize, inspect, or modify that submodule.\"\n ),\n },\n {\n \"command\": [\"python\", \"scripts/nvx.py\", \"test-aci-edge-sandboxes\"],\n \"reason\": (\n \"Requires a hypervisor and OpenVMM release artifacts, as do \"\n \"the ignored openvmm_e2e tests.\"\n ),\n },\n {\n \"command\": [\n \"cargo\",\n \"test\",\n \"--locked\",\n \"--lib\",\n \"--features\",\n \"bundled\",\n \"artifacts\",\n ],\n \"reason\": (\n \"Checks artifact bundling, which only build.rs, \"\n \"artifacts.json, and openvmm/artifacts.rs affect. \"\n \"Candidate selection excludes all three.\"\n ),\n },\n ],\n \"limits\": {\n \"pull_requests\": 1,\n \"files\": 4,\n \"changed_lines\": 99,\n \"recent_pull_requests_per_workflow_and_state\": 20,\n },\n}\n(output_dir / \"repository-context.json\").write_text(\n json.dumps(context, indent=2) + \"\\n\",\n encoding=\"utf-8\",\n)\nprint(\n json.dumps(\n {\n \"baseline_passed\": context[\"baseline_passed\"],\n \"passed\": sum(item[\"status\"] == \"passed\" for item in results),\n \"failed\": sum(item[\"status\"] == \"failed\" for item in results),\n }\n )\n)\nPY\npython /tmp/gh-aw/agent/validate_baseline.py\nfor history_workflow_id in \\\n code-improvement \\\n code-quality \\\n code-documentation \\\n code-deduplication \\\n code-reusability; do\n for state in open closed; do\n gh api --method GET search/issues \\\n -f q=\"repo:${REPO} is:pr is:${state} \\\"gh-aw-workflow-id: ${history_workflow_id}\\\" in:body\" \\\n -f sort=updated \\\n -f order=desc \\\n -f per_page=20 \\\n --jq \"[.items[] | {workflow: \\\"${history_workflow_id}\\\", number, title, state, merged_at: .pull_request.merged_at, closed_at, updated_at, html_url}]\"\n done\ndone |\n jq --slurp 'add | unique_by(.number) | sort_by(.updated_at) | reverse' \\\n > /tmp/gh-aw/agent/pull-request-history.json\n" - name: Configure Git credentials env: @@ -588,7 +588,7 @@ jobs: env: GH_AW_FILE_ROOT: "${{ runner.temp }}/gh-aw" GH_AW_FILE_CONFIG: "{\"files\":[{\"path\":\"safeoutputs/config.json\",\"content_env\":\"GH_AW_SAFE_OUTPUTS_CONFIG\"}]}" - GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-reusability/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-reusability] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-reusability/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-reusability] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" with: script: | const path = require('path'); @@ -2337,7 +2337,7 @@ jobs: GH_AW_ALLOWED_DOMAINS: "*.docker.com,*.docker.io,*.githubusercontent.com,*.pythonhosted.org,anaconda.org,api.snapcraft.io,archive.ubuntu.com,auth.docker.io,azure.archive.ubuntu.com,binstar.org,bootstrap.pypa.io,codeload.github.com,conda.anaconda.org,conda.binstar.org,crates.io,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,dl.k8s.io,docs.github.com,files.pythonhosted.org,gcr.io,ghcr.io,github-cloud.githubusercontent.com,github-cloud.s3.amazonaws.com,github.blog,github.com,github.githubassets.com,index.crates.io,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,lfs.github.com,mcr.microsoft.com,objects.githubusercontent.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,patch-diff.githubusercontent.com,patchdiff.githubusercontent.com,pip.pypa.io,pkgs.k8s.io,ppa.launchpad.net,production.cloudflare.docker.com,pypi.org,pypi.python.org,quay.io,raw.githubusercontent.com,registry.hub.docker.com,repo.anaconda.com,repo.continuum.io,s.symcb.com,s.symcd.com,security.ubuntu.com,sh.rustup.rs,static.crates.io,static.rust-lang.org,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-exit\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-reusability/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-reusability] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"create_pull_request\":{\"allowed_files\":[\"doc/*.md\",\"doc/**/*.md\",\"scripts/*.py\",\"scripts/**/*.py\",\"scripts/setup/*.sh\",\"scripts/setup/*.ps1\",\"aci_edge_sandboxes/src/*.rs\",\"aci_edge_sandboxes/src/**/*.rs\",\"aci_edge_sandboxes/tests/*.rs\",\"aci_edge_sandboxes/tests/**/*.rs\",\"aci_edge_sandboxes/examples/*.rs\",\"guest/common/init\",\"guest/alpine/nvx-container-enter\",\"guest/alpine/nvx-container-launch\",\"guest/common/nvx-hostmount\",\"guest/common/nvx-identity-probe\",\"guest/common/nvx-init-agent\",\"guest/common/nvx-sandbox-smoke\",\"guest/common/nvx-snapshot\",\"guest/common/nvx-virtio-restore-probe\",\"guest/ubuntu/nvx-bashrc\"],\"base_branch\":\"dev\",\"branch_prefix\":\"code-reusability/\",\"draft\":true,\"excluded_files\":[\"openvmm\",\"openvmm/**\",\".gitmodules\",\".github/workflows/*.lock.yml\",\"scripts/publish_development_release.py\",\"scripts/nvx_tools/release.py\",\"aci_edge_sandboxes/src/openvmm/artifacts.rs\",\"aci_edge_sandboxes/src/openvmm/contract.rs\",\"aci_edge_sandboxes/src/openvmm/platform/windows.rs\",\"aci_edge_sandboxes/target/**\",\"data/**\",\"build/**\",\".cache/**\",\"unittest_results.txt\"],\"expires\":336,\"fallback_as_issue\":false,\"if_no_changes\":\"ignore\",\"max\":1,\"max_patch_files\":4,\"max_patch_size\":4096,\"protect_top_level_dot_folders\":true,\"protected_files\":[\"package.json\",\"bun.lockb\",\"bunfig.toml\",\"deno.json\",\"deno.jsonc\",\"deno.lock\",\"global.json\",\"NuGet.Config\",\"Directory.Packages.props\",\"mix.exs\",\"mix.lock\",\"go.mod\",\"go.sum\",\"stack.yaml\",\"stack.yaml.lock\",\"pom.xml\",\"build.gradle\",\"build.gradle.kts\",\"settings.gradle\",\"settings.gradle.kts\",\"gradle.properties\",\"package-lock.json\",\"yarn.lock\",\"pnpm-lock.yaml\",\"npm-shrinkwrap.json\",\"requirements.txt\",\"Pipfile\",\"Pipfile.lock\",\"pyproject.toml\",\"setup.py\",\"setup.cfg\",\"Gemfile\",\"Gemfile.lock\",\"uv.lock\",\"CODEOWNERS\",\"DESIGN.md\",\"README.md\",\"CONTRIBUTING.md\",\"CHANGELOG.md\",\"SECURITY.md\",\"CODE_OF_CONDUCT.md\",\"AGENTS.md\"],\"protected_files_policy\":\"fallback-to-issue\",\"title_prefix\":\"[code-reusability] \"},\"create_report_incomplete_issue\":{},\"mentions\":{\"enabled\":false},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" GH_AW_CI_TRIGGER_TOKEN: ${{ secrets.GH_AW_CI_TRIGGER_TOKEN }} with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/shared/code-improvement.md b/.github/workflows/shared/code-improvement.md index af2c5f628..956d7911f 100644 --- a/.github/workflows/shared/code-improvement.md +++ b/.github/workflows/shared/code-improvement.md @@ -102,7 +102,6 @@ steps: "guest/common/init", "guest/alpine/nvx-container-enter", "guest/alpine/nvx-container-launch", - "guest/common/nvx-exit", "guest/common/nvx-hostmount", "guest/common/nvx-identity-probe", "guest/common/nvx-init-agent", @@ -553,7 +552,6 @@ safe-outputs: - "guest/common/init" - "guest/alpine/nvx-container-enter" - "guest/alpine/nvx-container-launch" - - "guest/common/nvx-exit" - "guest/common/nvx-hostmount" - "guest/common/nvx-identity-probe" - "guest/common/nvx-init-agent" diff --git a/doc/guests/ubuntu-guest.md b/doc/guests/ubuntu-guest.md index c01d32a81..ebcf4d72a 100644 --- a/doc/guests/ubuntu-guest.md +++ b/doc/guests/ubuntu-guest.md @@ -382,7 +382,6 @@ policy. The implementation keeps genuinely common sources in a neutral ```text guest/common/init -guest/common/nvx-exit guest/common/nvx-hostmount guest/common/nvx-init-agent guest/common/nvx-managed-agent.c diff --git a/doc/setup.md b/doc/setup.md index b8f9003ad..b77759b9f 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -168,7 +168,7 @@ Run all lint and formatting checks before submitting a change: python3 -m ruff check scripts .github/specula shellcheck --shell=sh \ guest/common/init guest/alpine/nvx-container-enter \ - guest/alpine/nvx-container-launch guest/common/nvx-exit \ + guest/alpine/nvx-container-launch \ guest/common/nvx-hostmount guest/common/nvx-identity-probe \ guest/common/nvx-init-agent guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ @@ -180,7 +180,7 @@ python3 -m pyright --pythonplatform Windows python3 -m ruff format --check scripts .github/specula shfmt -d -ln posix -i 4 -ci \ guest/common/init guest/alpine/nvx-container-enter \ - guest/alpine/nvx-container-launch guest/common/nvx-exit \ + guest/alpine/nvx-container-launch \ guest/common/nvx-hostmount guest/common/nvx-identity-probe \ guest/common/nvx-init-agent guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ @@ -220,7 +220,7 @@ Apply the configured Python and POSIX shell formatters with: python3 -m ruff format scripts .github/specula shfmt -w -ln posix -i 4 -ci \ guest/common/init guest/alpine/nvx-container-enter \ - guest/alpine/nvx-container-launch guest/common/nvx-exit \ + guest/alpine/nvx-container-launch \ guest/common/nvx-hostmount guest/common/nvx-identity-probe \ guest/common/nvx-init-agent guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ diff --git a/docker/Dockerfile b/docker/Dockerfile index 88630c59d..3a5eb998d 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -65,6 +65,8 @@ RUN set -eux; \ for helper in nvx-reseed nvx-mmio-write nvx-port-io nvx-console-pending nvx-managed-agent; do \ cc -static -Os -s -Wall -Wextra -Werror -o "/out/helpers/$helper" "/repo/guest/common/$helper.c"; \ done; \ + musl-gcc -static -Os -s -Wall -Wextra -Werror -o /out/helpers/nvx-exit \ + /repo/guest/common/nvx-exit.c; \ cc -nostdlib -static -Os -fno-builtin -fno-pie -fno-stack-protector -no-pie \ -Wl,--build-id=none -Wl,-z,noexecstack -s -o /out/helpers/nvx-device-io \ /repo/guest/common/nvx-device-io.c @@ -101,8 +103,7 @@ COPY --from=azurelinux-helpers /out/helpers/ /rootfs/sbin/ RUN <<'EOF' set -eux install -m 0755 /repo/guest/common/init /rootfs/init -install -m 0755 /repo/guest/common/nvx-exit \ - /repo/guest/common/nvx-hostmount \ +install -m 0755 /repo/guest/common/nvx-hostmount \ /repo/guest/common/nvx-identity-probe \ /repo/guest/common/nvx-snapshot \ /repo/guest/common/nvx-virtio-restore-probe \ diff --git a/guest/common/nvx-exit b/guest/common/nvx-exit deleted file mode 100644 index d8ef34455..000000000 --- a/guest/common/nvx-exit +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/sh -# Stop the VM through control port 0x604, carrying an 8-bit process exit code. -code=${1:-1} -case "$code" in - '' | *[!0-9]*) code=1 ;; -esac -if [ "$code" -lt 0 ] || [ "$code" -gt 255 ]; then - code=1 -fi -octal=$(printf '%03o' "$code") -printf '%b' "\\0$octal" | dd of=/dev/port bs=1 seek=1540 count=1 conv=notrunc 2>/dev/null diff --git a/guest/common/nvx-exit.c b/guest/common/nvx-exit.c new file mode 100644 index 000000000..6f581ad8a --- /dev/null +++ b/guest/common/nvx-exit.c @@ -0,0 +1,46 @@ +/* + * nvx-exit [CODE]: stop the VM with an 8-bit exit code, by writing one byte + * to OpenVMM's control port 0x604 through /dev/port. CODE defaults to 1, and + * any CODE that isn't a decimal number from 0 to 255 becomes 1. It prints + * nothing; if the write fails, it exits with status 1. + * + * This is a single static exec rather than a shell script, because a guest's + * first command after a restore pays a cold first touch for every fork, exec + * and exit, and the script ran sh, a command substitution and a printf | dd + * pipeline. + */ +#include +#include +#include + +/* Tests build against a regular file; the guest always writes /dev/port. */ +#ifndef NVX_EXIT_DEVICE +#define NVX_EXIT_DEVICE "/dev/port" +#endif +#define NVX_EXIT_PORT 0x604 + +static unsigned char exit_code(const char *argument) +{ + unsigned int value = 0; + + if (argument == NULL || *argument == '\0') + return 1; + for (; *argument != '\0'; argument++) { + if (*argument < '0' || *argument > '9') + return 1; + value = value * 10 + (unsigned int)(*argument - '0'); + if (value > 255) + return 1; + } + return (unsigned char)value; +} + +int main(int argc, char **argv) +{ + unsigned char code = exit_code(argc > 1 ? argv[1] : NULL); + int port = open(NVX_EXIT_DEVICE, O_WRONLY); + + if (port < 0) + return 1; + return pwrite(port, &code, 1, NVX_EXIT_PORT) == 1 ? 0 : 1; +} diff --git a/scripts/nvx_tools/build.py b/scripts/nvx_tools/build.py index 9063505c7..dd917075b 100644 --- a/scripts/nvx_tools/build.py +++ b/scripts/nvx_tools/build.py @@ -792,7 +792,6 @@ def _install_guest_files( helpers: dict[str, dict[str, str]] = {} scripts = [ ("init", common / "init", root / "init"), - ("nvx-exit", common / "nvx-exit", root / "sbin" / "nvx-exit"), ( "nvx-hostmount", common / "nvx-hostmount", diff --git a/scripts/nvx_tools/build_constants.py b/scripts/nvx_tools/build_constants.py index 11dc7c3ab..fd2c07cbe 100644 --- a/scripts/nvx_tools/build_constants.py +++ b/scripts/nvx_tools/build_constants.py @@ -327,10 +327,11 @@ class InitramfsBuildConstants: "nvx-console-pending", "nvx-managed-agent", ) - # Time tools are linked against musl: a static glibc binary adds about - # 700 KB to the initramfs, and every unpacked kilobyte delays cold boot. + # Time tools and nvx-exit are linked against musl: a static glibc binary + # adds about 700 KB to the initramfs, and every unpacked kilobyte delays + # cold boot. nvx-exit is also a restored guest's usual first exec. MUSL_COMPILER: Final = "musl-gcc" - MUSL_STATIC_HELPERS: Final = ("nvx-time",) + MUSL_STATIC_HELPERS: Final = ("nvx-time", "nvx-exit") TIME_PROBE_NAME: Final = "nvx-time-probe" TIME_PROBE_CFLAGS: Final = ( "-std=gnu11", diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index d856ebb8c..b08f6ffbf 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -6110,7 +6110,7 @@ def install(_source: Path, destination: Path) -> dict[str, str]: ) azure_stage = dockerfile.split("FROM base AS azurelinux-initramfs", 1)[1] azure_sbin_scripts = azure_stage.split( - "install -m 0755 /repo/guest/common/nvx-exit", 1 + "install -m 0755 /repo/guest/common/init /rootfs/init\n", 1 )[1].split("/rootfs/sbin/", 1)[0] self.assertIn("/repo/guest/common/nvx-hostmount", azure_sbin_scripts) init_script = ( @@ -7300,6 +7300,90 @@ def test_malformed_task_count_refuses_further_exec_until_verified_empty(self): self.assertEqual(self._check("settle")[0], 0) +class GuestExitTests(unittest.TestCase): + """nvx-exit's arguments, and its single one-byte write at port 0x604.""" + + SOURCE = Path(__file__).parents[1] / "guest" / "common" / "nvx-exit.c" + PORT = 0x604 + + def setUp(self): + if sys.platform != "linux": + self.skipTest("nvx-exit builds on Linux") + compiler = shutil.which("cc") or shutil.which("gcc") + if compiler is None: + self.skipTest("C compiler is unavailable") + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + root = Path(temporary.name) + # A regular file stands in for /dev/port, so no test writes a port. + self.device = root / "port" + flags = [ + flag + for flag in InitramfsBuildConstants.STATIC_HELPER_CFLAGS + if flag != "-static" + ] + self.binaries: dict[str, Path] = {} + for name, device in (("ok", self.device), ("missing", root / "no" / "port")): + binary = root / f"nvx-exit-{name}" + result = subprocess.run( + [ + compiler, + *flags, + f'-DNVX_EXIT_DEVICE="{device}"', + "-o", + str(binary), + str(self.SOURCE), + ], + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.binaries[name] = binary + + def _run(self, name: str, *arguments: str) -> subprocess.CompletedProcess[bytes]: + return subprocess.run( + [str(self.binaries[name]), *arguments], + capture_output=True, + check=False, + timeout=10, + ) + + def test_writes_the_exit_code_as_one_byte_at_the_control_port(self): + cases = ( + ((), 1), + (("",), 1), + (("0",), 0), + (("37",), 37), + (("255",), 255), + (("256",), 1), + (("-1",), 1), + (("+5",), 1), + ((" 7",), 1), + (("12a",), 1), + (("99999999999999999999",), 1), + # Decimal: the old script's printf read a leading zero as octal. + (("010",), 10), + (("7", "9"), 7), + ) + for arguments, expected in cases: + with self.subTest(arguments=arguments): + self.device.write_bytes(b"") + result = self._run("ok", *arguments) + self.assertEqual( + (result.returncode, result.stdout, result.stderr), (0, b"", b"") + ) + self.assertEqual( + self.device.read_bytes(), bytes(self.PORT) + bytes([expected]) + ) + + def test_fails_silently_when_the_port_cannot_be_opened(self): + result = self._run("missing", "0") + self.assertEqual( + (result.returncode, result.stdout, result.stderr), (1, b"", b"") + ) + + class ManagedAgentStopTests(unittest.TestCase): SOURCE = Path(__file__).parents[1] / "guest" / "common" / "nvx-managed-agent.c" OUTER = struct.Struct("<4sHBB16sQQI") From 088c4ca74dcb34ad248ad282188d0562ea260d6a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 11:03:53 -0700 Subject: [PATCH 147/188] doc: describe test hosts generically in the time ABI spec Both repositories are public, so the specification names no internal hosts. It describes them by role and CPU instead: the bare-metal KVM, MSHV, and WHP hosts, nested Azure KVM (8370C) and MSHV (8573C), and the Azure WHP runners (8370C and 8573C). No technical content changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 112 +++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 55 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c51db2ce1..59d8cd10d 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -268,7 +268,7 @@ settled cell on the registered hosts. | Obligation | KVM | MSHV | WHP | | --- | --- | --- | --- | | Identity CPUID (exact leaves, out-of-range rule) | `KVM_SET_CPUID2` with the identity and explicit zero leaves; every KVM `0x4xxxxxxx` entry removed. Other leaves in the range return KVM's Intel out-of-range result | No synthetic processor features, so the hypervisor reports no `0x400000xx` leaves; CPUID intercept results (`always_override`) for the identity leaves, while the explicit zero leaves already read zero in VP 0's view and get none (see the next row); preflight reads every entry back and requires VP 0 to read zero at six sentinel leaves (`0x40000006`, `0x40000081`, `0x400000ff`, `0x40000100`, `0x40000200`, and `0x4000ff00`) | CPUID exits for the six identity leaves, served from OpenVMM's table; with synthetic features off, WHP returns zero natively for `0x40000006..=0x400000ff` and the bases from `0x40000100`, which preflight asserts | -| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 11 of 61 entries on prometheus30 (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 12 of 65 on azure-azlinux-5 (the same plus invariant TSC), at every vCPU count, because VP 0's own view already presents leaf 4's shared L3; registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | +| Profile CPUID and time bits | `KVM_SET_CPUID2` with the effective CPUID. KVM 6.3 and newer hide invariant TSC while KVM's own `HV_X64_MSR_TSC_INVARIANT_CONTROL` is 0, so the backend mirrors the guest-visible value, which core saves with the VM, into it with a host-initiated `KVM_SET_MSRS` after every accepted guest write and before a VP runs after a restore or reset; without it the guest loses `constant_tsc` and `nonstop_tsc` and boots about 160 ms slower | Processor feature banks derived from the profile (`hv_banks`), plus a CPUID intercept result (`always_override`, subleaf-specific where the entry has a subleaf) for every partition-wide entry of the effective CPUID that VP 0's own view does not already present under its mask. Once VP 0 exists, one bulk `HvCallGetVpCpuidValues` reads every partition-wide entry before any result is registered, and a failed bulk read registers every entry. That leaves 11 of 61 entries on the bare-metal host (the hypervisor bit, one leaf-2 descriptor byte, ARAT, the six identity leaves, and two brand leaves) and 12 of 65 on the nested Azure host (the same plus invariant TSC), at every vCPU count, because VP 0's own view already presents leaf 4's shared L3; registering them, the read included, takes 63 to 71 µs and about 90 µs, against 200 and about 300 µs for every entry. Profile masks never cover runtime-owned or VM-owned bits, so a pinned value that VP 0 presents at reset it presents always. Reserved entries pass through from the hypervisor's guest view, which reads zero at every one with no result registered, and verification requires every candidate to read zero (`E_CPU_UNLISTED`). Per-VP fields cannot be left to the hypervisor: it does not implement `0xB` for these partitions and reads `EDX` as 0 on every VP, so Linux would log `APIC ID mismatch` (`G_APIC_ID_MISMATCH`). The `0xB` and `0x1F` entries are therefore per-VP intercept results with the VP's x2APIC ID in `EDX`, registered as each VP is created: the hypervisor refuses a per-VP result for a VP that does not exist yet (`InvalidVpIndex`). Leaf 1's initial APIC ID is the hypervisor's own, which is correct | Processor feature banks derived from the profile (`hv_banks`), and every effective-CPUID entry outside the hypervisor range as a `CpuidResultList2` result (the banks cannot express the hypervisor bit, ARAT, or, on Azure, invariant TSC); reserved entries pass through from WHP's guest view, which verification requires to be zero (`E_CPU_UNLISTED`). CPUID exits only for the identity leaves and the leaves with per-VP APIC fields (`1`, `0xB`, and, where listed, `0x1F` and `0x8000001E`), 8 exits for every v1 profile: each exit-list entry adds about 25 µs to partition setup, and the legacy 269-entry list cost about 7 ms per restore. The exit handler presents the effective CPUID's `0xB` terminator, with the VP's x2APIC ID, instead of zeroing subleaves from 2 | | Identity MSRs routed to OpenVMM | `KVM_CAP_X86_USER_SPACE_MSR` (`UNKNOWN`, `FILTER`) and `KVM_X86_SET_MSR_FILTER` denying reads and writes of `0x40000000..=0x400001ff`; the filter takes precedence over KVM's in-kernel Hyper-V MSRs. A Linux boot takes four MSR exits, all on the BSP, at any vCPU count | MSR-index intercepts (`HV_INTERCEPT_TYPE_X64_MSR_INDEX`, `READ_WRITE`) for `0x40000002`, `0x40000022`, `0x40000023`, and `0x40000118`. The hypervisor itself raises #GP for writes to the three read-only MSRs and for every other MSR in the range. Native synthetic MSRs are forbidden: they pre-empt the intercepts, and the native `HV_X64_MSR_TSC_FREQUENCY` returns the destination's rate instead of `F`. Verified on hypervisor builds 26100.30000 (bare metal) and 26100.9444 (Azure) | `X64MsrExitBitmap` with `UnhandledMsrs` (capability `0x3f` on every host) and the offloaded APIC, no synthetic features and no `hv1_emulator`: the identity MSRs exit to OpenVMM, which raises #GP for every other MSR in the range | | Native rate `F_d` | `KVM_GET_TSC_KHZ` × 1000 on VP 0 (1 kHz granularity) | `ProcessorClockFrequency` partition property | `WHvCapabilityCodeProcessorClockFrequency` | | No TSC scaling | Never `KVM_SET_TSC_KHZ`; every vCPU reports the host rate | No frequency override | No `ProcessorClockFrequency` partition property; the 1 GHz request is removed | @@ -280,7 +280,7 @@ settled cell on the registered hosts. | Capture anchor: VP 0 TSC paired with a host time sample within 100 µs, from at most 64 samples | Host `rdtsc` plus VP 0's `KVM_VCPU_TSC_OFFSET`, bracketed by two host `rdtsc` reads around the host clock reads; up to 16 attempts (0.06 to 0.4 µs) | The tightest of up to 64 brackets `[sample, HvCallGetVpRegisters(VP 0 TSC), sample]`, paired at the bracket midpoint (p50 3.5 µs on bare metal, 6.2 µs on Azure) | The tightest of up to 64 bracketed reads of VP 0's TSC register, paired at the bracket midpoint (2.6 to 6.4 µs on bare metal, 5.4 to 9.7 µs on 8370C runners, 5.8 to 11 µs on 8573C runners) | | Synchronized TSC set at one host instant | One `KVM_VCPU_TSC_OFFSET` value for every vCPU, `target(t) - (h0 + h1) / 2` from a host clock read `t` bracketed by host `rdtsc` reads `h0` and `h1` (Linux 5.16 or newer); no `IA32_TSC` writes, which Linux 6.6 can discard | Freeze partition time (already frozen since preflight), write the target to every created VP, read back, and clear `TimeFreeze` right after a successful read-back, inside the set rather than at the first VP run (56 to 312 µs for 1 to 8 VPs; equal on 20 of 20 restores) | Suspend partition time, write the target to every VP, read back, and resume with `WHvResumePartitionTime` right after a successful read-back, inside the set. Writing while time runs would skew the VPs by the write latency, about 11 µs per write on bare metal and 25 µs nested; `TscVirtualOffset` is unusable (writes fail) | | Read-back before any VP runs | Every vCPU's `KVM_VCPU_TSC_OFFSET` equals the written value, and a host `rdtsc` bracket around VP 0's `IA32_TSC` shows no scaling | Every created VP's TSC equals the target while time is frozen | Every VP's TSC equals the target while time is suspended; live reads cannot verify 1 µs (a register read takes 9.5 to 21 µs) | -| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on prometheus28, 8370C, and 8573C hosts | +| Live cross-vCPU skew after release at most 1 µs | Equal offsets: skew is the host's TSC skew, bounded by qualification. Measured at most 63 ns | Measured 0 warps; offsets within 516 ns on dual-socket bare metal and 195 ns on Azure, both bounded by the probe's round trip | Measured at most 70 ns over 60 s on the bare-metal host and on 8370C and 8573C runners | | VP instantiation | All `C` VPs exist before the set | Every instantiated VP is bound, which creates it, before the set; no VP is created after it | All `C` VPs exist before the set | | Partition capabilities, so that the identity leaves never enable the Hyper-V emulator or its saved-state elements | Derived from CPUID with the hypervisor range masked: `hv1` and `kvm_clock` are false | Same | Same | | Unknown MSRs | #GP, including the legacy L2-cache MSR stubs (see [MSRs](#msrs)) | #GP from the hypervisor | #GP, including the legacy L2-cache MSR stubs | @@ -291,7 +291,7 @@ Linux 6.6 KVM, which the Azure KVM runners run, treats a host `IA32_TSC` write within about 1 s of the TSC timeline started at vCPU creation as a synchronization attempt and discards the written value, the first write of a restore included; every CI shell snapshot is taken within a second of boot. -With the legacy restore path on `azure-kvm-5`, a restored guest's monotonic +With the legacy restore path on nested Azure KVM, a restored guest's monotonic clock advanced 54.7 ms across a 512.5 ms host interval, losing 458 ms. Linux 6.7 applies the heuristic only after a first user-space write (`user_set_tsc`). `KVM_VCPU_TSC_OFFSET` sets the offset exactly on every @@ -447,8 +447,8 @@ the Skylake-SP roots show as non-zero: - WHP's hardware sweep covers subleaves 0 to 63 of every indexed leaf and four leaves past each maximum, on all three host types. - MSHV's sweep, run with no result registered for any reserved entry, covers - the host-enumerated reserved entries (15 on prometheus30 and 13 on - azure-azlinux-5) plus 2,755 and 3,261 probes: the same near range as + the host-enumerated reserved entries (15 on the bare-metal host and 13 on + the nested Azure one) plus 2,755 and 3,261 probes: the same near range as WHP's, and far probes where the hypervisor answers by itself (subleaves 64 to 255 of every indexed leaf, the leaves past the near range up to `0xff` and `0x800000ff`, the hypervisor range up to `0x400001ff`, and nine distant @@ -548,7 +548,7 @@ bare-metal hosts (one per backend) and fifteen Azure hosts: | ID | Generation | Hosts | Source backends | | --- | --- | --- | --- | -| `intel.skylake-sp.v1` | 6/85, steppings 0 to 4 | Bare-metal prometheus hosts | KVM, MSHV, WHP | +| `intel.skylake-sp.v1` | 6/85, steppings 0 to 4 | The bare-metal hosts | KVM, MSHV, WHP | | `intel.icelake-sp.v1` | 6/106 | Xeon Platinum 8370C runners | KVM, MSHV, WHP | | `intel.emeraldrapids.v1` | 6/207 | Xeon Platinum 8573C runners | MSHV, WHP (no KVM host exists) | @@ -625,8 +625,8 @@ restored state, must reach its VP when the VP is released, even if the VP is halted. MSHV and WHP do not wake a halted VP for an IRR bit written with its LAPIC state, so their backends assert the vector again (see [Backend obligations](#backend-obligations)). Without that, on busy 8-vCPU -guests, 29 of 60 time ABI restores on MSHV (prometheus30) and 5 of 12 on WHP -(prometheus28) left a CPU halted with its timer vector pending until another +guests, 29 of 60 time ABI restores on bare-metal MSHV and 5 of 12 on +bare-metal WHP left a CPU halted with its timer vector pending until another interrupt woke it, about 2 s later or never; with it, 0 of 300 and 0 of 12. A periodic or TSC-deadline LAPIC timer is never valid: capture and restore reject one that is armed (`E_LAPIC_PERIODIC`, `E_LAPIC_TSC_DEADLINE`). @@ -727,7 +727,7 @@ resume at the read-back, at parity with KVM, against −0.9 to −1.6 ms when th first VP run thawed time; warps and restore latency were unchanged. The legacy path, which stopped guest time during restore setup, left guest `CLOCK_MONOTONIC` about 400 ms behind the host on every restore on -`azure-kvm-5` with 512 MiB (7 to 12 ms on prometheus32); the time ABI stays +nested Azure KVM with 512 MiB (7 to 12 ms on bare metal); the time ABI stays within 1.5 ms there. **Anchor pairing bound.** An anchor that pairs a TSC read with a host time @@ -1056,13 +1056,13 @@ difference: | Backend | Host | `epsilon` p50 / p99 | Worst sample | | --- | --- | --- | --- | -| KVM | prometheus32 (bare metal, Linux 7.0) | 4.0 to 4.1 µs / 4 to 15 µs | 33 µs | -| KVM | `azure-kvm-5` (nested, Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | -| MSHV | prometheus30 (bare metal) | 6.3 to 6.4 µs / 6.8 to 7.1 µs | 37 µs | -| MSHV | `azure-azlinux-5` (nested) | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | -| WHP | prometheus28 (bare metal) | 20.7 to 21.3 µs / 29.4 to 30.5 µs | 52 µs | -| WHP | `azure-windows-1` (8370C, nested) | 24.8 to 25.2 µs / 36.5 to 41.9 µs | 794 µs | -| WHP | `azure-windows-3` (8573C, nested) | 30.5 to 30.8 µs / 46.0 to 46.4 µs | 251 µs | +| KVM | Bare metal (Linux 7.0) | 4.0 to 4.1 µs / 4 to 15 µs | 33 µs | +| KVM | Nested Azure 8370C (Linux 6.6) | 3.8 to 4.1 µs / 4 to 9 µs | 75 µs | +| MSHV | Bare metal | 6.3 to 6.4 µs / 6.8 to 7.1 µs | 37 µs | +| MSHV | Nested Azure 8573C | 12.7 to 13.0 µs / 15 to 25 µs | 102 µs | +| WHP | Bare metal | 20.7 to 21.3 µs / 29.4 to 30.5 µs | 52 µs | +| WHP | Nested Azure 8370C | 24.8 to 25.2 µs / 36.5 to 41.9 µs | 794 µs | +| WHP | Nested Azure 8573C | 30.5 to 30.8 µs / 46.0 to 46.4 µs | 251 µs | The second run of each boot overlaps the console's drain of the first run's output, and it has the higher KVM p99s. The WHP rows come from the WHP @@ -1379,10 +1379,10 @@ Step 13 starts 150 ms after the acknowledgement (or after step 10, when none is required), and the asynchronous boot checks start 150 ms after the boot step. Starting them earlier slows the guest: their checks contend with the readiness path on the other vCPUs, even at `SCHED_IDLE`. On WHP -(prometheus28, 512 MiB, measurement-only builds), checks right after the +(bare metal, 512 MiB, measurement-only builds), checks right after the acknowledgement added 6 to 12 ms to 2-vCPU restores, whereas a 100 ms delay left them 0.8 and 1.2 ms over the release guest at 1 and 2 vCPUs. On -MSHV (prometheus30), boot checks right after shell-ready added about 3 ms +bare-metal MSHV, boot checks right after shell-ready added about 3 ms to 2-vCPU cold boots. KVM showed no clear effect either way. The delay is 150 ms rather than 100 ms because a workload's own activity right after readiness must not meet the checks either: the guest-exit window of @@ -1461,8 +1461,8 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: synchronization slews by a few ppm. The PLL's frequency integrator absorbs that rate over about an hour at the 64 s cadence. Until then the offset settles near 256 s times the rate: 1.76 ms behind host UTC at the 6.88 ppm - of an Azure WHP runner, and an expected 4.2 ms at prometheus28's 16.5 ppm. - Both are far below the step threshold. + of an Azure WHP runner, and an expected 4.2 ms at the 16.5 ppm of the + bare-metal WHP host. Both are far below the step threshold. The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. @@ -1649,7 +1649,7 @@ Hyper-V PTP clock: - Against `CLOCK_MONOTONIC_RAW` and `QueryPerformanceCounter`, 35 short-schedule runs on the Azure KVM, MSHV, and WHP runners agree within 0.000 to 0.050 ppm, with the largest `u_i` between 0.04 and 0.106 ppm. The - long schedule agrees within 0.002 ppm on prometheus30, where + long schedule agrees within 0.002 ppm on the bare-metal MSHV host, where `CLOCK_MONOTONIC` gave 0.402 ppm. - The WHP hosts show 0.000 ppm with a residual of at most 0.07 µs over 118 s. @@ -1658,10 +1658,10 @@ orders of magnitude where the host's clocksource does not follow the TSC's steps: the Hyper-V reference page and `QueryPerformanceCounter` in a VM, which the hypervisor computes from the TSC but keeps continuous when it corrects TSC offsets, or the HPET, which has its own oscillator. -Where the host clocksource is `tsc` itself (the Azure KVM runners and -prometheus32), every kernel clock derives from the TSC, so no host clock can -catch a TSC step; the guest warp probe (`H6` and the CI schedule) is the -detector there. +Where the host clocksource is `tsc` itself (the Azure KVM runners and the +bare-metal KVM host), every kernel clock derives from the TSC, so no host +clock can catch a TSC step; the guest warp probe (`H6` and the CI schedule) +is the detector there. The 100 ppm bound keeps the guest's wall-clock discipline, which steers at most 500 ppm, clear of the TSC's error plus a restore's rate deviation of up @@ -1693,8 +1693,8 @@ gate, on any backend: - On Azure, WHP and nested MSHV cannot offer the invariant-TSC bit to their guests through their feature banks, although the host OS sees an invariant - TSC on every runner except `azure-azlinux-2`. The profile exposes the bit - through CPUID anyway, so `H4` and `H6` measure what it promises. + TSC on every runner except one 8370C MSHV runner. The profile exposes the + bit through CPUID anyway, so `H4` and `H6` measure what it promises. - MSHV roots and Azure KVM hosts run `hyperv_clocksource_tsc_page`, not `tsc`. The guest has no kvmclock, so no guest clock derives from the host clocksource, and the downtime comes from host monotonic time and UTC, whose @@ -1704,7 +1704,7 @@ gate, on any backend: `hyperv_clocksource_tsc_page`. `H6` measures the warps that this would cause. -`azure-azlinux-2` (8370C), whose host OS lacks the bit and which showed the +That 8370C MSHV runner, whose host OS lacks the bit and which showed the #265 warps, is out of rotation and unqualified because our account cannot run guests there, not because of this rule; its host-level warp probe saw backward steps of at most 2.1 ns. @@ -1727,10 +1727,11 @@ Routing CI jobs by generation labels is optional future work. **Gate.** Per backend and vCPU count, no p50 is worse than the base-branch median by more than `max(5%, 2 ms)`. The gate covers all 36 one-vCPU metrics and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. It is measured on -the CI Azure platforms; bare-metal results are informational. prometheus32 -idles at 800 MHz (`intel_pstate` powersave without HWP), which inflates its -absolute restore latency (`restored_ms` p50 at one vCPU 39 ms, against 23 ms -at full clock) and its `cpu_us`; the arms of an A/B share the effect. +the CI Azure platforms; bare-metal results are informational. The bare-metal +KVM host idles at 800 MHz (`intel_pstate` powersave without HWP), which +inflates its absolute restore latency (`restored_ms` p50 at one vCPU 39 ms, +against 23 ms at full clock) and its `cpu_us`; the arms of an A/B share the +effect. Expected effects: @@ -1741,17 +1742,17 @@ Expected effects: | Wall clock from the packet instead of RTC polling | Removes at least 32 CMOS port exits and the update wait per tiered restore; the gate's base skips the RTC reads in its untiered restores | | The kernel's RTC write | Because the discipline clears `STA_UNSYNC`, the guest kernel writes the CMOS RTC within a second of each clock step at boot and restore and then every 11 minutes: 26 port exits per write, about 0.1 ms on KVM and more where port exits cost more. Keeping `STA_UNSYNC` set would avoid it but report `TIME_ERROR` to applications | | No capture-time clocksource waits | Removes the harness's wait for `tsc-early` to become `tsc`: 0.73 to 0.92 s per capture on MSHV and 0.47 to 0.67 s on WHP; outside the gated metrics | -| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on prometheus28, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | +| `no_timer_check` from the Hyper-V identity, no LAPIC calibration, and no `tsc-early` window at cold boot | About 43 to 52 ms (9 to 15%) faster `cold_start_base` and other quiet cold boots on MSHV and WHP (WHP measured 31 to 55 ms on bare metal, and 61 to 71 ms at one vCPU on Azure); KVM unchanged | | Invariant TSC from the profile where the backend cannot offer it to guests (Azure MSHV and WHP) | Removes each AP's delay calibration of about 150 ms: an 8-vCPU cold boot takes 299 ms instead of 1,385 ms on Azure MSHV, and cold boots are 0.2 to 1.1 s faster at 2 to 8 vCPUs on Azure WHP | | Fixed restore work: the restore clock (0.7 to 1.1 ms), restore verification, and the backend preflight | A one-vCPU WHP restore is at parity on Azure 8370C runners (p50 −0.6 ms) and about 2 ms slower on bare metal, where no emulation cost is recovered; the counting LAPIC accounts for at most 0.75 ms of it | -| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On prometheus30 (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first: the second takes about 6 ms and each later one 14 to 15 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | -| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (prometheus32 idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | +| MSHV root-driver costs | Host characteristics, which the time ABI and the legacy path pay alike, not ABI obligations. On the bare-metal MSHV host (hypervisor 26100.30000) three root-driver costs make up 50 of the 70 ms of a one-vCPU, 512 MiB restore and 130 of 150 ms at eight vCPUs: registering guest RAM, which the driver maps no-access 4 KiB at a time (28 ms; 9.5 ms on Azure); creating each application processor after the first: the second takes about 6 ms and each later one 14 to 15 ms, because `MSHV_CREATE_VP` pre-deposits 90 pages, a VP needs about 172 there, and each failed creation deposits one more page (21 and 78 ms at 4 and 8 vCPUs, and the same at cold boot; 1.9 ms at 8 vCPUs on Azure); and the first touch of each 2 MiB chunk after resume (about 22 ms, see below). The first two serialize on the driver's partition mutex. Prefaulting the snapshot's resident chunks before registration (`MADV_POPULATE_WRITE`) cuts the third to 4 to 10 ms but costs about 45 ms itself, so the remedies are driver changes (larger deposits, and read-only mappings for read faults or a smaller fault granule for private file mappings) or guest RAM filled from the snapshot in parallel. The frozen synchronized TSC set adds 40 to 170 µs for 1 to 4 VPs and no per-VP serialized work | +| Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (the bare-metal KVM host idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | -| v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (bare-metal prometheus32: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on prometheus32 against the pre-profile head, and about +25 ms on azure-kvm-5. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, prometheus32 cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate measures azure-kvm-5. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | +| v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | +| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate measures nested Azure KVM. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | -| No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on `azure-kvm-5`; violation events still print | +| No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | Expected wins are tracked separately and do not relax the gate. @@ -1763,9 +1764,9 @@ wiring v7m on the production kernel (`vmlinux-lockstep`), with at least | Backend | Boot | Capture | Restore | Measured on | | --- | --- | --- | --- | --- | -| KVM | 6 + 2 | 1 + 0.4 | 6.5 + 1.5 | prometheus32 and `azure-kvm-5`, 128 MiB | -| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | prometheus30 and `azure-azlinux-5`, 128 and 512 MiB | -| WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | prometheus28 and the 8370C and 8573C runners, 512 MiB | +| KVM | 6 + 2 | 1 + 0.4 | 6.5 + 1.5 | Bare metal and nested Azure 8370C, 128 MiB | +| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | Bare metal and nested Azure 8573C, 128 and 512 MiB | +| WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | Bare metal and the Azure 8370C and 8573C runners, 512 MiB | Fleet validation records every sample's `cpu_us` and checks each one against these budgets. CI reports each phase against its budget @@ -1782,7 +1783,7 @@ their first touches of restored RAM: - On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The restore budget leaves room for one more stall above every maximum. -- On `azure-kvm-5`, KVM's restore checks fault in the file-backed restored +- On nested Azure KVM, KVM's restore checks fault in the file-backed restored RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs, and maxima of 5.15 ms at 1 vCPU and 13.14 ms at 8, over 72 restores. @@ -1793,7 +1794,7 @@ touch, which doesn't depend on the guest's memory size. **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), -measured on prometheus32 (KVM) with the kvm agent's attribution harness; the +measured on the bare-metal KVM host with an attribution harness; the flip requires it. At the integration head (`e06ed4d4a`), up to the backend preflight at one vCPU, a restore starts 0.32 ms and a cold boot 0.30 ms faster than `e7ec0ca6c`. The only added work is the worker's time ABI CPU @@ -1812,7 +1813,7 @@ or an effective CPUID: and the recomputed effective CPUID with the binary record, about 5 µs; - `verify_support` costs 2 to 10 µs of CPU time, plus the backend's surface. -On prometheus32, selection, the pinned-record and generation checks, the +On bare-metal KVM, selection, the pinned-record and generation checks, the effective CPUID merge, `verify_support`, and the unlisted-entry check cost about 26 µs together (cold medians over 30 fresh processes). The codecs and digests (`CpuProfile::{encode, decode, digest, digest_string, @@ -1829,17 +1830,17 @@ The backends add their own CPUID work: Every CPUID instruction exits in a Hyper-V root, so a full enumeration costs 70 to 100 µs on the bare-metal roots and 0.23 to 0.31 ms on Azure's nested roots. Skipping the range took MSHV's from 96 to 49 µs on - prometheus30 and from 305 to 220 µs nested, more than its share of the + bare metal and from 305 to 220 µs nested, more than its share of the queries suggests. KVM takes its surface from `KVM_GET_SUPPORTED_CPUID` and does not enumerate. MSHV enumerates on a thread that partition creation starts, so the read leaves the start path; starting the thread costs about 50 µs in a root partition, and keeping its channel until the partition drops avoids freeing memory there, which costs 12 to 18 µs. - MSHV reads VP 0's report in one rep `HvCallGetVpCpuidValues`: 73 µs p50 on - prometheus30 and 51 µs on azure-azlinux-5, against 0.51 and 0.82 ms with + bare metal and 51 µs nested, against 0.51 and 0.82 ms with one call per entry. WHP has no batched read: its report takes 57 to 60 native reads, one call each, which with its one host enumeration cost - 0.81 ms on prometheus28 and 1.48 to 1.86 ms on Azure (medians). Reading + 0.81 ms on bare metal and 1.48 to 1.86 ms on Azure (medians). Reading them on a few threads before any VP runs is the allowed lever if a gate needs it. @@ -1889,8 +1890,9 @@ backend's budget for that phase in [Performance expectations](#performance-expectations-and-acceptance-gate); CI reports it without gating on it. The fleet runs them on the hosts our SSH account can use, -which for KVM are prometheus32 and `azure-kvm-5` and for MSHV prometheus30 -and `azure-azlinux-5`: the account cannot open `/dev/kvm` or `/dev/mshv` on +which for KVM and for MSHV are a bare-metal Skylake-SP host and a nested +Azure VM (8370C for KVM, 8573C for MSHV): the account cannot open `/dev/kvm` +or `/dev/mshv` on the other KVM and MSHV runners, which only CI jobs exercise. On the Azure WHP runners, a cold boot with an empty command line, which logs to the console, keeps `tsc` as its clocksource (#292: 6 of 6 runs with the time ABI, against @@ -1907,15 +1909,15 @@ can use as the fleet restore matrix. | --- | --- | --- | | Same host, immediate | All | Restored; `DOWNTIME_UTC` clear | | Same host, downtime of 30 s (over the 21 s RCU stall timeout), at 1 and 8 vCPUs, with and without `rcupdate.rcu_expedited=1` | All | Restored; no RCU stall; `rcu_stall_count` stays 0 | -| Same host under DVFS load | prometheus32 | Restored | +| Same host under DVFS load | The bare-metal KVM host | Restored | | Simulated host reboot: hooks `force-utc-downtime`, `boot-id-mismatch`, and `dest-rate-offset-ppm=+200`, then `-200` | One host per backend | Restored; `DOWNTIME_UTC` and `TEST_HOOKS` set; rate deviation reported | | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | -| Across VMs of one generation | `azure-windows-1` to `-2`; `azure-windows-3` to `-4`. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | -| Across generations | `azure-windows-1` (8370C) to `-3` (8573C); prometheus32 to `azure-kvm-5` (KVM); prometheus30 to `azure-azlinux-5` (MSHV) | `E_CPU_GENERATION` | -| Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | `azure-windows-1` to `-4`, `azure-azlinux-5` | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | -| Across backends | prometheus32 (KVM) to prometheus30 (MSHV) | `E_BACKEND_MISMATCH` | +| Across VMs of one generation | Between two 8370C Azure WHP runners, and between two 8573C ones. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | +| Across generations | An 8370C Azure WHP runner to an 8573C one; bare-metal Skylake-SP KVM to nested Azure 8370C KVM; bare-metal Skylake-SP MSHV to nested Azure 8573C MSHV | `E_CPU_GENERATION` | +| Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | +| Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | | Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | @@ -2018,7 +2020,7 @@ Migration impact: - CPUID becomes profile-defined, so guests can lose host features that no profile of their generation pins. - Hosts that fail qualification cannot run microVMs until replaced. - `azure-azlinux-2` stays out of rotation and unqualified because our + One 8370C MSHV runner stays out of rotation and unqualified because our account cannot run guests there. - Per-PR CI captures and restores on the same runner. Same-generation cross-VM restore is validated by the fleet restore matrix on WHP only; no From e979ec9ad92f2b230355ce490ebabe7b1abfb439 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 11:44:53 -0700 Subject: [PATCH 148/188] doc: describe skipped polls in the wall-clock convergence A second Azure WHP run, on an 8573C runner whose host UTC runs -12.89 ppm against its counter, settled about 3.2 ms ahead of host UTC and reached 4.1 ms after two skipped polls in a row: each skipped poll lets the offset grow at the remaining rate for another 64 s. Say so, with the sign rule, and add the convergence check to the restore matrix: no step, synchronized, the frequency moving toward the host's UTC rate r, and the offset within 512 s times r plus 1 ms, which allows four skipped polls in a row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 59d8cd10d..54b66bc67 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1460,9 +1460,15 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: at which the host's own UTC clock runs against its TSC, which host time synchronization slews by a few ppm. The PLL's frequency integrator absorbs that rate over about an hour at the 64 s cadence. Until then the offset - settles near 256 s times the rate: 1.76 ms behind host UTC at the 6.88 ppm - of an Azure WHP runner, and an expected 4.2 ms at the 16.5 ppm of the - bare-metal WHP host. Both are far below the step threshold. + settles near 256 s times the rate, behind host UTC when the host's UTC + runs fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of + an Azure 8370C WHP runner, about 3.2 ms ahead at the -12.89 ppm of an + 8573C one, and an expected 4.2 ms at the 16.5 ppm of the bare-metal WHP + host. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets the offset grow at the + remaining rate for another 64 s, and the nested WHP runners, whose sample + uncertainty approaches the 50 µs bound, skip some: the 8573C runner + reached 4.1 ms after two skipped polls in a row. All of these are far + below the step threshold. The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. @@ -1914,6 +1920,7 @@ can use as the fleet restore matrix. | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | +| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate `r` measured over 60 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward `r`; and `abs(theta)` at most 512 s × `abs(r)` + 1 ms at every accepted poll, which leaves room for four skipped polls in a row. Skipped polls are reported, and fail the case only through that bound | | Across VMs of one generation | Between two 8370C Azure WHP runners, and between two 8573C ones. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | An 8370C Azure WHP runner to an 8573C one; bare-metal Skylake-SP KVM to nested Azure 8370C KVM; bare-metal Skylake-SP MSHV to nested Azure 8573C MSHV | `E_CPU_GENERATION` | | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | From 03102c041c4709d22d035db6ffbbc34a1f4871a7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 12:09:03 -0700 Subject: [PATCH 149/188] doc: describe the post-resume work that runs after readiness The p6 performance gate showed that part of a restored guest's post-resume work runs after readiness: the timers that came due during the downtime, including those due during the VMM's restore setup and the discipline's poll, the clock step and the kernel's RTC write after it, and the restore checks from 150 ms after the acknowledgement. Readiness doesn't wait for it, but a request made in that first second does. Describe it in the performance section, with the measured effect: an exit requested right at readiness on the Azure WHP runners takes 7 to 9 ms longer than on the legacy path, 3.5 ms longer 0.3 s later, and no longer from 1 s on; on nested Azure KVM a network restore after the shell-snapshot benchmarks is about 12 ms slower. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 54b66bc67..ef9ba06f5 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1760,6 +1760,24 @@ Expected effects: | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | +**The first second after a restore.** Not all post-resume work finishes +before readiness. At resume the guest runs every timer that came +due during the downtime: the kernel's, those that came due during the VMM's +restore setup (the legacy path stopped guest time during that setup, so its +guest ran them later), and the discipline's poll when the downtime outlasts +the poll period. Restore repair also steps the clock, which the kernel +follows with its RTC write, and the restore checks start 150 ms after the +acknowledgement. Readiness doesn't wait for this work, but a request made +during it does, most where first touches of restored RAM are expensive. On +the Azure WHP runners, an exit requested right at readiness takes 7 to 9 ms +longer than on the legacy path +(`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, and +24% on +bare metal). Requested 0.3 s after readiness it takes 3.5 ms longer, and +from 1 s on the difference is within noise. On nested Azure KVM, a network +restore that follows the shell-snapshot benchmarks' captures and restores +is about 12 ms (8.5%) slower and is no slower on its own, which fits the +same work faulting in restored RAM from a colder host page cache. + Expected wins are tracked separately and do not relax the gate. **`cpu_us` budgets.** Each backend budgets the CPU time of each phase's From ac774f12d671d2dce99962dea77c7e3d4fdeaec1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 12:37:10 -0700 Subject: [PATCH 150/188] doc: correct the gate's metric count and the multi-vCPU cold-boot row CI's one-vCPU result has 34 metrics since the two 64 MiB shell metrics were retired, so the gate covers 37 cells: all 34 one-vCPU metrics and the 512 MiB shell restore at 2, 4, and 8 vCPUs, not 36 one-vCPU metrics. The gate's cold boots are all one-vCPU, so it doesn't measure multi-vCPU cold boot on nested Azure KVM. Replace that claim with the boot-only comparison against the dev base: no significant difference at 2, 4, and 8 vCPUs, which excludes the +25 ms seen before the L3 fix. At 8 vCPUs the VMM's gain and the guest's cost cancel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index ef9ba06f5..6c954b90e 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1731,13 +1731,13 @@ Routing CI jobs by generation labels is optional future work. ## Performance expectations and acceptance gate **Gate.** Per backend and vCPU count, no p50 is worse than the base-branch -median by more than `max(5%, 2 ms)`. The gate covers all 36 one-vCPU metrics -and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. It is measured on -the CI Azure platforms; bare-metal results are informational. The bare-metal -KVM host idles at 800 MHz (`intel_pstate` powersave without HWP), which -inflates its absolute restore latency (`restored_ms` p50 at one vCPU 39 ms, -against 23 ms at full clock) and its `cpu_us`; the arms of an A/B share the -effect. +median by more than `max(5%, 2 ms)`. The gate covers 37 cells: all 34 +one-vCPU metrics and `shell_snapshot_restore_512_mib` at 2, 4, and 8 vCPUs. +It is measured on the CI Azure platforms; bare-metal results are +informational. The bare-metal KVM host idles at 800 MHz (`intel_pstate` +powersave without HWP), which inflates its absolute restore latency +(`restored_ms` p50 at one vCPU 39 ms, against 23 ms at full clock) and its +`cpu_us`; the arms of an A/B share the effect. Expected effects: @@ -1755,7 +1755,7 @@ Expected effects: | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (the bare-metal KVM host idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate measures nested Azure KVM. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | +| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | From 14bd7706cd68b2d2b9dfcc6238ea1bce67aaeae1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 12:40:54 -0700 Subject: [PATCH 151/188] doc: note the KVM paravirtual features the Hyper-V identity forgoes Under the Hyper-V identity, a KVM guest has no KVM paravirtual IPIs, TLB flushes, halt polling, or steal time. Say so in the hypervisor identity section, with the borderline, unattributed measurement that may reflect it: the guest's side of an 8-vCPU cold boot on nested Azure KVM was about 8 ms slower than the dev base's, offset by the VMM's gain. Hyper-V's own enlightenments could restore paravirtual IPIs and TLB flushes, but they need the guest's CONFIG_HYPERV and hypercall support in OpenVMM, so they are left for after v1. Point the multi-vCPU cold-boot row at the note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 6c954b90e..ca2f50029 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -254,9 +254,19 @@ Because the kernel no longer checks cross-vCPU TSC consistency, the Side effects on Linux are limited to selecting the Hyper-V platform: an unknown-NMI handler is registered, `pv_info.name` is `Hyper-V`, the hard-lockup detector is off, and `no_timer_check` is set. No KVM -paravirtual feature (kvmclock, PV EOI, steal time, async page faults, PV TLB -flush, or PV spinlocks) exists on any backend; `sched_clock` and the vDSO use -the raw TSC. +paravirtual feature (kvmclock, PV EOI, PV IPIs, PV TLB flush, PV spinlocks, +halt polling, steal time, or async page faults) exists on any backend; +`sched_clock` and the vDSO use the raw TSC. + +On KVM, the identity therefore forgoes the paravirtual IPIs, TLB flushes, +halt polling, and steal time that a KVM signature would give a multi-vCPU +guest. That may be why the guest's side of an 8-vCPU cold boot on nested +Azure KVM measured 8.2 [0.4, 15.9] ms slower than the `dev` base's; the +difference is unattributed, and the VMM's gain offsets it (see +[Performance expectations](#performance-expectations-and-acceptance-gate)). +Hyper-V's own enlightenments could restore paravirtual IPIs and TLB flushes +under the identity, but they need the guest's `CONFIG_HYPERV` and hypercall +support in OpenVMM, so they are left for after v1. ## Backend obligations @@ -1755,7 +1765,7 @@ Expected effects: | Boot check and daemon start | Off the cold-boot path. Before shell-ready only the initial time sample and clock step remain (`C12`: one port write and four reads). The other checks and the daemon start run after shell-ready, and `nvx-time status` reports their wall time (`elapsed_us`, not budgeted) and CPU time (`cpu_us`), which each backend budgets per phase (see `cpu_us` budgets below). Wiring v7 measured medians of 1.6 to 3.7 ms at boot and 0.4 to 1.0 ms at restore on MSHV, 3.5 to 5.1 ms and 2.8 to 5.3 ms on KVM (the bare-metal KVM host idles at 800 MHz, which slows the delayed checks), and 2.6 to 4.8 ms and 7.7 to 11.5 ms on WHP at 1 to 8 vCPUs, with captures under 3 ms everywhere | | Counting LAPIC instead of TSC-deadline on KVM | Different timer-programming exits; covered by the gate | | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | -| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel. The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | +| The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel; the guest's part may be the KVM paravirtual features that the identity forgoes (see [Hypervisor identity](#hypervisor-identity)). The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | From fb35745c187b16446385f04ca2e7c1368bb5d1e3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 13:45:33 -0700 Subject: [PATCH 152/188] doc: record the bare-metal WHP convergence measurement The wall-clock convergence bullet predicted 4.2 ms for the bare-metal WHP host. A 12-minute run after a restore measured it: the host's UTC ran +16.07 ppm against its counter, and the offset settled at 4.06 ms behind host UTC, against 4.11 ms predicted by 256 s times the rate. With the two Azure runners, all three hosts now match the model. Also use a proper minus sign for the 8573C runner's rate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index ca2f50029..40e1b13bc 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1470,15 +1470,15 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: at which the host's own UTC clock runs against its TSC, which host time synchronization slews by a few ppm. The PLL's frequency integrator absorbs that rate over about an hour at the 64 s cadence. Until then the offset - settles near 256 s times the rate, behind host UTC when the host's UTC - runs fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of - an Azure 8370C WHP runner, about 3.2 ms ahead at the -12.89 ppm of an - 8573C one, and an expected 4.2 ms at the 16.5 ppm of the bare-metal WHP - host. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets the offset grow at the - remaining rate for another 64 s, and the nested WHP runners, whose sample - uncertainty approaches the 50 µs bound, skip some: the 8573C runner - reached 4.1 ms after two skipped polls in a row. All of these are far - below the step threshold. + settles near 256 s times the rate, behind host UTC when the host's UTC runs + fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of an + Azure 8370C WHP runner, about 3.2 ms ahead at the −12.89 ppm of an 8573C + one, and 4.06 ms behind at the +16.07 ppm of the bare-metal WHP host, + against 4.11 ms predicted. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets the + offset grow at the remaining rate for another 64 s, and the nested WHP + runners, whose sample uncertainty approaches the 50 µs bound, skip some: the + 8573C runner reached 4.1 ms after two skipped polls in a row. All of these + are far below the step threshold. The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. From 7b208db365f6a6ab851cd0a09e1a1ba39f977787 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 14:27:09 -0700 Subject: [PATCH 153/188] doc: say a rolled-back capture leaves guest time running No backend stops guest time for a capture: the guest TSC keeps running while the VPs are stopped, so a capture that rolls back leaves the guest's clocks continuous and the guest sees the quiesce as elapsed time. Say so in the capture paragraph, with the bare-metal WHP and KVM measurements. In the failed-capture row, require that the status reports no clock step, note that cancel-capture records nothing in the status, and say that the parent must be unwritable to the OpenVMM process itself: root's CAP_DAC_OVERRIDE and an administrator's backup and restore privileges bypass file permissions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 40e1b13bc..8305fc0b2 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -896,6 +896,15 @@ state units are quiesced, capture: Each of these failures is a rollback-safe capture failure. +No backend stops guest time for a capture: the guest TSC keeps running while +the VPs are stopped. So a capture that rolls back leaves the guest's clocks +continuous, and the guest sees the quiesce as elapsed time, as it would a long +preemption. On the bare-metal WHP host, the guest's `CLOCK_MONOTONIC` covered +quiesce windows of 0.39 to 0.65 s within 2.2 ms of the host interval, and +`CLOCK_REALTIME` minus `CLOCK_MONOTONIC` moved by at most 0.14 µs. On the +bare-metal KVM host, `CLOCK_MONOTONIC` stayed within −0.38 to +0.40 ms of the +host over requests of 0.2 to 1.1 s. + **Machine contract changes.** `SnapshotMachineContract` retires protobuf fields 11 (`capture_wall_clock`), 12 (`tsc_frequency_hz`), 13 (`tsc_tolerance_ppm`), 14 (`cpu_contract`), 15 (`cpu_contract_sha256`), 17 @@ -1954,7 +1963,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back; then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0`, the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From f3f27b06e8e611a06a86c342563d693451a63b7f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 14:36:47 -0700 Subject: [PATCH 154/188] doc: add the existing-destination companion to the failed-capture row A request whose destination already exists is rejected by preflight before quiesce: OpenVMM logs that preflight failed and the guest continues, with no capture anchor and no rollback, and the guest side is the same as for the post-quiesce failure. It passes on the bare-metal MSHV and WHP hosts. Add it to the failed-capture row, and say to probe the unwritable parent with the same credentials OpenVMM gets: an elevated token's backup and restore privileges pass a deny ACE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 8305fc0b2..061eaa638 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1963,7 +1963,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores. The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From 6ddda5d7c58cbbf8146f35c6f95ad8630c91e970 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 15:22:36 -0700 Subject: [PATCH 155/188] doc: add the MSHV hosts to the wall-clock convergence A 12-minute run after a restore on the bare-metal MSHV host, whose UTC ran +14.82 ppm against its TSC, left the offset 3.50 ms behind host UTC and still approaching the 3.79 ms plateau that 256 s times the rate predicts. On a nested Azure MSHV host, whose own UTC its host was slewing, the offset followed the changing rate. Both passed the convergence check. Add them to the Convergence bullet, so it covers MSHV as well as WHP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 061eaa638..5b2935c10 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1483,8 +1483,11 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of an Azure 8370C WHP runner, about 3.2 ms ahead at the −12.89 ppm of an 8573C one, and 4.06 ms behind at the +16.07 ppm of the bare-metal WHP host, - against 4.11 ms predicted. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets the - offset grow at the remaining rate for another 64 s, and the nested WHP + against 4.11 ms predicted. The bare-metal MSHV host, at +14.82 ppm, was 3.50 + ms behind after 12 minutes and still approaching its 3.79 ms plateau. Where + the host's own UTC is being slewed, as on a nested Azure MSHV host, the + offset follows the changing rate. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets + the offset grow at the remaining rate for another 64 s, and the nested WHP runners, whose sample uncertainty approaches the 50 µs bound, skip some: the 8573C runner reached 4.1 ms after two skipped polls in a row. All of these are far below the step threshold. From 79c09f257b32da825a5dbe0d01d9bdfd80a8b341 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 15:42:01 -0700 Subject: [PATCH 156/188] doc: start the convergence checks at the first discipline poll The boot check and restore repair take their own time sample before they step the clock, and offset_ns keeps that pre-step theta until the first discipline poll: at cold boot it is the RTC's whole-second error, a few hundred milliseconds. Read literally, the convergence row's bound would fail every cold boot on that value. Say that the checks start at the first discipline poll, that skipped polls are reported with their count and longest run, and what offset_ns holds before the first poll. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 5b2935c10..419ed724c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1519,7 +1519,7 @@ every change, and `nvx-time status` reads it. The sandbox agent bind-mounts | `last_downtime_ns` | `D` of the last restore | | `last_downtime_source` | `monotonic` or `utc` | | `synchronized` | 1 when the last accepted sample is at most 128 s old | -| `offset_ns` | Last measured `theta` | +| `offset_ns` | Last measured `theta`. Until the first discipline poll, the boot check's or restore repair's `theta` from before its step | | `uncertainty_ns` | Last accepted `epsilon` | | `frequency_ppb` | Current kernel frequency correction | | `samples`, `rejected_samples` | Sample counters | @@ -1960,7 +1960,7 @@ can use as the fleet restore matrix. | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | -| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate `r` measured over 60 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward `r`; and `abs(theta)` at most 512 s × `abs(r)` + 1 ms at every accepted poll, which leaves room for four skipped polls in a row. Skipped polls are reported, and fail the case only through that bound | +| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate `r` measured over 60 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward `r`; and `abs(theta)` at most 512 s × `abs(r)` + 1 ms at every accepted discipline poll, which leaves room for four skipped polls in a row. The checks start at the first discipline poll: the boot check's and restore repair's own samples are taken before their steps (at cold boot, the RTC's whole-second error). Skipped polls are reported with their count and longest run, and fail the case only through that bound | | Across VMs of one generation | Between two 8370C Azure WHP runners, and between two 8573C ones. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | An 8370C Azure WHP runner to an 8573C one; bare-metal Skylake-SP KVM to nested Azure 8370C KVM; bare-metal Skylake-SP MSHV to nested Azure 8573C MSHV | `E_CPU_GENERATION` | | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | From 1869b4a376bb6759b4bd6001b40c015d1aca76f7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 16:27:38 -0700 Subject: [PATCH 157/188] doc: take the convergence bound's rate over the 512 s before each poll The convergence row measured the host's UTC rate over 60 s alongside the window. On a host whose own UTC is being slewed, such as a nested Azure host, the per-minute rate swings through zero (from -7.6 to +8.5 ppm in one window), so a minute near zero would collapse the bound to about 1 ms and fail a healthy window. Sample the rate every 10 s instead, and bound each accepted poll with the mean rate over the 512 s before it, which covers the PLL's memory, or over the window so far before then. Judge the frequency against the window's mean rate, and report the per-minute spread. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 419ed724c..029b3ed29 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1960,7 +1960,7 @@ can use as the fleet restore matrix. | Simulated rate beyond tolerance: `dest-rate-offset-ppm=+251` | One host per backend | `E_TSC_RATE_TOLERANCE` | | Downtime bounds: `downtime-add-s=2592001`; `force-utc-downtime` with `utc-offset-ms=-`, `n` above the elapsed time | One host per backend | `E_DOWNTIME_EXCESSIVE`; `E_DOWNTIME_NEGATIVE` | | Sample uncertainty: `sample-delay-us=200`; then `sample-delay-us=3000` on restore and on cold boot | One host per backend | Restored and running, with `last_sample_error=G_SAMPLE_UNCERTAIN` in `nvx-time status`; `G_REPAIR_SAMPLE` (195); `G_CONFORMANCE_C12` (193) | -| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate `r` measured over 60 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward `r`; and `abs(theta)` at most 512 s × `abs(r)` + 1 ms at every accepted discipline poll, which leaves room for four skipped polls in a row. The checks start at the first discipline poll: the boot check's and restore repair's own samples are taken before their steps (at cold boot, the RTC's whole-second error). Skipped polls are reported with their count and longest run, and fail the case only through that bound | +| Wall-clock convergence, for 12 minutes after a restore and after a cold boot, with the host's UTC rate sampled every 10 s alongside (Linux: `CLOCK_REALTIME` against `CLOCK_MONOTONIC_RAW`; Windows: UTC against QPC) | One host per backend | No step after the initial one; `synchronized=1`; `frequency_ppb` moving toward the window's mean rate; and, at every accepted discipline poll, `abs(theta)` at most 512 s × `abs(r)` + 1 ms, where `r` is the host's mean UTC rate over the 512 s before that poll (before then, over the window so far, at least 60 s), which leaves room for four skipped polls in a row. The checks start at the first discipline poll: the boot check's and restore repair's own samples are taken before their steps (at cold boot, the RTC's whole-second error). Skipped polls are reported with their count and longest run, and fail the case only through that bound. The spread of the host's per-minute rate is reported too: where the host's own UTC is being slewed, a minute's rate can swing through zero, so no single minute sets the bound | | Across VMs of one generation | Between two 8370C Azure WHP runners, and between two 8573C ones. KVM and MSHV have no usable pair of one generation, so the simulated host reboot covers their cross-host path | Restored | | Across generations | An 8370C Azure WHP runner to an 8573C one; bare-metal Skylake-SP KVM to nested Azure 8370C KVM; bare-metal Skylake-SP MSHV to nested Azure 8573C MSHV | `E_CPU_GENERATION` | | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | From a162ceb0ad35e45811a7997cbc2fe91ed965e2c0 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 16:44:25 -0700 Subject: [PATCH 158/188] doc: add the scratch-device variant to the failed-capture row The failed-capture case ran without a scratch device, so the snapshot agent's freezer and scratch barriers never engaged on a rolled-back capture. Add a variant with a paired scratch device mounted and a workload in the container cgroup: after the failed request the cgroup is thawed, the scratch filesystem takes a write with fsync within 1 s, the workload keeps running, and the real capture and restore with the scratch device pass. It passed 3 of 3 on the bare-metal MSHV host at one vCPU. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 029b3ed29..2f770486e 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1966,7 +1966,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce | Each bare-metal host at 1 and 8 vCPUs, 3 times each; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores. The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce. Scratch variant: the failed request in a VM with a paired scratch device mounted at `/run/nvx/scratch` and a workload in the `container` cgroup, so that the snapshot agent's freezer and scratch barriers engage | Each bare-metal host at 1 and 8 vCPUs, 3 times each, and the scratch variant on one bare-metal host at 1 vCPU, 3 times; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores. The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same. In the scratch variant, the failed request also leaves the workload cgroup thawed (`cgroup.freeze` reads 0), the scratch filesystem taking a write with `fsync` within 1 s, and the workload running, and the real capture and restore with the scratch device pass | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From 86cf15b4fecead4b69f1f625e14cc900b2bc31ef Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 16:55:08 -0700 Subject: [PATCH 159/188] doc: price WHP's first touch of restored memory The monotonic-downtime row said a first touch costs about 1 ms on WHP. A traced comparison on the bare-metal WHP host priced it: each restored 4 KiB page that the guest first touches costs about 50 us of wall time, a nested page fault of which about 11 us is in the hypervisor and the rest is resolved by the root, unseen by the VMM. The lazily registered 2 MiB chunks cost about 90 us each and all finish before readiness, and a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 us. Say so, and describe WHP's restore checks as first-touching copy-on-write RAM page by page rather than as faulting in lazily registered RAM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 2f770486e..c361846d8 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1779,7 +1779,7 @@ Expected effects: | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | | The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel; the guest's part may be the KVM paravirtual features that the identity forgoes (see [Hypervisor identity](#hypervisor-identity)). The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | -| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs), where a first touch costs about 1 ms. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | +| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). On WHP each restored 4 KiB page that the guest first touches costs about 50 µs of wall time on bare metal: a nested page fault, about 11 µs of it in the hypervisor and the rest resolved by the root, which the VMM never sees. WHP's lazily registered 2 MiB chunks add about 90 µs each, all before readiness, and a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 µs. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | **The first second after a restore.** Not all post-resume work finishes @@ -1820,12 +1820,12 @@ against these budgets. CI reports each phase against its budget above covers latency. The checks' CPU time also counts the host stalls of their first touches of restored RAM: -- WHP's restore checks fault in its lazily registered copy-on-write RAM, - which the workload would otherwise do. Their cost grows with guest memory: - medians are 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and - 8 vCPUs on Azure. So WHP's budgets come from 512 MiB guests, which bound - smaller ones. Its Azure maxima are 17.03 ms at boot with 2 vCPUs and - 42.20 ms at restore with 4. +- WHP's restore checks first-touch restored copy-on-write RAM page by page, + about 50 µs each on bare metal, which the workload would otherwise do. Their + cost grows with guest memory: medians are 1.3 to 2.4 times higher at 512 MiB + than at 128 MiB at 4 and 8 vCPUs on Azure. So WHP's budgets come from 512 + MiB guests, which bound smaller ones. Its Azure maxima are 17.03 ms at boot + with 2 vCPUs and 42.20 ms at restore with 4. - On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The restore budget leaves room for one more stall above every maximum. From 9300d7d075fbcf7376068c1846b0e3742ce8de5d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 17:17:04 -0700 Subject: [PATCH 160/188] doc: add WHP's Azure page cost and end-to-end parity The first-touch cost of restored memory on WHP was priced on bare metal at about 50 us of wall time per 4 KiB page. The same measurement on the Azure WHP runners gives about 70 to 95 us per page. Add that to the monotonic-downtime row and to the cpu_us note on WHP's restore checks. The slower exit right after a restore is also not a slower run overall: from process launch to exit, a WHP restore under the time ABI is at most about 2 ms slower than on the legacy path, and up to 6 ms faster on bare metal. Say so in the paragraph about the first second after a restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 47 +++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index c361846d8..2d9d9872f 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1779,26 +1779,27 @@ Expected effects: | v1 profiles without `ITS_NO` | Linux's ITS mitigation at boot, about 6 ms of a one-vCPU cold boot where the host's KVM advertises `ITS_NO` (the bare-metal KVM host: +6.3 [5.5, 7.3] ms against the same build booted with `indirect_target_selection=off`). The gated Azure KVM, MSHV, and WHP guests were already mitigated and are unchanged; restore is unaffected | | The profile's CPU view at 2 or more vCPUs on KVM | None at the integration head. Before the L3 fix, multi-vCPU cold boots were slower: +9.2 [5.0, 13.7] ms at 2 vCPUs on the bare-metal KVM host against the pre-profile head, and about +25 ms on nested Azure KVM. OpenVMM's `CPUID.4` reported a private L3 cache per vCPU, which made Linux's cache-info initialization wait a 10 ms tick for CPU 1 on many boots. With the L3 shared by the socket, the bare-metal host's cold boots match the pre-profile head: −0.3 [−3.2, +2.7], −0.0 [−2.1, +2.0], and −1.0 [−3.6, +1.7] ms at 2, 4, and 8 vCPUs. The gate's cold boots are one-vCPU; a boot-only comparison against the `dev` base on nested Azure KVM found no significant difference: +1.7 [−2.5, +6.9] ms at 2 vCPUs, +3.7 [−5.4, +13.6] at 4, and +3.3 [−7.0, +11.5] at 8, which excludes the earlier +25 ms. At 8 vCPUs, the time ABI's VMM boots 9.5 [4.6, 16.4] ms faster and its guest 8.2 [0.4, 15.9] ms slower, which cancel; the guest's part may be the KVM paravirtual features that the identity forgoes (see [Hypervisor identity](#hypervisor-identity)). The msr driver's per-CPU hotplug callback (`msr_init`) waits one or two ticks on multi-vCPU boots, as it did before the profiles; it competes for CPU with the asynchronous initramfs unpack and is off the critical path | | Restore work before the acknowledgement | Only the packet read, the clock set, CPU and memory activation, and entropy and identity repair, in one helper process. The restore checks, the RCU release, and the deferred `C7` start at `SCHED_IDLE` 150 ms after the acknowledgement, after the readiness path | -| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). On WHP each restored 4 KiB page that the guest first touches costs about 50 µs of wall time on bare metal: a nested page fault, about 11 µs of it in the hypervisor and the rest resolved by the root, which the VMM never sees. WHP's lazily registered 2 MiB chunks add about 90 µs each, all before readiness, and a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 µs. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | +| Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). On WHP each restored 4 KiB page that the guest first touches costs about 50 µs of wall time on bare metal and about 70 to 95 µs on the Azure runners: a nested page fault, about 11 µs of it in the hypervisor on bare metal and the rest resolved by the root, which the VMM never sees. WHP's lazily registered 2 MiB chunks add about 90 µs each, all before readiness, and on bare metal a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 µs. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | -**The first second after a restore.** Not all post-resume work finishes -before readiness. At resume the guest runs every timer that came -due during the downtime: the kernel's, those that came due during the VMM's -restore setup (the legacy path stopped guest time during that setup, so its -guest ran them later), and the discipline's poll when the downtime outlasts -the poll period. Restore repair also steps the clock, which the kernel -follows with its RTC write, and the restore checks start 150 ms after the -acknowledgement. Readiness doesn't wait for this work, but a request made -during it does, most where first touches of restored RAM are expensive. On -the Azure WHP runners, an exit requested right at readiness takes 7 to 9 ms -longer than on the legacy path -(`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, and +24% on -bare metal). Requested 0.3 s after readiness it takes 3.5 ms longer, and -from 1 s on the difference is within noise. On nested Azure KVM, a network -restore that follows the shell-snapshot benchmarks' captures and restores -is about 12 ms (8.5%) slower and is no slower on its own, which fits the -same work faulting in restored RAM from a colder host page cache. +**The first second after a restore.** Not all post-resume work finishes before +readiness. At resume the guest runs every timer that came due during the +downtime: the kernel's, those that came due during the VMM's restore setup +(the legacy path stopped guest time during that setup, so its guest ran them +later), and the discipline's poll when the downtime outlasts the poll period. +Restore repair also steps the clock, which the kernel follows with its RTC +write, and the restore checks start 150 ms after the acknowledgement. +Readiness doesn't wait for this work, but a request made during it does, most +where first touches of restored RAM are expensive. On the Azure WHP runners, +an exit requested right at readiness takes 7 to 9 ms longer than on the legacy +path (`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, and +24% +on bare metal). Requested 0.3 s after readiness it takes 3.5 ms longer, and +from 1 s on the difference is within noise. End to end, from process launch to +exit, a WHP restore under the time ABI is at most about 2 ms slower than on +the legacy path, and up to 6 ms faster on bare metal. On nested Azure KVM, a +network restore that follows the shell-snapshot benchmarks' captures and +restores is about 12 ms (8.5%) slower and is no slower on its own, which fits +the same work faulting in restored RAM from a colder host page cache. Expected wins are tracked separately and do not relax the gate. @@ -1821,11 +1822,11 @@ above covers latency. The checks' CPU time also counts the host stalls of their first touches of restored RAM: - WHP's restore checks first-touch restored copy-on-write RAM page by page, - about 50 µs each on bare metal, which the workload would otherwise do. Their - cost grows with guest memory: medians are 1.3 to 2.4 times higher at 512 MiB - than at 128 MiB at 4 and 8 vCPUs on Azure. So WHP's budgets come from 512 - MiB guests, which bound smaller ones. Its Azure maxima are 17.03 ms at boot - with 2 vCPUs and 42.20 ms at restore with 4. + about 50 µs each on bare metal and 70 to 95 µs on Azure, which the workload + would otherwise do. Their cost grows with guest memory: medians are 1.3 to + 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure. So + WHP's budgets come from 512 MiB guests, which bound smaller ones. Its Azure + maxima are 17.03 ms at boot with 2 vCPUs and 42.20 ms at restore with 4. - On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The restore budget leaves room for one more stall above every maximum. From ddbc8c4cf229cd72f534330b3497d219b5eb86e1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 17:36:53 -0700 Subject: [PATCH 161/188] doc: attribute the post-restore exit cost to cold process paths A joint measurement of the WHP exit-at-readiness cell, with per-VP page counts, finds that none of the time ABI's post-resume work runs while the exit is slower. The cause is the restore's process layout: an untiered restore with nothing to activate starts no process before readiness, while the legacy path's status read forked and executed one, first-touching the guest kernel's fork, exec, and exit paths. Under the time ABI the first process after readiness touches about 80 to 110 more pages instead. One small process started ahead of the exit removes 64 to 92% of the difference. Say so, replace the earlier wording that attributed the delay to the post-resume timer work, and retitle the paragraph to Right after a restore, since the cost follows the first process rather than a time window. Give the size of the restore-setup window whose timers the legacy path ran later: about 400 ms on nested Azure KVM and 7 to 12 ms on bare-metal KVM, but only 1 to 2 ms on WHP, too short to explain the extra pages. Also correct the end-to-end figure: on the Azure WHP runners, the launch-to- exit difference isn't significant, and on bare metal readiness comes 9.5 ms earlier. The KVM network restore that follows the shell-snapshot benchmarks takes no major faults in either arm, so drop the colder-page-cache explanation. What the time ABI's restore adds is about 4% more KVM exits, including 26 port I/O exits, mostly the packet read, and more retried EPT faults and remote TLB flushes, whose cost on a nested runner depends on the host's state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 52 +++++++++++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 18 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 2d9d9872f..ca7cd4312 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1782,24 +1782,40 @@ Expected effects: | Monotonic time that includes the downtime | Right after resume, the guest kernel runs the timer work that came due during the downtime (kworkers, softirqs, and RCU) and first-touches cold pages doing so: at one vCPU about 1.4 ms on MSHV, 2.2 ms on KVM, and 17 ms on WHP (26 ms at two vCPUs). On WHP each restored 4 KiB page that the guest first touches costs about 50 µs of wall time on bare metal and about 70 to 95 µs on the Azure runners: a nested page fault, about 11 µs of it in the hypervisor on bare metal and the rest resolved by the root, which the VMM never sees. WHP's lazily registered 2 MiB chunks add about 90 µs each, all before readiness, and on bare metal a WHP restore's latency after the VPs are released is roughly the pages it touches times 53 µs. It falls inside the gated restore metric whatever the helper's priority. On MSHV the cost is per 2 MiB chunk: the root driver resolves the first touch of each chunk with 512 copy-on-write copies of the private snapshot mapping, about 1.1 ms. A one-vCPU restore touches about 24 chunks, 25 ms of its restore on both the time ABI and the legacy path, and the time ABI's catch-up adds about one chunk | | No time ABI console lines in production | Each console byte costs one or two port exits, about 10 to 12 µs on bare-metal KVM and an estimated 20 to 45 µs nested on Azure. Without the boot marker (about 110 bytes) and the restore marker (about 117 bytes), boots and restores save 1 to 5 ms, enough to fail the gate on nested Azure KVM; violation events still print | -**The first second after a restore.** Not all post-resume work finishes before -readiness. At resume the guest runs every timer that came due during the -downtime: the kernel's, those that came due during the VMM's restore setup -(the legacy path stopped guest time during that setup, so its guest ran them -later), and the discipline's poll when the downtime outlasts the poll period. -Restore repair also steps the clock, which the kernel follows with its RTC -write, and the restore checks start 150 ms after the acknowledgement. -Readiness doesn't wait for this work, but a request made during it does, most -where first touches of restored RAM are expensive. On the Azure WHP runners, -an exit requested right at readiness takes 7 to 9 ms longer than on the legacy -path (`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, and +24% -on bare metal). Requested 0.3 s after readiness it takes 3.5 ms longer, and -from 1 s on the difference is within noise. End to end, from process launch to -exit, a WHP restore under the time ABI is at most about 2 ms slower than on -the legacy path, and up to 6 ms faster on bare metal. On nested Azure KVM, a -network restore that follows the shell-snapshot benchmarks' captures and -restores is about 12 ms (8.5%) slower and is no slower on its own, which fits -the same work faulting in restored RAM from a colder host page cache. +**Right after a restore.** Not all post-resume work finishes before readiness. +At resume the guest runs every timer that came due during the downtime: the +kernel's, those that came due during the VMM's restore setup (the legacy path +stopped guest time during that setup, so its guest ran them later; it left out +about 400 ms of a nested Azure KVM restore and 7 to 12 ms on bare-metal KVM, +but only 1 to 2 ms on WHP), and the discipline's poll when the downtime +outlasts the poll period. Restore repair also steps the clock, which the +kernel follows with its RTC write, and the restore checks start 150 ms after +the acknowledgement. Readiness doesn't wait for this work. + +The first process that a restored guest starts pays a separate cost. An +untiered restore with nothing to activate starts no process before readiness, +while the legacy path's status read forked and executed one there, which +first-touched the guest kernel's fork, exec, and exit paths. Under the time +ABI the first process after readiness touches them instead, about 80 to 110 +more pages than on the legacy path, which matters where first touches of +restored RAM are expensive. On the Azure WHP runners, the first process after +readiness, an exit the gate requests, took 6.7 and 8.8 ms longer than on the +legacy path (`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, +and +24% on bare metal). None of the time ABI's post-resume work, the kernel's +RTC write included, runs in that window; one small process started ahead of +the exit removes 64 to 92% of the difference. On bare metal the cost is a +shift: readiness comes 9.5 ms earlier, and readiness plus the exit takes +4.7 ms less. On the Azure runners readiness isn't earlier, and the difference +from process launch to exit isn't significant (point estimates from −1.9 to ++4.0 ms). + +On nested Azure KVM, a network restore that follows the shell-snapshot +benchmarks' captures and restores is up to about 12 ms (8.5%) slower, by an +amount that varies from run to run, and is no slower on its own. Neither arm +takes major faults. The time ABI's restore makes about 4% more KVM exits, 26 +of them port I/O, mostly the restore packet's read, and more retried EPT +faults and remote TLB flushes, each of which costs more on a nested runner, +depending on the host's state. Expected wins are tracked separately and do not relax the gate. From 97ce2b3c18aa0cd5a7eca096053b51e7787e175b Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 17:58:19 -0700 Subject: [PATCH 162/188] doc: disclose that the static nvx-exit offsets the first command's cost The gate's teardown metric times the first process after a restore, an exit requested right at readiness, which under the time ABI pays for the guest kernel's cold fork, exec, and exit paths: a plain restore starts no process before readiness, where the legacy path's status read did. Without a faster nvx-exit the time ABI would be about 5 to 9 ms (31 to 37%) slower there on the Azure WHP runners, which would fail the gate. nvx-exit is now a single static executable, which makes every guest exit faster and clears the metric: it then measured 20 to 30% below the legacy path. Disclose that this offsets the cost rather than removing it: the first command after a plain restore still pays it on nested Azure WHP, while readiness itself is unchanged and the launch-to-exit difference, about +2 to +4 ms or less, isn't significant. The remedy, a working-set prefetch, is a follow-up after v1. Also split the nested KVM network restore's difference into the part before the first good network probe (5.7 ms) and the teardown after it (2.9 ms). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 50 ++++++++++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index ca7cd4312..b0e0f37ef 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1792,28 +1792,40 @@ outlasts the poll period. Restore repair also steps the clock, which the kernel follows with its RTC write, and the restore checks start 150 ms after the acknowledgement. Readiness doesn't wait for this work. -The first process that a restored guest starts pays a separate cost. An -untiered restore with nothing to activate starts no process before readiness, -while the legacy path's status read forked and executed one there, which -first-touched the guest kernel's fork, exec, and exit paths. Under the time -ABI the first process after readiness touches them instead, about 80 to 110 -more pages than on the legacy path, which matters where first touches of -restored RAM are expensive. On the Azure WHP runners, the first process after -readiness, an exit the gate requests, took 6.7 and 8.8 ms longer than on the -legacy path (`openvmm_snapshot_restore_guest_exit_teardown`: +31% and +37%, -and +24% on bare metal). None of the time ABI's post-resume work, the kernel's -RTC write included, runs in that window; one small process started ahead of -the exit removes 64 to 92% of the difference. On bare metal the cost is a -shift: readiness comes 9.5 ms earlier, and readiness plus the exit takes -4.7 ms less. On the Azure runners readiness isn't earlier, and the difference -from process launch to exit isn't significant (point estimates from −1.9 to -+4.0 ms). +The first process that a restored guest starts pays a separate cost. A plain +(untiered) restore with nothing to activate starts no process before +readiness, while the legacy path's status read forked and executed one there, +which first-touched the guest kernel's fork, exec, and exit paths. Under the +time ABI the first process after readiness touches them instead, about 80 to +110 more pages than on the legacy path, which matters where first touches of +restored RAM are expensive: on the Azure WHP runners, the first process after +readiness takes about 5 to 9 ms longer than on the legacy path, while +readiness itself is unchanged. None of the time ABI's post-resume work, the +kernel's RTC write included, runs while it does, and one small process started +ahead of it removes 64 to 92% of the difference. On bare metal the cost is a +shift: readiness comes 9.5 ms earlier, and readiness plus the first process +takes 4.7 ms less. On the Azure runners, from process launch to exit, the +difference is about +2 to +4 ms or less and isn't significant. + +The gate's teardown metric (`openvmm_snapshot_restore_guest_exit_teardown`) +times such a first process: an exit requested right at readiness. Without a +faster `nvx-exit`, the time ABI would be about 5 to 9 ms (31 to 37%) slower +there than the legacy path on the Azure WHP runners, which would fail the +gate, and 24% slower on bare metal. The metric passes because `nvx-exit` is a +single static executable, which makes every guest exit faster, with or without +the time ABI: with it, the metric measured 20% below the legacy path on bare +metal and 30% below on an Azure 8573C runner. That offsets the cost in the +gate rather than removing it: the first command after a plain restore still +pays it on nested Azure WHP. The remedy, a working-set prefetch that populates +the pages a restore touches before its vCPUs run, is a follow-up after v1. On nested Azure KVM, a network restore that follows the shell-snapshot benchmarks' captures and restores is up to about 12 ms (8.5%) slower, by an -amount that varies from run to run, and is no slower on its own. Neither arm -takes major faults. The time ABI's restore makes about 4% more KVM exits, 26 -of them port I/O, mostly the restore packet's read, and more retried EPT +amount that varies from run to run, and is no slower on its own. Of the +difference, 5.7 ms comes before the first good network probe and 2.9 ms in the +teardown after it; the static `nvx-exit` doesn't measurably change it. Neither +arm takes major faults. The time ABI's restore makes about 4% more KVM exits, +26 of them port I/O, mostly the restore packet's read, and more retried EPT faults and remote TLB flushes, each of which costs more on a nested runner, depending on the host's state. From 719fe8c1827f1161e25dcd85e879601fe001188a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 12:46:38 -0700 Subject: [PATCH 163/188] doc: count the 34 one-vCPU benchmark metrics left after #116 #116 retired the 64 MiB shell snapshot pair, shell_snapshot_cold_64_mib and shell_snapshot_restore_64_mib, from the benchmark (d368fe79) and the collector (2cfeab77), but the benchmark and usage docs kept the old counts and memory lists. Since then the shared workload suite has 21 metrics, the lifecycle merge gives 29, and the device operation-rate merge gives the final 34 one-vCPU metrics. run-benchmark already requires 34, and dev's latest results have exactly those 34 on all three CI platforms. Correct the counts and the shell-snapshot and snapshot-profile memory lists, and note that data/*.csv keeps the retired pair's results only as history, the last ones from b8912df. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 24 ++++++++++++++---------- doc/usage.md | 4 ++-- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 2bbd9b579..a5a754cb9 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -1,9 +1,9 @@ # Benchmark The supported OpenVMM benchmark coordinator provides acceptance and diagnostic suites, a -23-metric microVM non-Python workload suite, and five device operation-rate metrics on +21-metric microVM non-Python workload suite, and five device operation-rate metrics on Linux/KVM, Linux/MSHV, and Windows/WHP. At one vCPU, CI combines those workloads with eight -128 MiB shell lifecycle metrics and reports all 36 median (p50) values. At 2, 4, and 8 vCPUs, CI records only +128 MiB shell lifecycle metrics and reports all 34 median (p50) values. At 2, 4, and 8 vCPUs, CI records only `shell_snapshot_restore_512_mib`. Latency and resident-memory metrics are lower-is-better; throughput and operation-rate metrics are higher-is-better. @@ -64,10 +64,10 @@ python3 scripts/nvx.py benchmark --suite e2e --backend kvm --platform linux-kvm- Run the complete performance suite with: ```console -# Linux/KVM: run all 23 metrics and write collector-compatible logs +# Linux/KVM: run all 21 metrics and write collector-compatible logs python3 scripts/nvx.py benchmark --suite performance --backend kvm --platform linux-kvm-baremetal --processors 1 --runs 5 --virtfs-runs 3 --skip-build --output-dir data/runs/linux-kvm-baremetal/microvm-v2/1vcpu -# Linux/MSHV: run all 23 metrics +# Linux/MSHV: run all 21 metrics python3 scripts/nvx.py benchmark --suite performance --backend mshv --platform linux-mshv-baremetal --processors 1 --runs 5 --virtfs-runs 3 --skip-build --output-dir data/runs/linux-mshv-baremetal/microvm-v2/1vcpu # Windows/WHP @@ -238,15 +238,15 @@ python3 scripts/nvx.py performance collect --platform linux-kvm-baremetal --comm | Benchmark | Command | Description | | --- | --- | --- | | Shell lifecycle | `benchmark --suite e2e` | Measures cold start, snapshot generation, snapshot restore, teardown, and peak RSS using a shell-ready guest. | -| All supported non-Python workloads | `benchmark --suite performance` | Runs 23 metrics and writes collector-compatible logs. | +| All supported non-Python workloads | `benchmark --suite performance` | Runs 21 metrics and writes collector-compatible logs. | | Device operation rates | `benchmark --suite device-io` | Measures five random storage and UDP round-trip operation rates with resumable raw attempts. | | Cold start | `benchmark --suite cold-start` | Measures a quiet shell-ready baseline and isolated one-parameter kernel command-line variants. | | Virtual file system | `benchmark --suite virtfs` | Measures live host-directory throughput and verifies host-to-guest plus guest-to-host visibility in one running VM. | -| Shell snapshot | `benchmark --suite shell-snapshot` | Compares cold boot with lifecycle-aligned snapshot restore at 64, 128, 256, and 512 MiB. | +| Shell snapshot | `benchmark --suite shell-snapshot` | Compares cold boot with lifecycle-aligned snapshot restore at 128, 256, and 512 MiB. | | Shell snapshot restore | `benchmark --suite shell-snapshot-restore` | Captures an unmeasured lifecycle-aligned snapshot and measures only restore latency for the selected memory sizes. | | Restore-time vCPU activation | `benchmark --suite snapshot-restore-vcpu --processors 8` | Restores one boot-online-1, capacity-8 snapshot at online targets 1/2/4/8 and reports latency plus peak RSS. | | Network snapshot | `benchmark --suite network-snapshot` | Compares a network-ready cold boot with snapshot restore and verifies gateway connectivity. | -| Snapshot lifecycle profile | `benchmark --suite snapshot-profile` | Retains raw capture and restore phase records and summarizes 64/128/256/512/1024 MiB warm/cold restores. | +| Snapshot lifecycle profile | `benchmark --suite snapshot-profile` | Retains raw capture and restore phase records and summarizes 128/256/512/1024 MiB warm/cold restores. | | Virtio device restore profile | `benchmark --suite device-restore-profile` | Verifies active and driver-unbound deferred restore for console, network, and virtio-fs; emits standalone diagnostic JSON and raw logs. | ## Kernel command lines @@ -420,6 +420,10 @@ This lifecycle-aligned methodology supersedes the earlier pre-banner `shellsnap` capture. Historical `shell_snapshot_restore_*` values produced by that methodology are not comparable with newly collected values. +The 64 MiB pair, `shell_snapshot_cold_64_mib` and +`shell_snapshot_restore_64_mib`, was retired in #116. The platform CSVs in +`data/` keep its results only as history; the last ones are `b8912df`'s. + | Metric | Description | | --- | --- | | `shell_snapshot_cold_128_mib` | OpenVMM launch to a shell-ready guest with 128 MiB of memory. | @@ -507,10 +511,10 @@ successful packets. ## CI collection `python scripts/nvx.py performance collect --require-shared-suite` rejects a workload result -unless it contains exactly the 23 shared metrics. Supplying `--lifecycle-input` requires and merges -the eight lifecycle metrics, producing a 31-metric one-vCPU result. CI collects the five device +unless it contains exactly the 21 shared metrics. Supplying `--lifecycle-input` requires and merges +the eight lifecycle metrics, producing a 29-metric one-vCPU result. CI collects the five device operation-rate metrics from their separate raw-log directory and merges them by ABI, processor -count, commit, and metric, producing the final 36-metric ABI-2 one-vCPU result. Higher-vCPU +count, commit, and metric, producing the final 34-metric ABI-2 one-vCPU result. Higher-vCPU collection uses `--require-shell-snapshot-restore-512`, which requires exactly `shell_snapshot_restore_512_mib` plus canonical one-warmup/five-sample metadata for a 2-, 4-, or 8-vCPU guest. A `device-io` directory is recognized from metadata and must contain exactly its five diff --git a/doc/usage.md b/doc/usage.md index 9eeeee0fc..fc86728d9 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -589,9 +589,9 @@ reject incomplete inputs for their respective workload sets. `--require-shell-snapshot-restore-512` accepts only the canonical 512 MiB restore metric from a 2-, 4-, or 8-vCPU run. `--lifecycle-input` validates and merges a 128 MiB, guest-exit `e2e` JSON -result, producing the 31-metric microVM CI result. A directory whose metadata +result, producing the 29-metric microVM CI result. A directory whose metadata selects `device-io` is collected as five additional ABI-2, one-vCPU `ops/s` -metrics; CI merges them into a 36-metric one-vCPU result. +metrics; CI merges them into a 34-metric one-vCPU result. `--summary` writes the p50 table plus lifecycle min/max/sample-count and RSS diagnostics. #### `performance validate-openvmm` From 32c8fff6edc15135ae9386bce19e6fc097b59006 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 14:55:50 -0700 Subject: [PATCH 164/188] doc: note CI's capture samples and the rollback exemption The evidence paragraph listed the newest checks that status queries report as boot after a cold boot and restore after a restore. Since snapshot-core's released-request checks (04354c44), its no-destination VM also reports one capture check, so CI's evidence carries a capture sample for each snapshot-core run. The spec exempts from the capture budget the first capture after a rolled-back capture in the same VM process: OpenVMM's pre-staging RAM flush makes the guest re-dirty each flushed page once. CI never makes such a capture. Each VM process sends OpenVMM at most one snapshot request, since snapshot-tier's mismatched tier is rejected inside the guest, and a failed capture fails its scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 9b464434e..7e53c9767 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -120,11 +120,15 @@ line and adds it to the GitHub job summary, because CI keeps the guest logs only for failed jobs: the number of warp probe runs with their worst `max_abs_offset_ns` and `max_backward_ns`, the count and `elapsed_us` range of the newest check that each `nvx-time status` query reported (boot after a cold -boot, restore after a restore), the exhaustive check's summary, and the +boot, restore after a restore, and capture after `snapshot-core`'s released +request), the exhaustive check's summary, and the `restore-downtime` stall counts. Where the guest reports a check's CPU time (`cpu_us`), the line also gives its range and how many checks exceed the backend's [CPU-time budget](design/time-abi.md#performance-expectations-and-acceptance-gate) -for their phase, which the spec sets from guest measurements. +for their phase, which the spec sets from guest measurements. The spec exempts +the first capture after a rolled-back capture in the same VM process; CI never +makes one, because each VM process sends OpenVMM at most one snapshot request +and a failed capture fails its scenario. `CHECK_CPU_BUDGET_US` in `scripts/nvx_tools/time_abi.py` holds the budgets, one per backend and phase. `elapsed_us` is wall time, including waits behind the workload, and has no budget. CI gates on neither: the performance gate is the From 70efa17f7a66bf3e02af360891e9ea79dc8eeeb7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 18:43:40 -0700 Subject: [PATCH 165/188] openvmm: bump to the time ABI v1 PR head Advance nanvix/openvmm nvx/time-abi-v1 from e06ed4d4a to 0303eefca, a fast-forward of documentation, comments and test code only: - Guide: state that an OpenVMM build with a different effective CPUID needs recapture; - cpu_profile, virt_mshv, virt_whp: name the test host fixtures by their CPU rather than by host name; - virt: describe the test hosts generically in comments. Outside the Guide and comments, every changed line is test code, so the built OpenVMM behaves as at e06ed4d4a. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index e06ed4d4a..0303eefca 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit e06ed4d4a0033200b0583a12b76a84dec2c511e1 +Subproject commit 0303eefcaebffb345b2c40d49f900ca5fdc8256d From 1bb1ac847f275f5695c380816895cae7e1303578 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 22:30:54 -0700 Subject: [PATCH 166/188] Install the time ABI tools in the Azure Linux image Ownership exception: guest image changes belong to the guest agent. The ci agent makes this commit on its behalf, as the orchestrator approved, because the guest's parity commit didn't arrive by the deadline for the time ABI's merge of dev (Azure Linux 3 guest, #110). The Azure Linux initramfs is built in its own Dockerfile stage rather than by build.py, so it missed the time ABI's musl helpers. Its init runs /sbin/nvx-time boot first, so without the agent every Azure Linux boot would power off with status 193. The stage now builds nvx-time and nvx-exit with build.py's MUSL_STATIC_HELPERS recipe, and nvx-time-probe with TIME_PROBE_CFLAGS. All three binaries are byte-identical to the Alpine image's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docker/Dockerfile | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 3a5eb998d..2dea1185a 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -65,8 +65,11 @@ RUN set -eux; \ for helper in nvx-reseed nvx-mmio-write nvx-port-io nvx-console-pending nvx-managed-agent; do \ cc -static -Os -s -Wall -Wextra -Werror -o "/out/helpers/$helper" "/repo/guest/common/$helper.c"; \ done; \ - musl-gcc -static -Os -s -Wall -Wextra -Werror -o /out/helpers/nvx-exit \ - /repo/guest/common/nvx-exit.c; \ + for helper in nvx-time nvx-exit; do \ + musl-gcc -static -Os -s -Wall -Wextra -Werror -o "/out/helpers/$helper" "/repo/guest/common/$helper.c"; \ + done; \ + musl-gcc -std=gnu11 -static -O2 -pthread -s -Wall -Wextra -Werror -o /out/helpers/nvx-time-probe \ + /repo/guest/common/nvx-time-probe.c; \ cc -nostdlib -static -Os -fno-builtin -fno-pie -fno-stack-protector -no-pie \ -Wl,--build-id=none -Wl,-z,noexecstack -s -o /out/helpers/nvx-device-io \ /repo/guest/common/nvx-device-io.c From ca1ccbfc15b41cbf5b2805269cc16d40ea0ff118 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 19:44:14 -0700 Subject: [PATCH 167/188] doc: state how each restore path uses the restore entropy A plain untiered restore, with nothing to activate, has never reseeded the guest CRNG. Before the time ABI the snapshot agent left the restore packet unread on that path; now nvx-time reads packet v4 and writes its 64 entropy bytes to /run/nvx/restore-entropy for the caller. The spec said only "the existing entropy and identity repair for the tier". Add a "Restore entropy" subsection with the per-path table: the clone policy, and untiered restores that activate processors or memory, reseed before the acknowledgement; instance-checkpoint refreshes only the generation ID, by design; a plain untiered restore leaves the entropy to its caller. State that v1 leaves each path unchanged, and record the open policy question for the plain untiered path. Documentation only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index b0e0f37ef..0d015b8d8 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1365,9 +1365,9 @@ acknowledgement. `restore_generation=g` in the same state-file update, so that `nvx-time status` waits for this restore. 9. Run the existing processor and memory activation. -10. Run the existing entropy and identity repair for the tier. The RTC-based - wall-clock refresh is removed; `instance-checkpoint` restores also get - step 8. +10. Run the existing entropy and identity repair for the restore path + ([restore entropy](#restore-entropy)). The RTC-based wall-clock refresh + is removed; `instance-checkpoint` restores also get step 8. 11. Defer the restore checks to step 13: `C1`, `C2`, and `C5` on newly onlined CPUs; `C3` for CPU 0; `C6`; and `C10`. They do not wait, and a failure still emits its violation event and powers off with status 193. @@ -1421,6 +1421,32 @@ always run at normal priority, and stall suppression stays set until the release. Repair failures emit a violation event and power off with status 195. +### Restore entropy + +Restoring a snapshot also restores the guest kernel's CRNG state, so every +restore packet carries 64 bytes of fresh host entropy. Step 10 uses them +according to the restore path; the tiers follow the clone and resume +policies of [Time and entropy](snapshot-and-restore.md#time-and-entropy): + +| Restore path | Guest CRNG | Machine identity | +| --- | --- | --- | +| `platform` or `workload-start` tier (clone policy) | Reseeded before the acknowledgement: `nvx-reseed` credits the 64 bytes to the kernel entropy pool (`RNDADDENTROPY`) and forces a CRNG reseed (`RNDRESEEDCRNG`) | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | +| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for the clone policy | Refreshed as for the clone policy | +| `instance-checkpoint` tier (resume policy) | Not reseeded, by design: only the generation ID is refreshed | Preserved | +| Untiered, with nothing to activate (`online_vp_count` and `memory_range_count` both 0) | Not reseeded: `nvx-time` writes the 64 bytes to `/run/nvx/restore-entropy` for the caller of `nvx-snapshot` | Preserved | + +Each path's handling is unchanged from before the time ABI. v1 changes only +the delivery: the agent used to read packets v1 to v3 a byte at a time, and +on the last path it did not read the packet, leaving it to its caller. It +now reads packet v4 on every restore, for the time fields, and checks `g` +and the generation-ID change first (step 7). + +On the last path, VMs restored from one snapshot share the snapshot's CRNG +state until something reseeds it, such as a caller that credits +`/run/nvx/restore-entropy` with `nvx-reseed`. Whether that path should +reseed before readiness, given that one snapshot can be restored more than +once, is an open policy question; v1 keeps the existing behavior. + ### RCU grace period release Stall suppression is required: on KVM, restores after 30 s of downtime From f54279d007c1e5843ef80bcc36ce51b7fea56620 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 15:38:08 -0700 Subject: [PATCH 168/188] doc: make the cpu_us budget checks mechanical Fleet validation records every sample's cpu_us and checks it against the budgets after the run. Replace the earlier host-noise wording with a mechanical rule: an over-budget sample is a triaged outlier only if a rerun of the same case on the same host, with at least 30 samples, has none over the budget and no other sample of that case is over it; its elapsed_us exceeds the rerun's nearest-rank 99th percentile; and its VM shows no violation, snapshot error, or status 193 to 195. Any other over-budget sample is a budget failure, fixed in the budget or the code. Exempt exactly one sample: the first capture after a rolled-back capture in the same VM. OpenVMM's pre-staging RAM flush, the same with or without the time ABI, leaves that RAM clean, and the next capture's checks pay once to re-dirty the pages they write. Define the headroom as a budget of at least 1.2 times the largest sample of fleet validation, over bare metal and Azure and both kernels, without triaged outliers or the exempt sample. Raise KVM's capture budget to 1.5 + 0.4 ms, its restore budget to 7 + 2 ms, MSHV's boot budget to 3 + 1 ms, WHP's boot budget to 25 + 1.5 ms, and WHP's capture budget to 1.2 + 0.4 ms, which those maxima need: MSHV's debug kernel reaches 4.94 ms at a 4-vCPU boot, and a WHP rerun's 1-vCPU capture 0.99 ms on a nested runner. Record the outcomes: KVM's 2-vCPU captures over the earlier budget were budget failures, fixed by its new capture budget; MSHV's 47.1 ms debug-kernel boot, WHP's 3.91 ms 1-vCPU capture on a nested runner, and KVM's 14.05 ms 4-vCPU restore on nested Azure KVM are triaged outliers; and WHP's 21.84 ms 2-vCPU boot on a nested runner was a budget failure, because the boot checks' elapsed time there varies too much to show it stretched, fixed by its new boot budget. MSHV's next capture after a rollback costs what the rolled-back one did, and KVM's little more; the exemption still applies on every backend, because the flush does. Add the bare-metal KVM host to the Convergence bullet, so that it has a measured example from every backend: at -5.98 ppm, the guest was 1.45 ms ahead of host UTC after 12 minutes, near its 1.53 ms plateau. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 105 +++++++++++++++++++++++++++++++---------- 1 file changed, 81 insertions(+), 24 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 0d015b8d8..52e13a2c0 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1244,10 +1244,11 @@ the background cost to the workload. It includes interrupts that arrive while the checks run, and it scales with the host processor's clock: a host idling at a low frequency runs the checks more slowly, and because the Hyper-V identity gives the guest no steal-time accounting, host preemption is -charged to the checks too. The fleet matrix driver checks every `cpu_us` -sample against the backend's budget for the phase in -[Performance expectations](#performance-expectations-and-acceptance-gate) -during validation; CI reports it without gating on it. +charged to the checks too. Fleet validation records every `cpu_us` sample +and checks it after the run against the backend's budget for the phase, with +one exempt sample and a triage rule for samples over budget, both in +[Performance expectations](#performance-expectations-and-acceptance-gate); +CI reports it without gating on it. A failed check prints no marker: it emits the violation event with code `G_CONFORMANCE_` (`G_KERNEL_WX` for `K1`) and powers off with status 193, @@ -1509,14 +1510,15 @@ The daemon keeps `CLOCK_REALTIME` on host UTC through the kernel's PLL: fast and ahead when it runs slow: 1.76 ms behind at the +6.88 ppm of an Azure 8370C WHP runner, about 3.2 ms ahead at the −12.89 ppm of an 8573C one, and 4.06 ms behind at the +16.07 ppm of the bare-metal WHP host, - against 4.11 ms predicted. The bare-metal MSHV host, at +14.82 ppm, was 3.50 - ms behind after 12 minutes and still approaching its 3.79 ms plateau. Where - the host's own UTC is being slewed, as on a nested Azure MSHV host, the - offset follows the changing rate. A skipped poll (`G_SAMPLE_UNCERTAIN`) lets - the offset grow at the remaining rate for another 64 s, and the nested WHP - runners, whose sample uncertainty approaches the 50 µs bound, skip some: the - 8573C runner reached 4.1 ms after two skipped polls in a row. All of these - are far below the step threshold. + against 4.11 ms predicted. The bare-metal MSHV host, at +14.82 ppm, was + 3.50 ms behind after 12 minutes and still approaching its 3.79 ms plateau. + The bare-metal KVM host, at −5.98 ppm, was 1.45 ms ahead after 12 minutes, + near its 1.53 ms plateau. Where the host's own UTC is being slewed, as on a + nested Azure MSHV host, the offset follows the changing rate. A skipped poll + (`G_SAMPLE_UNCERTAIN`) lets the offset grow at the remaining rate for + another 64 s, and the nested WHP runners, whose sample uncertainty + approaches the 50 µs bound, skip some: the 8573C runner reached 4.1 ms after + two skipped polls in a row. All of these are far below the step threshold. The discipline never powers off the guest: a host wall-clock step is followed, not reported as a violation. @@ -1859,18 +1861,20 @@ Expected wins are tracked separately and do not relax the gate. **`cpu_us` budgets.** Each backend budgets the CPU time of each phase's checks (boot, capture, and restore) at a + b·(n − 1) ms for n online vCPUs, -per sample. A budget is the larger of the bare-metal and Azure maxima of -wiring v7m on the production kernel (`vmlinux-lockstep`), with at least -20% headroom: +per sample. A budget has at least 20% headroom over the largest sample that +fleet validation recorded for its backend, phase, and vCPU count: it is at +least 1.2 times that maximum, taken over bare metal and Azure and over the +production and debug kernels (wiring v7m), without triaged outliers or the +exempt sample below: | Backend | Boot | Capture | Restore | Measured on | | --- | --- | --- | --- | --- | -| KVM | 6 + 2 | 1 + 0.4 | 6.5 + 1.5 | Bare metal and nested Azure 8370C, 128 MiB | -| MSHV | 3 + 0.75 | 1 + 0.4 | 2.5 + 0.5 | Bare metal and nested Azure 8573C, 128 and 512 MiB | -| WHP | 20 + 1.5 | 1 + 0.4 | 35 + 6 | Bare metal and the Azure 8370C and 8573C runners, 512 MiB | +| KVM | 6 + 2 | 1.5 + 0.4 | 7 + 2 | Bare metal and nested Azure 8370C, 128 MiB | +| MSHV | 3 + 1 | 1 + 0.4 | 2.5 + 0.5 | Bare metal and nested Azure 8573C, 128 and 512 MiB | +| WHP | 25 + 1.5 | 1.2 + 0.4 | 35 + 6 | Bare metal and the Azure 8370C and 8573C runners, 128 and 512 MiB | Fleet validation records every sample's `cpu_us` and checks each one -against these budgets. CI reports each phase against its budget +against these budgets after the run. CI reports each phase against its budget (`_cpu_over_budget`) but does not gate on it, because the A/B gate above covers latency. The checks' CPU time also counts the host stalls of their first touches of restored RAM: @@ -1879,20 +1883,73 @@ their first touches of restored RAM: about 50 µs each on bare metal and 70 to 95 µs on Azure, which the workload would otherwise do. Their cost grows with guest memory: medians are 1.3 to 2.4 times higher at 512 MiB than at 128 MiB at 4 and 8 vCPUs on Azure. So - WHP's budgets come from 512 MiB guests, which bound smaller ones. Its Azure - maxima are 17.03 ms at boot with 2 vCPUs and 42.20 ms at restore with 4. + WHP's restore budget comes from 512 MiB guests, which bound smaller ones; + its Azure maximum is 42.20 ms with 4 vCPUs. Its boot and capture budgets + come from two 128 MiB samples on the nested runners: a 21.84 ms boot with + 2 vCPUs on an 8573C runner (see below) and a 0.99 ms capture with 1 vCPU on + an 8370C one. - On MSHV, each first touch of a 2 MiB chunk of restored RAM stalls the toucher about 1.1 ms on bare metal (see MSHV root-driver costs). The restore budget leaves room for one more stall above every maximum. - On nested Azure KVM, KVM's restore checks fault in the file-backed restored - RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs, and maxima - of 5.15 ms at 1 vCPU and 13.14 ms at 8, over 72 restores. + RAM 4 KiB at a time: medians of 3.4 to 7.6 ms at 1 to 8 vCPUs over 72 + restores, and fleet validation maxima of 5.53 ms at 1 vCPU and 16.16 ms at + 8. KVM's budgets come from 128 MiB guests, the size that CI and the fleet matrices run. MSHV's hold at 128 and 512 MiB alike: MSHV registers all guest RAM before the restore, and the checks pay only for the chunks they touch, which doesn't depend on the guest's memory size. +**After a rolled-back capture.** A capture flushes file-backed guest RAM +before it stages: this is OpenVMM's pre-staging RAM flush, the same with or +without the time ABI. So a capture that rolls back leaves that RAM clean, and +the guest's first write to each page traps once to re-dirty it. The next +capture's checks pay that for the pages they write: on the bare-metal WHP +host, 0.91 to 1.23 ms at one vCPU and 1.33 to 1.88 ms at eight, against 0.22 +to 0.25 and 0.63 to 0.72 ms before the rollback. On bare-metal MSHV the cost +doesn't show: medians of 0.21 and 0.57 ms at one and eight vCPUs, against 0.20 +and 0.57 ms before. On bare-metal KVM it's small: 0.42 to 0.63 ms at one vCPU +and 1.52 to 2.58 ms at eight, against 0.38 to 0.46 and 2.36 to 2.51 ms after a +rejected request. Later captures are back to normal, and a request that +preflight rejects flushes nothing and costs nothing extra. So the capture +budget doesn't apply to exactly one sample, the first capture after a +rolled-back capture in the same VM, on any backend: validation reports its +`cpu_us` without checking it, and checks the next capture normally. + +**Over-budget samples.** The checks' CPU time includes host preemption (see +[Conformance checks](#conformance-checks-and-the-nvx-time-abi-marker)), so a +host can stretch a single sample past its budget. Validation accepts an +over-budget sample as a triaged outlier only if all three of these hold: + +1. **Isolated:** a rerun of the same case on the same host, with at least 30 + samples of the phase, has none over the budget, and no other sample of + that case on that host is over it. +2. **Stretched:** the sample's `elapsed_us` exceeds the nearest-rank 99th + percentile of the rerun's `elapsed_us` for the phase; host preemption + stretches the wall time along with the CPU time. +3. **Clean:** the same VM shows no violation event, no `NVX-SNAPSHOT-ERROR`, + no exit status 193 to 195, and `violations=0`. + +Any other over-budget sample is a budget failure, fixed in the budget or in +the code. Fleet validation's after-the-run check found: + +- KVM's five 2-vCPU captures on bare metal over the earlier 1 + 0.4 ms were + budget failures, fixed by the capture budget above. +- MSHV's debug-kernel 4-vCPU boot at 47.1 ms, 56.8 ms elapsed, is a triaged + outlier: a rerun of its case had none of 30 samples over, and at most 3.4 ms + elapsed. +- WHP's 1-vCPU capture at 3.91 ms on a nested 8370C runner is a triaged + outlier: a rerun of its case had none of 30 samples over, and at most + 0.99 ms elapsed. +- WHP's 2-vCPU boot at 21.84 ms on a nested 8573C runner, 51.6 ms elapsed, was + a budget failure: a rerun of its case had none of 30 samples over, but up to + 59.0 ms elapsed, because the boot checks there wait across vCPUs for a long + and variable time. The boot budget above covers it. +- KVM's 4-vCPU restore at 14.05 ms on nested Azure KVM (8370C), after + 10 minutes of downtime and 19.0 ms elapsed, is a triaged outlier: a rerun of + its case had none of 30 samples over, and at most 14.4 ms elapsed. + **Start-up budget.** The time ABI and CPU profile work of a cold boot or restore costs less than 0.5 ms over the pre-profile head (`e7ec0ca6c`), measured on the bare-metal KVM host with an attribution harness; the @@ -2021,7 +2078,7 @@ can use as the fleet restore matrix. | Backend that cannot offer invariant TSC to its guests, on a host OS that sees it | The Azure WHP runners and nested Azure MSHV | Restored; the guest has `constant_tsc` and `nonstop_tsc`; `H4` and `H6` pass | | Across backends | Bare-metal KVM to bare-metal MSHV | `E_BACKEND_MISMATCH` | | Pre-v1 snapshot | Any | `E_SNAPSHOT_VERSION` | -| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce. Scratch variant: the failed request in a VM with a paired scratch device mounted at `/run/nvx/scratch` and a workload in the `container` cgroup, so that the snapshot agent's freezer and scratch barriers engage | Each bare-metal host at 1 and 8 vCPUs, 3 times each, and the scratch variant on one bare-metal host at 1 vCPU, 3 times; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores. The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same. In the scratch variant, the failed request also leaves the workload cgroup thawed (`cgroup.freeze` reads 0), the scratch filesystem taking a write with `fsync` within 1 s, and the workload running, and the real capture and restore with the scratch device pass | +| Failed capture: in one VM, a request whose destination's parent the host made unwritable to the OpenVMM process after launch, so that creating the staging directory fails after quiesce and OpenVMM rolls back (root's `CAP_DAC_OVERRIDE` and an administrator's backup and restore privileges bypass file permissions, so drop them first and probe with the same credentials, or use an immutable or read-only mount); then, with the parent writable again and after 30 s idle, a second request. Companion: a request whose destination already exists, which preflight rejects before quiesce. Scratch variant: the failed request in a VM with a paired scratch device mounted at `/run/nvx/scratch` and a workload in the `container` cgroup, so that the snapshot agent's freezer and scratch barriers engage | Each bare-metal host at 1 and 8 vCPUs, 3 times each, and the scratch variant on one bare-metal host at 1 vCPU, 3 times; [CI's `snapshot-core`](../ci.md) exercises a request without a destination, which OpenVMM releases before preflight, and asserts that the guest continues once, that `nvx-time status` passes at `generation=0` with no restore, and that `rcu_cpu_stall_suppress` reads 0 | OpenVMM logs the rollback (`microVM snapshot failed before commit; attempting rollback`, then `microVM snapshot rollback succeeded; guest resumed`), with no time ABI code, and each failed request returns in the same VM: status bit 1 clear, `nvx-time status` passing at `generation=0` with `discontinuities` unchanged (from the capture's checks; `cancel-capture` records nothing), the saved values back at their values before the request (`rcu_cpu_stall_suppress`, and on the debug kernel `soft_watchdog` and `hung_task_timeout_secs`) with no saved-values file left, and no 193, 194, or 195 and no clock message; the second request captures, which step 2 of the snapshot agent refuses while the first request's saved values remain, and its snapshot restores; that capture's `cpu_us` is reported but not budgeted (see [`cpu_us` budgets](#performance-expectations-and-acceptance-gate)). The companion logs `microVM snapshot preflight failed; guest continues` with no capture anchor and no rollback, and its guest side is the same. In the scratch variant, the failed request also leaves the workload cgroup thawed (`cgroup.freeze` reads 0), the scratch filesystem taking a write with `fsync` within 1 s, and the workload running, and the real capture and restore with the scratch device pass | | Processor activation from one boot-online CPU to 2, 4, and 8 | All backends | Restored; warp probe passes | | Every tier and an untiered snapshot, restored more than once | All backends | Restored; each restore of a snapshot captured at `g = 0` carries `g = 1` and a new generation ID. OpenVMM cannot capture a restored process, so unit tests cover the generation arithmetic beyond one restore, including `E_GENERATION_EXHAUSTED` | From 77dc13fae880daae8b78d3459a3a5f645ddc9ae6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 19:42:42 -0700 Subject: [PATCH 169/188] time_abi: raise the KVM, MSHV and WHP cpu_us budgets The spec's final review raises five cpu_us budgets (per sample, a base plus an increment per additional online CPU): - KVM capture, from 1 + 0.4 ms to 1.5 + 0.4 ms; - KVM restore, from 6.5 + 1.5 ms to 7 + 2 ms; - MSHV boot, from 3 + 0.75 ms to 3 + 1 ms; - WHP boot, from 20 + 1.5 ms to 25 + 1.5 ms; - WHP capture, from 1 + 0.4 ms to 1.2 + 0.4 ms. KVM's boot budget, MSHV's capture and restore, and WHP's restore are unchanged. The table's comment now gives the spec's clarified basis. Every budget is at least 1.2 times the largest sample from all fleet validation (bare metal and Azure, production and debug kernels). Triaged outliers don't count, and neither does the first capture after a rolled-back capture, which the spec exempts and CI never makes. CI only reports over-budget counts and gates on none of them. The budget test pins the new values at their binding cells: - KVM capture at 1 and 8 vCPUs, 1,500 and 4,300 us; - KVM restore at 1 and 8, 7,000 and 21,000 us; - MSHV boot at 4 vCPUs, 6,000 us; - WHP boot at 2 vCPUs, 26,500 us; - WHP capture at 1 and 2, 1,200 and 1,600 us. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/nvx_tools/time_abi.py | 20 +++++++++++--------- scripts/test_time_abi.py | 18 +++++++++++------- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index d8d6bd32e..121d6f5ce 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -36,17 +36,19 @@ # doc/design/time-abi.md, "Performance expectations and acceptance gate", # "cpu_us budgets": each backend has a CPU-time (cpu_us) budget per phase # (boot, capture, and restore): a base plus an increment per additional online -# CPU, in microseconds, held per sample. These are the spec's final values: the -# larger of the bare-metal and Azure maxima of v7m on vmlinux-lockstep, with at -# least 20% headroom. This table is the one place the harness reads them. The -# checks' wall time (elapsed_us) has no budget. CI reports both and gates on -# neither: the performance gate is the A/B comparison outside CI. +# CPU, in microseconds, held per sample. These are the spec's final values, each +# at least 1.2 times the largest sample from all fleet validation (bare metal +# and Azure, production and debug kernels). Triaged outliers don't count, and +# neither does the first capture after a rolled-back capture, which the spec +# exempts and CI never makes. This table is the one place the harness reads +# them. The checks' wall time (elapsed_us) has no budget. CI reports both and +# gates on neither: the performance gate is the A/B comparison outside CI. CHECK_CPU_BUDGET_US: Mapping[str, Mapping[str, tuple[int, int]]] = { - "kvm": {"boot": (6_000, 2_000), "capture": (1_000, 400), "restore": (6_500, 1_500)}, - "mshv": {"boot": (3_000, 750), "capture": (1_000, 400), "restore": (2_500, 500)}, + "kvm": {"boot": (6_000, 2_000), "capture": (1_500, 400), "restore": (7_000, 2_000)}, + "mshv": {"boot": (3_000, 1_000), "capture": (1_000, 400), "restore": (2_500, 500)}, "whp": { - "boot": (20_000, 1_500), - "capture": (1_000, 400), + "boot": (25_000, 1_500), + "capture": (1_200, 400), "restore": (35_000, 6_000), }, } diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 9cb6d9d87..f08c994a6 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -144,15 +144,19 @@ def test_computes_the_checks_cpu_time_budget(self): {"boot", "capture", "restore"}, ) self.assertEqual(time_abi.check_cpu_budget_us("kvm", "boot", 1), 6_000) - self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 1), 6_500) - self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 8), 17_000) - self.assertEqual(time_abi.check_cpu_budget_us("mshv", "boot", 4), 5_250) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "capture", 1), 1_500) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "capture", 8), 4_300) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 1), 7_000) + self.assertEqual(time_abi.check_cpu_budget_us("kvm", "restore", 8), 21_000) + # The spec's binding MSHV boot cell: 4 vCPUs (4.94 ms measured). + self.assertEqual(time_abi.check_cpu_budget_us("mshv", "boot", 4), 6_000) self.assertEqual(time_abi.check_cpu_budget_us("mshv", "restore", 8), 6_000) - self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 2), 1_400) self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 1), 35_000) - # The spec's binding WHP cells: boot at 2 vCPUs (17.03 ms measured) - # and restore at 4 (42.20 ms). - self.assertEqual(time_abi.check_cpu_budget_us("whp", "boot", 2), 21_500) + # The spec's binding WHP cells: boot at 2 vCPUs (21.84 ms measured on an + # 8573C runner), capture at 1 (0.986 ms), and restore at 4 (42.20 ms). + self.assertEqual(time_abi.check_cpu_budget_us("whp", "boot", 2), 26_500) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 1), 1_200) + self.assertEqual(time_abi.check_cpu_budget_us("whp", "capture", 2), 1_600) self.assertEqual(time_abi.check_cpu_budget_us("whp", "restore", 4), 53_000) self.assertIsNone(time_abi.check_cpu_budget_us("hvf", "boot", 1)) self.assertIsNone(time_abi.check_cpu_budget_us("kvm", "runtime", 1)) From 2bdeafc3ed2bf6eacdabf0396e30fa2f962c5e34 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 2 Oct 2026 06:19:36 -0700 Subject: [PATCH 170/188] doc: drop the Proposed marker from the time ABI spec The time-abi-v1 branches passed validation: the two performance-gate cells they fail are attributed and disclosed in the PRs. The time ABI is therefore the machine contract, and the specification describes it as implemented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 52e13a2c0..38b086446 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -2,11 +2,9 @@ [Design index](../design.md) -**Proposed.** This document specifies NVX time ABI v1, the single -guest-visible time, timer, and clock contract of the microVM profile on KVM, -MSHV, and WHP. NVX and its OpenVMM fork (`nanvix/openvmm`) implement it on -their `time-abi-v1` integration branches, and it becomes part of the current -machine contract when they pass validation and merge. It replaces the earlier +This document specifies NVX time ABI v1, the single guest-visible time, +timer, and clock contract of the microVM profile on KVM, MSHV, and WHP, as NVX +and its OpenVMM fork (`nanvix/openvmm`) implement it. It replaces the earlier time rules of [Snapshot and restore](snapshot-and-restore.md#time-and-entropy) and the clock tokens of [Cold boot](cold-boot.md#effective-command-line). From aa75ec5de6fce29161a23333836683e1eb6ec766 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 00:56:45 -0700 Subject: [PATCH 171/188] openvmm: pin the time ABI v1 PR head rebased onto microvm/v2.1 openvmm#107 merged into microvm/v2.1 (4355c010e), so nanvix/openvmm nvx/time-abi-v1 is rebased onto it, and the cherry-pick of 4355c010e that 08612c119 carried is dropped. The new head 53c4d1ae2 has the same tree as 08612c119: the OpenVMM source is unchanged, and only the commit IDs, and so the version string of a build, differ. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 0303eefca..53c4d1ae2 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 0303eefcaebffb345b2c40d49f900ca5fdc8256d +Subproject commit 53c4d1ae2f1db503f59470418197a03fa186c0af From dd0ce434a21fcdc625efc866fe341d643feb274d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:42:30 -0700 Subject: [PATCH 172/188] harness: monitor every TCP console to its end, and query its cold boots TcpConsole.finish() drained the console's last bytes straight into the output, so a time ABI violation or failed check that arrived after the last awaited marker never reached TimeAbiMonitor. finish() now feeds the drained bytes, and the unterminated last line, through the monitor and raises what it found. Error paths, which persist the console log in a finally block, pass check=False, so the error being handled is not masked. No production TcpConsole had a monitor. Every one now gets TimeAbiMonitor(command): the managed lifecycle, the console-snapshot capture and restores, and the snapshot-tier capture, restore and gate-timeout restore. The two captures, whose guest shell is on the virtio console, ask nvx-time status there before the scenario's first input and require a passing boot line, with OpenvmmProcess's contract. Restores ask nothing, as on OpenVMM's console. The managed lifecycle has no shell on that console (init starts the managed agent), so it is only scanned, and a failed boot check still powers it off with 193. OpenvmmProcess.close() keeps scanning nothing and never raises, so a failure path's error stays visible: every scenario reaches wait(), which scans to EOF and checks the exit status, unconditionally. A test now pins that for each OpenvmmProcess context. doc/ci.md says which consoles are scanned and which cold boots are queried. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 11 +++- scripts/nvx_tools/microvm_tests.py | 58 +++++++++++++---- scripts/nvx_tools/openvmm_process.py | 96 ++++++++++++++++++++++++---- scripts/test_microvm_tests.py | 70 ++++++++++++++++++++ scripts/test_time_abi.py | 75 ++++++++++++++++++++++ 5 files changed, 285 insertions(+), 25 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 7e53c9767..888419aa9 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -26,7 +26,8 @@ one-vCPU smoke set, under the same time ABI checks; the debug-kernel jobs skip it, as they skip the Ubuntu tests. Failure logs from the NVX layer are uploaded per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM -console for the guest's [time ABI](design/time-abi.md) output. An +console for the guest's [time ABI](design/time-abi.md) output, and the tests +scan every virtio console they read over TCP the same way, to its end. An `NVX-TIME-ABI-VIOLATION` event or a failed `NVX-TIME-ABI` conformance line fails the scenario at once with the guest's code and detail. Guests keep the console quiet, because every console byte costs a port exit. Only the initial @@ -36,8 +37,12 @@ middle of a scenario. A guest prints one `NVX-TIME-ABI` line per recorded check phase (boot, then capture and restore after a restore) and a runtime line only when `/sbin/nvx-time status` asks, after waiting up to 30 s for pending checks. The test runners ask after every cold boot whose shell is on -the OpenVMM console, before any other input, and wait for the query to exit, -so the console's echo of later input cannot split its lines. The query must +a console the harness reads, the OpenVMM console or a virtio console over +TCP, before any other input, and wait for the query to exit, so the console's +echo of later input cannot split its lines. Cold boots with no shell there are +only scanned: one-shot workloads, and the managed lifecycle, where init starts +the managed agent instead of a shell. A failing boot check still powers them +off with status 193, which fails the run. The query must exit 0 with a passing `NVX-TIME-ABI` boot line, which reports ABI version 1, generation 0, a plausible TSC rate, and the backend's LAPIC rate; a missing line means the guest image or OpenVMM does not implement the ABI, a check diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 574575293..efb768c51 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -512,8 +512,10 @@ def _persist_console_log( output: bytes, log_path: Path, ) -> bytes: + # Only failure paths reach here with an open console; the monitor still + # scans the tail, but nothing is raised over the error being handled. if console is not None: - output = console.finish() + output = console.finish(check=False) log_path.write_bytes(output) return output @@ -804,6 +806,7 @@ def run_managed_lifecycle( ) ) log_path = output_dir / "managed-lifecycle.log" + guest_log_path = output_dir / "managed-lifecycle-guest.log" process: subprocess.Popen[bytes] | None = None boot_console: TcpConsole | None = None with log_path.open("wb") as log: @@ -822,7 +825,14 @@ def run_managed_lifecycle( raise RuntimeError("failed to create control capability pipe") process.stdin.write(capability) process.stdin.close() - boot_console = TcpConsole.connect(boot_console_address, timeout) + # The guest console has no shell to query: init hands the boot + # to the managed agent, which serves the control console. The + # monitor still fails the run on a violation or failed check + # printed there, and a failed boot check powers the guest off + # with status 193, which fails the exit check below. + boot_console = TcpConsole.connect( + boot_console_address, timeout, monitor=TimeAbiMonitor(command) + ) with ControlSession.connect( Path(endpoint_value), capability, timeout ) as session: @@ -1002,11 +1012,14 @@ def run_managed_lifecycle( ) if not all(report["teardown"].values()): raise RuntimeError("managed lifecycle reported incomplete teardown") + # Keep the guest log, then fail on anything the monitor found + # in the console's tail. + guest_log_path.write_bytes(boot_console.finish(check=False)) + boot_console.finish() + boot_console = None finally: if boot_console is not None: - (output_dir / "managed-lifecycle-guest.log").write_bytes( - boot_console.finish() - ) + guest_log_path.write_bytes(boot_console.finish(check=False)) if process is not None and process.poll() is None: process.terminate() try: @@ -3319,7 +3332,14 @@ def run_console_snapshot( output_dir / "console-snapshot-capture-process.log", ) as process: try: - console = TcpConsole.connect(address, timeout) + # The guest's shell is on this virtio console, so it answers + # the cold boot's status query there. + console = TcpConsole.connect( + address, + timeout, + monitor=TimeAbiMonitor(capture_command), + time_abi_status=True, + ) console.wait_for(BOOT_MARKER, timeout) console.wait_for(b"/ # ", timeout) console.send_bytes( @@ -3361,12 +3381,17 @@ def run_console_snapshot( ) restored_console = b"" console = None + restore_command = snapshot_restore_command( + executable, backend, snapshot_path + ) with OpenvmmProcess( - snapshot_restore_command(executable, backend, snapshot_path), + restore_command, output_dir / f"console-snapshot-restore-{restore_index}-process.log", ) as process: try: - console = TcpConsole.connect(address, timeout) + console = TcpConsole.connect( + address, timeout, monitor=TimeAbiMonitor(restore_command) + ) if backend != "mshv": console.wait_for_line(CONSOLE_RX_RESTORED_MARKER, timeout) console.wait_for_line(CONSOLE_TX_DONE_MARKER, timeout) @@ -4386,7 +4411,14 @@ def _run_snapshot_tier( output_dir / f"snapshot-tier-{tier}-capture-process.log", ) as process: try: - console = TcpConsole.connect(address, timeout) + # The guest's shell is on this virtio console, so it answers + # the cold boot's status query there. + console = TcpConsole.connect( + address, + timeout, + monitor=TimeAbiMonitor(capture_command), + time_abi_status=True, + ) console.wait_for(BOOT_MARKER, timeout) script = _snapshot_tier_script(tier) console.send_bytes( @@ -4466,7 +4498,9 @@ def _run_snapshot_tier( output_dir / f"snapshot-tier-{tier}-restore-process.log", ) as process: try: - console = TcpConsole.connect(address, timeout) + console = TcpConsole.connect( + address, timeout, monitor=TimeAbiMonitor(restore_command) + ) console.send_bytes(b"Z") console.wait_for(repair_marker, timeout) if workload_start: @@ -4508,7 +4542,9 @@ def _run_snapshot_tier( output_dir / f"snapshot-tier-{tier}-timeout-process.log", ) as process: try: - timeout_console = TcpConsole.connect(address, timeout) + timeout_console = TcpConsole.connect( + address, timeout, monitor=TimeAbiMonitor(timeout_command) + ) timeout_console.send_bytes(b"Z") timed_out = process.wait(timeout) timeout_output = timeout_console.finish() diff --git a/scripts/nvx_tools/openvmm_process.py b/scripts/nvx_tools/openvmm_process.py index 0502e2a0e..79740827e 100644 --- a/scripts/nvx_tools/openvmm_process.py +++ b/scripts/nvx_tools/openvmm_process.py @@ -270,6 +270,13 @@ def wait(self, timeout: float) -> OpenvmmProcessResult: return OpenvmmProcessResult(returncode, bytes(self._output)) def close(self) -> None: + """Stop OpenVMM if it still runs, and keep its output in the log. + + It never raises a time ABI failure, so the error that led here on a + failure path surfaces unmasked. Every scenario's success path ends in + ``wait``, which scans the output to EOF and checks the exit status + (``test_every_openvmm_process_success_path_waits``). + """ if not self._finished and self.process.poll() is None: terminate(self.process) self._drain_available() @@ -278,6 +285,7 @@ def close(self) -> None: self._finished = True def _drain_available(self) -> None: + """Keep the output still queued; failure paths only, so no scan.""" while True: try: chunk = self._chunks.get_nowait() @@ -314,11 +322,19 @@ def __init__( self, connection: socket.socket, monitor: TimeAbiMonitor | None = None, + *, + time_abi_status: bool = False, ) -> None: self._connection = connection self._output = bytearray() self._search_offset = 0 self._monitor = monitor + self._query_status = ( + time_abi_status and monitor is not None and monitor.cold_boot + ) + self._status_sent = False + self._closed = False + self._failure: TimeAbiFailure | None = None @classmethod def connect( @@ -327,8 +343,16 @@ def connect( timeout: float, *, monitor: TimeAbiMonitor | None = None, + time_abi_status: bool = False, ) -> TcpConsole: - """Connect to a virtio console; ``monitor`` checks its time ABI lines.""" + """Connect to a virtio console. + + ``monitor`` scans the console's output for the time ABI, as + OpenvmmProcess scans OpenVMM's. With ``time_abi_status``, a + cold-booted guest whose shell is on this console answers a time ABI + status query before the scenario's first input, the same contract as + OpenvmmProcess's ``time_abi_status``. + """ deadline = time.monotonic() + timeout while True: try: @@ -338,7 +362,7 @@ def connect( except BaseException: connection.close() raise - return cls(connection, monitor) + return cls(connection, monitor, time_abi_status=time_abi_status) except OSError as error: if time.monotonic() >= deadline: raise TimeoutError( @@ -351,8 +375,37 @@ def _consume(self, chunk: bytes) -> None: self._monitor.feed(chunk) def send_bytes(self, data: bytes) -> None: + self._query_status_first() self._connection.sendall(data) + def _query_status_first(self) -> None: + if ( + not self._query_status + or self._status_sent + # The guest's shell is not on this console, or has not booted. + or not any(marker in self._output for marker in _GUEST_BOOT_MARKERS) + ): + return + assert self._monitor is not None + self._status_sent = True + self._connection.sendall(status_script().encode()) + deadline = time.monotonic() + STATUS_TIMEOUT_SECONDS + while self._monitor.status_queries == 0: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError( + "nvx-time status did not exit within " + f"{STATUS_TIMEOUT_SECONDS:g}s on the TCP console" + ) + self._connection.settimeout(min(remaining, 0.25)) + try: + chunk = self._connection.recv(4096) + except TimeoutError: + continue + if not chunk: + raise RuntimeError("TCP console closed before nvx-time status exited") + self._consume(chunk) + @property def output(self) -> bytes: return bytes(self._output) @@ -404,16 +457,37 @@ def wait_for_line(self, marker: bytes, timeout: float) -> None: raise RuntimeError(f"TCP console closed before line marker {marker!r}") self._consume(chunk) - def finish(self) -> bytes: - self._connection.settimeout(0.1) - try: - while chunk := self._connection.recv(4096): - self._output.extend(chunk) - except (TimeoutError, ConnectionError, OSError): - pass - finally: - self._connection.close() + def finish(self, *, check: bool = True) -> bytes: + """Drain the console until it goes quiet, close it, and return its output. + + The drained bytes reach the monitor too, so a violation or failed check + that arrived after the last awaited marker still fails the scenario. + Error paths pass ``check=False``: the monitor still scans the bytes for + the log, but nothing is raised over the error being handled. + """ + if not self._closed: + self._closed = True + self._connection.settimeout(0.1) + try: + while chunk := self._connection.recv(4096): + try: + self._consume(chunk) + except TimeAbiFailure as error: + # Keep draining, so the log holds everything. + self._failure = self._failure or error + except (TimeoutError, ConnectionError, OSError): + pass + finally: + self._connection.close() + if self._monitor is not None: + try: + self._monitor.finish() + except TimeAbiFailure as error: + self._failure = self._failure or error + if check and self._failure is not None: + raise self._failure return bytes(self._output) def close(self) -> None: + self._closed = True self._connection.close() diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 58c7b9841..cf490c8a3 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2,6 +2,7 @@ # pyright: reportPrivateUsage=false import argparse +import ast import io import json import os @@ -1938,6 +1939,75 @@ def test_tcp_console_connect_closes_failed_connection(self): console.close() connected.close.assert_called_once_with() + def test_every_openvmm_process_success_path_waits(self): + # close() never raises, so a late violation or a 193-195 power-off is + # only caught by wait(), which scans the output to EOF and checks the + # exit status. Every scenario's OpenVMM context must therefore reach + # wait() on its success path: unconditionally, with no early exit. + source = Path(microvm_tests.__file__).read_text(encoding="utf-8") + blocks = 0 + for node in ast.walk(ast.parse(source)): + if not isinstance(node, ast.With): + continue + for item in node.items: + call = item.context_expr + if not ( + isinstance(call, ast.Call) + and ast.unparse(call.func) == "OpenvmmProcess" + ): + continue + blocks += 1 + assert isinstance(item.optional_vars, ast.Name) + name = item.optional_vars.id + statements = list(node.body) + # A try body whose only handler is finally is unconditional. + while ( + len(statements) == 1 + and isinstance(statements[0], ast.Try) + and not statements[0].handlers + ): + statements = list(statements[0].body) + with self.subTest(line=node.lineno): + self.assertTrue( + any( + f"{name}.wait(" in ast.unparse(statement) + and not isinstance( + statement, (ast.If, ast.For, ast.While, ast.Try) + ) + for statement in statements + ) + ) + self.assertFalse( + any( + isinstance(sub, (ast.Return, ast.Break, ast.Continue)) + for sub in ast.walk(node) + ) + ) + self.assertGreater(blocks, 30) + + def test_every_tcp_console_is_monitored_and_cold_boot_shells_are_queried(self): + tree = ast.parse(Path(microvm_tests.__file__).read_text(encoding="utf-8")) + connects = [ + {keyword.arg: ast.unparse(keyword.value) for keyword in node.keywords} + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and ast.unparse(node.func) == "TcpConsole.connect" + ] + # The managed lifecycle, the console-snapshot capture and restore, and + # the snapshot-tier capture, restore, and gate-timeout restore. + self.assertEqual(len(connects), 6) + for keywords in connects: + self.assertRegex( + keywords.get("monitor", ""), r"^TimeAbiMonitor\(\w*command\)$" + ) + # Only the cold boots whose shell is on the virtio console ask for + # nvx-time status. Restores never ask, and in the managed lifecycle + # init starts the managed agent instead of a shell. + self.assertEqual( + [k["monitor"] for k in connects if k.get("time_abi_status") == "True"], + ["TimeAbiMonitor(capture_command)"] * 2, + ) + def test_snapshot_core_script_handles_no_clocksource(self): # The time ABI fixes the clocksource on every backend, so snapshot-core # neither selects nor waits for one. diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index f08c994a6..cef581292 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -1021,6 +1021,81 @@ def test_tcp_console_monitor_reports_violations(self): console.close() peer.close() + def test_tcp_console_finish_scans_the_tail(self): + # A violation after the last awaited marker still fails the scenario, + # including an unterminated final line. + connection, peer = socket.socketpair() + console = openvmm_process.TcpConsole(connection, TimeAbiMonitor(KVM_BOOT)) + peer.sendall(b"READY\n") + console.wait_for(b"READY", 1.0) + peer.sendall(VIOLATION.rstrip(b"\n")) + peer.close() + with self.assertRaisesRegex(TimeAbiFailure, "G_RCU_STALL"): + console.finish() + self.assertIn(b"G_RCU_STALL", console.finish(check=False)) + with self.assertRaisesRegex(TimeAbiFailure, "G_RCU_STALL"): + console.finish() + + # Error paths keep the tail in the log without raising over the error. + connection, peer = socket.socketpair() + console = openvmm_process.TcpConsole(connection, TimeAbiMonitor(KVM_BOOT)) + peer.sendall(VIOLATION) + peer.close() + self.assertEqual(console.finish(check=False), VIOLATION) + + def test_tcp_console_queries_a_cold_boot_before_the_first_input(self): + connection, peer = socket.socketpair() + peer.settimeout(1.0) + monitor = TimeAbiMonitor(KVM_BOOT) + console = openvmm_process.TcpConsole(connection, monitor, time_abi_status=True) + peer.sendall(b"NVX-GUEST-BOOT-OK: alpine\n") + console.wait_for(b"NVX-GUEST-BOOT-OK", 1.0) + peer.sendall((BOOT_LINE + RUNTIME_LINE + STATUS_OK).encode()) + console.send_bytes(b"INPUT\n") + console.send_bytes(b"MORE\n") + expected = time_abi.status_script().encode() + b"INPUT\nMORE\n" + sent = b"" + while len(sent) < len(expected): + sent += peer.recv(4096) + self.assertEqual(sent, expected) + self.assertEqual(monitor.status_queries, 1) + self.assertIsNotNone(monitor.boot) + console.close() + peer.close() + + # A query without a passing boot line fails before the input is sent. + connection, peer = socket.socketpair() + console = openvmm_process.TcpConsole( + connection, TimeAbiMonitor(KVM_BOOT), time_abi_status=True + ) + peer.sendall(b"NVX-GUEST-BOOT-OK: alpine\n") + console.wait_for(b"NVX-GUEST-BOOT-OK", 1.0) + peer.sendall(STATUS_OK.encode()) + with self.assertRaisesRegex(TimeAbiFailure, "boot marker"): + console.send_bytes(b"INPUT\n") + console.close() + peer.close() + + def test_tcp_console_queries_only_cold_boots_that_ask(self): + for command, enabled in ( + (MSHV_RESTORE, True), + (KVM_BOOT, False), + ): + with self.subTest(command=command, enabled=enabled): + connection, peer = socket.socketpair() + peer.settimeout(1.0) + monitor = TimeAbiMonitor(command) + console = openvmm_process.TcpConsole( + connection, monitor, time_abi_status=enabled + ) + peer.sendall(b"NVX-GUEST-BOOT-OK: alpine\n") + console.wait_for(b"NVX-GUEST-BOOT-OK", 1.0) + console.send_bytes(b"Z") + self.assertEqual(peer.recv(4096), b"Z") + self.assertEqual(monitor.status_queries, 0) + console.close() + peer.close() + def doctor_context(root: Path, backend: str = "kvm") -> doctor.DoctorContext: return doctor.DoctorContext( From 9ed2352c89f4f75598af15d4231f5f87182a3600 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:43:17 -0700 Subject: [PATCH 173/188] ci: keep the nonstop_tsc runner gate in front of host qualification The time ABI branch had turned validate-runner's Linux nonstop_tsc check, and setup-linux-runner.sh's require_invariant_tsc, into evidence only, on the grounds that the measured checks replace them. But only the microVM jobs run the guest warp probe (doctor H6). The OpenVMM vmm-tests and the platform benchmarks run guests after H1, H2 and H4 alone, which the #265 host (an Ice Lake 8370C VM whose host OS has no invariant TSC) can pass, and a VM scale set can land a runner on such hardware later. Restore dev's fail-closed gate, verbatim: validate-runner's "Validate host TSC" step in every Linux job, before the doctor, and the setup script's require_invariant_tsc at provisioning, with dev's test. The doctor's checks stay measured-only and now add to the gate rather than replace it; the action's comment, the qualification test and doc/ci.md say so. The gate can go once every job that runs guests also runs H6. Every live labeled Linux runner prints "TSC flag nonstop_tsc: present" in the latest dev CI run, and no job here can land on the runner without a backend label. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 42 +++++-- doc/ci.md | 63 ++++++----- scripts/setup/setup-linux-runner.sh | 16 +-- scripts/test_nvx_tools.py | 122 +++++++++++++-------- 4 files changed, 152 insertions(+), 91 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index 210de1674..6c0df078f 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -124,13 +124,41 @@ runs: throw "Hyper-V SVGA firmware was not found: $svgaFirmware" } - # Time ABI host qualification (doc/design/time-abi.md, "Host - # qualification"): the backend, the CPU fingerprint and generation, and - # the TSC rate stability on its short CI schedule, and, in jobs that - # downloaded OpenVMM, its CPU profile check (H2) and preflight (H3). Other - # jobs qualify without OpenVMM. It replaces the nonstop_tsc check, which - # is now recorded as evidence only. The guest warp probe runs in the - # microVM scenarios, which boot the time ABI. + # Host time. On Linux, the nonstop_tsc gate stays fail-closed in front of + # the time ABI host qualification (doc/design/time-abi.md, "Host + # qualification"), which then checks the backend, the CPU fingerprint and + # generation, and the TSC rate stability on its short CI schedule, and, + # in jobs that downloaded OpenVMM, its CPU profile check (H2) and + # preflight (H3); other jobs qualify without OpenVMM. These checks are + # host-level, and the guest warp probe (H6) runs only in the microVM + # scenarios, so the gate keeps a host whose guests warp (#265) out of + # every other job. + - name: Validate host TSC + if: runner.os != 'Windows' + shell: bash + run: | + set -euo pipefail + # Guests on a host without an invariant TSC intermittently see + # cross-vCPU TSC warps, which make Linux mark the guest TSC unstable + # (#211). + echo "Kernel: $(uname -r)" + echo "CPU: $(awk -F': ' '/^model name/ { print $2; exit }' /proc/cpuinfo)" + echo "Clocksource: $(cat /sys/devices/system/clocksource/clocksource0/current_clocksource 2>/dev/null || echo unknown)" + flags=" $(awk -F': ' '/^flags/ { print $2; exit }' /proc/cpuinfo) " + for flag in constant_tsc nonstop_tsc tsc_known_freq tsc_reliable; do + case "${flags}" in + *" ${flag} "*) echo "TSC flag ${flag}: present" ;; + *) echo "TSC flag ${flag}: absent" ;; + esac + done + case "${flags}" in + *" nonstop_tsc "*) ;; + *) + echo "::error::Runner host does not expose an invariant TSC (nonstop_tsc); redeploy the VM on a host that does" >&2 + exit 1 + ;; + esac + - name: Qualify host time on Linux if: runner.os != 'Windows' shell: bash diff --git a/doc/ci.md b/doc/ci.md index 888419aa9..370ea6351 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -157,26 +157,29 @@ development release, and performance persistence. The GitHub-hosted own key, so a kernel rebuild delays only the debug jobs, and a failed debug kernel build fails the required status check and blocks the release. -Every job that uses the `validate-runner` action qualifies its runner for the -time ABI with `nvx.py doctor --checks H1 H2 H4 --ci-schedule` (see -[Host qualification](#host-qualification)): the backend, the CPU fingerprint -and generation, and the TSC rate stability on its short schedule. The microVM -and platform jobs run it after downloading OpenVMM and add OpenVMM's CPU -profile check (H2) and preflight (H3), and keep the fingerprint as an artifact -when the profile check fails; the other jobs pass `--no-openvmm`. This replaces -the earlier `nonstop_tsc` check and takes a few seconds. It reports the CPU -generation, the CPU profile that `auto` selects, and the measured TSC rate in -the log and the job summary, and fails the job with a stable code when the -runner is not qualified, for example `E_PROFILE_HOST_UNKNOWN` on an unknown CPU -generation. -Qualification gates only on measured properties, alike on every backend: the -host OS's invariant-TSC flags and clocksource are recorded as evidence, and -the guest warp probe in the microVM scenarios measures the skew that a host -without an invariant TSC causes. On such an MSHV runner VM, never-restored -guests hit cross-vCPU TSC warps when an idle host CPU woke (#211, #265), which -is why the probe schedule includes idle gaps. Runner labels do not encode the -generation; per-PR CI captures and restores on one runner, so generations -never mix. +Every job that uses the `validate-runner` action first requires an invariant +TSC on a Linux runner (`nonstop_tsc` in `/proc/cpuinfo`) and fails without +one, as before the time ABI. It then qualifies the runner for the time ABI +with `nvx.py doctor --checks H1 H2 H4 --ci-schedule` (see [Host +qualification](#host-qualification)): the backend, the CPU fingerprint and +generation, and the TSC rate stability on its short schedule. The microVM and +platform jobs run it after downloading OpenVMM and add OpenVMM's CPU profile +check (H2) and preflight (H3), and keep the fingerprint as an artifact when +the profile check fails; the other jobs pass `--no-openvmm`. This takes a few +seconds. It reports the CPU generation, the CPU profile that `auto` selects, +and the measured TSC rate in the log and the job summary, and fails the job +with a stable code when the runner is not qualified, for example +`E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation. The doctor gates only +on measured properties, alike on every backend: it records the host OS's +invariant-TSC flags and clocksource as evidence, and the guest warp probe in +the microVM scenarios measures the skew that a host without an invariant TSC +causes. On such an MSHV runner VM, never-restored guests hit cross-vCPU TSC +warps when an idle host CPU woke (#211, #265), which is why the probe schedule +includes idle gaps. The `nonstop_tsc` gate stays in front of it until every +job that runs guests also runs the warp probe: only the microVM jobs do, while +the OpenVMM vmm-tests and the platform benchmarks run guests after host-level +checks that such a host can pass. Runner labels do not encode the generation; +per-PR CI captures and restores on one runner, so generations never mix. The warp probe replaced the `restore-tsc-sync` scenario, its test-only `clearcpuid=tsc_adjust` kernel option, the guest's scan of the kernel log for @@ -304,17 +307,19 @@ override their default build paths. `--no-openvmm` qualifies a host without OpenVMM: H2 then checks the CPU identity and generation only, and H3 and H6 cannot run. -Qualification gates on measured properties, alike on every backend: the guest +The doctor gates on measured properties, alike on every backend: the guest warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource -are evidence only. In CI, `validate-runner` runs the cheap host-level checks -H1, H2, and H4 in every job that uses it, on the short `--ci-schedule`. The -microVM and platform jobs run it after downloading OpenVMM and the guest -artifacts and add H2's CPU profile check and H3, so that H4 also compares the -measured rate with the backend's native rate; the other jobs pass -`--no-openvmm`. The guest warp probe needs a time ABI boot, so the microVM boot -and restore scenarios run CI's warp schedule after every boot and restore and -assert its verdict. H5 and H7 remain available for interactive qualification. +are evidence only in the doctor. In CI, `validate-runner` keeps the +`nonstop_tsc` gate on Linux runners in front of it, because only the microVM +jobs run H6, and runs the cheap host-level checks H1, H2, and H4 in every job +that uses it, on the short `--ci-schedule`. The microVM and platform jobs run +it after downloading OpenVMM and the guest artifacts and add H2's CPU profile +check and H3, so that H4 also compares the measured rate with the backend's +native rate; the other jobs pass `--no-openvmm`. The guest warp probe needs a +time ABI boot, so the microVM boot and restore scenarios run CI's warp +schedule after every boot and restore and assert its verdict. H5 and H7 remain +available for interactive qualification. ## Rust crate diff --git a/scripts/setup/setup-linux-runner.sh b/scripts/setup/setup-linux-runner.sh index d195318d4..6b36c51bc 100644 --- a/scripts/setup/setup-linux-runner.sh +++ b/scripts/setup/setup-linux-runner.sh @@ -60,15 +60,11 @@ version_at_least() { [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -n 1)" = "$2" ] } -report_invariant_tsc() { - # The host OS's invariant-TSC flag is evidence only. CI qualifies a runner - # on measured properties instead: the guest warp probe and the TSC rate - # stability (doc/ci.md, "Host qualification"; #211, #265). - if grep -Eq '^flags[[:space:]]*:.*[[:space:]]nonstop_tsc([[:space:]]|$)' "$1"; then - echo "host exposes an invariant TSC (nonstop_tsc)" - else - echo "warning: host does not expose an invariant TSC (nonstop_tsc); CI qualifies it on the measured warp and rate stability" >&2 - fi +require_invariant_tsc() { + # Guests on a host without an invariant TSC intermittently see cross-vCPU + # TSC warps, which make Linux mark the guest TSC unstable (#211). + grep -Eq '^flags[[:space:]]*:.*[[:space:]]nonstop_tsc([[:space:]]|$)' "$1" || + die "host does not expose an invariant TSC (nonstop_tsc); redeploy the VM on a host that does" } validate_runner_name() { @@ -822,7 +818,7 @@ case "$backend" in kvm | mshv) ;; *) die "--backend must be kvm or mshv" ;; esac -report_invariant_tsc /proc/cpuinfo +require_invariant_tsc /proc/cpuinfo [ "$(id -u)" -ne 0 ] || die "run this script as the SSH administrator, not root" require_command sudo sudo -n true || die "passwordless sudo is required" diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index b08f6ffbf..41dc5bcec 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -3308,7 +3308,7 @@ def test_windows_runner_requires_inbox_pcat_firmware(self): ): self.assertIn(firmware, configuration) - def test_runners_qualify_their_host_time_before_every_job(self): + def test_linux_runners_require_an_invariant_tsc(self): validate_runner = ( BuildConstants.REPO_ROOT / ".github" @@ -3320,10 +3320,82 @@ def test_runners_qualify_their_host_time_before_every_job(self): BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" ).read_text(encoding="utf-8") - # The doctor replaces the nonstop_tsc check: the backend, the CPU - # fingerprint and generation, and the TSC rate stability. Jobs have no - # OpenVMM binary yet, so H2 skips OpenVMM's CPU profile check. - self.assertNotIn("Validate host TSC", validate_runner) + step = validate_runner.split(" - name: Validate host TSC\n", 1)[1] + step = step.split("\n\n - name: ", 1)[0] + self.assertIn(" if: runner.os != 'Windows'\n", step) + check = "\n".join( + line.removeprefix(" ") + for line in step.split(" run: |\n", 1)[1].splitlines() + ) + function = linux_setup.split("require_invariant_tsc() {\n", 1)[1] + function = "require_invariant_tsc() {\n" + function.split("\n}\n", 1)[0] + function += "\n}\n" + self.assertLess( + linux_setup.index("require_invariant_tsc /proc/cpuinfo\n"), + linux_setup.index('sudo -n true || die "passwordless sudo is required"'), + ) + if os.name != "posix": + return + + with tempfile.TemporaryDirectory() as temporary: + cpuinfo = Path(temporary) / "cpuinfo" + for flags, invariant in ( + ("fpu tsc constant_tsc nonstop_tsc tsc_known_freq", True), + ("fpu tsc constant_tsc tsc_known_freq", False), + ("fpu tsc constant_tsc nonstop_tsc_x", False), + ): + cpuinfo.write_text( + f"model name\t: Test CPU\nflags\t\t: {flags}\n", + encoding="utf-8", + ) + with self.subTest(flags=flags, check="validate-runner"): + result = subprocess.run( + ["bash", "-c", check.replace("/proc/cpuinfo", str(cpuinfo))], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode == 0, invariant, result.stderr) + self.assertIn("CPU: Test CPU", result.stdout) + self.assertEqual( + "::error::Runner host does not expose an invariant TSC" + in result.stderr, + not invariant, + ) + with self.subTest(flags=flags, check="setup-linux-runner"): + result = subprocess.run( + [ + "sh", + "-c", + 'die() { echo "$*" >&2; exit 1; }\n' + f"{function}" + f"require_invariant_tsc '{cpuinfo}'\n", + ], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + self.assertEqual(result.returncode == 0, invariant, result.stderr) + + def test_runners_qualify_their_host_time_before_every_job(self): + validate_runner = ( + BuildConstants.REPO_ROOT + / ".github" + / "actions" + / "validate-runner" + / "action.yml" + ).read_text(encoding="utf-8") + + # The doctor adds to the nonstop_tsc gate, which stays first and + # fail-closed on Linux: the backend, the CPU fingerprint and + # generation, and the TSC rate stability. Jobs that haven't downloaded + # OpenVMM skip its CPU profile check (H2) and preflight (H3). + self.assertLess( + validate_runner.index(" - name: Validate host TSC\n"), + validate_runner.index(" - name: Qualify host time on Linux\n"), + ) for name, shell, command, summary in ( ( "Qualify host time on Linux", @@ -3407,46 +3479,6 @@ def test_runners_qualify_their_host_time_before_every_job(self): validate_runner.index("Qualify host time on Linux"), ) - # The runner setup records the host's invariant-TSC flag as evidence. - self.assertNotIn("require_invariant_tsc", linux_setup) - function = linux_setup.split("report_invariant_tsc() {\n", 1)[1] - function = "report_invariant_tsc() {\n" + function.split("\n}\n", 1)[0] - function += "\n}\n" - self.assertLess( - linux_setup.index("report_invariant_tsc /proc/cpuinfo\n"), - linux_setup.index('sudo -n true || die "passwordless sudo is required"'), - ) - if os.name != "posix": - return - - with tempfile.TemporaryDirectory() as temporary: - cpuinfo = Path(temporary) / "cpuinfo" - for flags, invariant in ( - ("fpu tsc constant_tsc nonstop_tsc tsc_known_freq", True), - ("fpu tsc constant_tsc tsc_known_freq", False), - ("fpu tsc constant_tsc nonstop_tsc_x", False), - ): - cpuinfo.write_text( - f"model name\t: Test CPU\nflags\t\t: {flags}\n", - encoding="utf-8", - ) - with self.subTest(flags=flags, check="setup-linux-runner"): - result = subprocess.run( - [ - "sh", - "-c", - 'die() { echo "$*" >&2; exit 1; }\n' - f"{function}" - f"report_invariant_tsc '{cpuinfo}'\n", - ], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual("warning:" in result.stderr, not invariant) - def test_runner_setups_install_backend_native_openvmm_targets(self): linux_setup = ( BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" From ed83d4ff8c663ce897d4c07f1d581a95799acfb8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:43:17 -0700 Subject: [PATCH 174/188] notices: nvx-exit links musl statically too MUSL_STATIC_HELPERS builds /sbin/nvx-exit with musl-gcc -static, like nvx-time, and the Azure Linux image stage does the same, so the musl notice names it with nvx-time and nvx-time-probe. The built binary is a static ELF with no interpreter and no glibc symbols. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- THIRD_PARTY_NOTICES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index bc9ab58f9..2990dad39 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -9,7 +9,7 @@ the distribution tooling. | Linux 6.18.38 | Generated under `.cache/linux` | GPL-2.0-only; the upstream archive and patch series are pinned in `SOURCE-MANIFEST.json` and `kernel/patches`. Each build exports its exact config, and `collect-sources` emits the complete patched tree. | | OpenVMM | `openvmm` | MIT; see `openvmm/LICENSE`. Source is a private Git submodule and is available only to authorized users. | | NVX guest/build sources | `guest`, `ubuntu`, `kernel`, `scripts/nvx_tools` | MIT; see `LICENSE`. | -| musl libc (Debian 12 `musl`) | statically linked into `/sbin/nvx-time` and `/sbin/nvx-time-probe` in generated initramfs images | MIT; Copyright © 2005-2020 Rich Felker, et al. The build image's `/usr/share/doc/musl/copyright` carries the full notice, and the source is Debian's `musl` source package. | +| musl libc (Debian 12 `musl`) | statically linked into `/sbin/nvx-time`, `/sbin/nvx-time-probe`, and `/sbin/nvx-exit` in generated initramfs images | MIT; Copyright © 2005-2020 Rich Felker, et al. The build image's `/usr/share/doc/musl/copyright` carries the full notice, and the source is Debian's `musl` source package. | | Alpine Linux packages | generated initramfs images | Per-package licenses. Every build emits `*.packages.json`; retain that manifest and collect the corresponding aports recipes and upstream sources before distributing an image. | | Azure Linux 3.0 packages | generated Azure Linux initramfs | Per-package licenses, including GPLv2 BusyBox. The base image is digest-pinned in `SOURCE-MANIFEST.json`, every supplemental RPM is SHA-256 pinned in `azurelinux/packages.lock.json`, and every build emits `initramfs-azurelinux.cpio.gz.packages.json`. `collect-sources` does not collect this corresponding source yet, so `--include-source` packages omit the Azure Linux initramfs; publish the matching source before distributing a binary-only package that contains it. | | Ubuntu Base 26.04.1 LTS and supplemental packages | generated Ubuntu initramfs and EROFS images | Per-package licenses. The base archive and every supplemental `.deb` are SHA-256 pinned. Retain the generated Ubuntu manifests and publish the matching `.dsc` plus all referenced source members before distributing an image. Ubuntu artifacts contain Ubuntu userland with the NVX kernel, not the Ubuntu kernel. | From 9b0f5143893aa10565607d9c1d7b28a5b85889f4 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:43:18 -0700 Subject: [PATCH 175/188] doc: describe nvx-port-io as it is now nvx-port-io no longer reads restore packets: the time ABI moved packet parsing to nvx-time and removed the read-restore-packet subcommand. What remains is single-byte port reads and writes and the VM generation ID read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/build.md b/doc/build.md index 8edb168d1..bf328c470 100644 --- a/doc/build.md +++ b/doc/build.md @@ -140,7 +140,8 @@ complete per-scenario logs under The Alpine initramfs includes the sandbox PID-1 bootstrap, its container namespace helpers, the static `nvx-device-io` benchmark helper, the static `nvx-port-io` -restore packet helper, the static `nvx-time-probe` guest time probe, and the +port helper (single-byte port reads and writes, and the VM generation ID read), +the static `nvx-time-probe` guest time probe, and the static `nvx-time` guest time component under `/sbin`. The probe measures TSC and clock read latency, reports the guest's clock, clocksource, TSC flags, and hypervisor identity, and runs the cross-vCPU warp test that the From 59b7c448fb17265e5dc937bb22d00beefe8219c7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:43:18 -0700 Subject: [PATCH 176/188] test: pin nvx-exit codes that would wrap a 32-bit value nvx-exit's parser rejects the value as soon as it passes 255, after each digit, so it never exceeds 2,559 and cannot wrap. Add the cases that would expose a wrap, 4294967296 (2^32) and 4294967551 (2^32 + 255), which must still become 1, and a long zero-padded 255. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/test_nvx_tools.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 41dc5bcec..7443d94b7 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -7394,6 +7394,10 @@ def test_writes_the_exit_code_as_one_byte_at_the_control_port(self): ((" 7",), 1), (("12a",), 1), (("99999999999999999999",), 1), + # Out of range after any digit, so the value never wraps. + (("4294967296",), 1), + (("4294967551",), 1), + (("00000000000000000255",), 255), # Decimal: the old script's printf read a leading zero as octal. (("010",), 10), (("7", "9"), 7), From 93a5b9305bb43ded316f29249dfc9501fe2b4245 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 01:56:29 -0700 Subject: [PATCH 177/188] doc: keep the nonstop_tsc runner gate in the time ABI's host qualification 4b5324c9 restored validate-runner's fail-closed nonstop_tsc check in every Linux job, in front of the doctor, and require_invariant_tsc at runner provisioning, until every job that runs guests also runs the warp probe (H6). The spec still said that the time checks replace it. Host qualification now says that CI keeps the gate in front of the doctor, and why. The qualification gates say that the doctor records the host OS's invariant-TSC bit as evidence only, and the removed list no longer lists the runner check. The test matrix now says exactly where CI runs nvx-time status: after every cold boot whose shell is on a console the harness reads, and after the restores of smp-snapshot, restore-processors and restore-downtime. CI's other restores and the guests without a shell are only scanned, and a failed check still powers them off with 193. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index 38b086446..aaf07da3c 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -1634,7 +1634,13 @@ with `H4` and `H6` on their long schedules. The `validate-runner` action runs and every CI microVM boot and restore scenario runs the CI warp schedule; `H5` and `H7` run in `doctor` only. Both tools print one `NVX-DOCTOR: check= status= detail=...` line per check and -fail if any check fails. The time checks replace the `nonstop_tsc` check. +fail if any check fails. In CI, `validate-runner` first requires +`nonstop_tsc` in `/proc/cpuinfo` on Linux runners and fails without it, as +before the time ABI, and the Linux runner setup script requires it at +provisioning. This gate stays until every job that runs guests also runs +`H6`: only the microVM jobs do, while the OpenVMM vmm-tests and the platform +benchmarks run guests after host-level checks, which a host without an +invariant TSC can pass. | ID | Check | | --- | --- | @@ -1739,11 +1745,11 @@ of the warps in #265: ### Qualification gates -Qualification gates alike on every backend, on measured properties only: the +The doctor qualifies alike on every backend, on measured properties only: the CPU profile (`H2`, `H3`), rate stability (`H4`), and the idle-scheduled warp -probe (`H6`, and the CI schedule in every microVM job). The host OS's -invariant-TSC bit and the host clocksource are recorded as evidence and never -gate, on any backend: +probe (`H6`, and the CI schedule in every microVM job). It records the host +OS's invariant-TSC bit and the host clocksource as evidence and never gates +on them, on any backend: - On Azure, WHP and nested MSHV cannot offer the invariant-TSC bit to their guests through their feature banks, although the host OS sees an invariant @@ -1760,8 +1766,9 @@ gate, on any backend: That 8370C MSHV runner, whose host OS lacks the bit and which showed the #265 warps, is out of rotation and unqualified because our account cannot run -guests there, not because of this rule; its host-level warp probe saw -backward steps of at most 2.1 ns. +guests there, not because of the doctor's rules; CI's `nonstop_tsc` gate +would also reject it. Its host-level warp probe saw backward steps of at +most 2.1 ns. ### Generations and runner placement @@ -2035,11 +2042,16 @@ parser; four-byte portb reads; and the generation counter. **Conformance.** The boot and restore checks pass at 1, 2, 4, and 8 vCPUs on all 18 registered hosts, plus the exhaustive CI check and the warp probe on -every backend. Production paths print no marker, so CI scenarios and the -matrix driver run `/sbin/nvx-time status` over the console after shell-ready -and after every restore. They require exit status 0 and, after a restore, a -`phase=restore` line with `status=ok`, the restored `generation`, and `cpus` -equal to the online CPUs; after a cold boot, a `phase=boot` line with +every backend. Production paths print no marker, so the matrix driver runs +`/sbin/nvx-time status` over the console after shell-ready and after every +restore. CI runs it after every cold boot whose shell is on a console the +harness reads, and after every restore in the `smp-snapshot`, +`restore-processors`, and `restore-downtime` scenarios. CI's other restores +and its guests without a shell, the managed lifecycle and one-shot workloads, +are only scanned for violation events; a failed check still powers the guest +off with status 193. Each query requires exit status 0 and, after a restore, +a `phase=restore` line with `status=ok`, the restored `generation`, and +`cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with `generation=0`. During fleet validation (`p6`), the matrix driver records each phase's `cpu_us`, and every recorded sample must be within the backend's budget for that phase in @@ -2163,7 +2175,6 @@ Removed from NVX: in `restore-processors.sh`, which the watcher replaces. The `cold_start_clocksource` benchmark scenario passes `clocksource=tsc` on every backend. -- CI: the `nonstop_tsc` runner check, replaced by host qualification. Migration impact: From 40481a2f46fab37f7e6f5cd50802924929f87867 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 02:19:58 -0700 Subject: [PATCH 178/188] harness: snapshot-tiers' restored guests report their deferred restore checks The debug-kernel lane gates on snapshot-tiers, but a restored tier guest exited right after its repair and layer assertions. The restore's deferred checks start 150 ms after the restore, so the run could pass with a check still pending, or with the watchdog and suppression check never reached; the monitor only catches a violation that is emitted. Every restored tier guest now runs nvx-time status after its tier assertions and before nvx-exit, as the other restore scenarios do through post_restore_checks(). The query waits for the deferred checks and prints the restore line, and the harness requires a passing one: the console's monitor checks the query when it exits (TcpConsole.wait_for_time_abi_status), and _check_restore_status then checks the restore phase, generation 1, and the online CPUs. The script turns set -e off first, so a failing status still reports its exit status. The guest then waits for one byte from the host before nvx-exit, and the harness sends it once the query's exit line has arrived. Without that handshake, WHP dropped the query's lines from the virtio console when nvx-exit stopped the VM right after them. snapshot-tiers measures nothing, so no timing moves; the gate-timeout check stops its guest at the restore gate on purpose and has no query. The warp probe isn't added: the tier guests have one vCPU, so there's no CPU pair to compare. doc/ci.md says what the debug-kernel lane proves for each restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 31 +++++++++++-------- .../microvm_test_scripts/snapshot-tier.sh.in | 3 ++ scripts/nvx_tools/microvm_tests.py | 20 ++++++++++++ scripts/nvx_tools/openvmm_process.py | 25 +++++++++++++++ scripts/test_microvm_tests.py | 28 +++++++++++++++++ scripts/test_time_abi.py | 23 ++++++++++++++ 6 files changed, 117 insertions(+), 13 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 370ea6351..07858a86e 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -139,20 +139,25 @@ per backend and phase. `elapsed_us` is wall time, including waits behind the workload, and has no budget. CI gates on neither: the performance gate is the A/B comparison outside CI. -The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run -`test-microvm --debug-kernel` on the CI debug kernel (`build/vmlinux-debug`, -built from `kernel/config-microvm-debug`), whose soft-lockup and hung-task -detectors production kernels leave out. It selects the same-host restore -scenarios `smp`, `smp-snapshot`, `restore-processors`, `restore-downtime`, and +The `nvx-microvm-debug-{kvm,mshv,whp}` jobs run `test-microvm --debug-kernel` +on the CI debug kernel (`build/vmlinux-debug`, built from +`kernel/config-microvm-debug`), whose soft-lockup and hung-task detectors +production kernels leave out. It selects the same-host restore scenarios +`smp`, `smp-snapshot`, `restore-processors`, `restore-downtime`, and `snapshot-tiers`. Any RCU stall, soft lockup, or hung task makes the guest's -time ABI watcher power off with status 194, which fails the run. The harness -refuses a kernel whose `vmlinux-debug.config` lacks the detectors, because the -guest's `C11` check passes vacuously without them. To bound the cost, pull -requests run the debug kernel on KVM only and `dev` pushes run it on every -backend; each job takes about five minutes on its own runner, in parallel with -the other microVM jobs. The jobs gate the required status check, the -development release, and performance persistence. The GitHub-hosted -`debug-kernel` job builds the debug kernel beside the shared `artifacts` job +time ABI watcher power off with status 194, which fails the run. Every guest +those scenarios restore asks `nvx-time status` before it exits +(`snapshot-tiers` after its tier assertions), which waits for the restore's +deferred checks and must report a passing restore line, so no restore leaves +its checks pending. The one exception is `snapshot-tiers`' gate-timeout check, +whose guest OpenVMM stops at the restore gate on purpose. The harness refuses +a kernel whose `vmlinux-debug.config` lacks the detectors, because the guest's +`C11` check passes vacuously without them. To bound the cost, pull requests +run the debug kernel on KVM only and `dev` pushes run it on every backend; +each job takes about five minutes on its own runner, in parallel with the +other microVM jobs. The jobs gate the required status check, the development +release, and performance persistence. The GitHub-hosted `debug-kernel` job +builds the debug kernel beside the shared `artifacts` job (`build-guest-artifacts` with `guest-images: "false"`) and caches it under its own key, so a kernel rebuild delays only the debug jobs, and a failed debug kernel build fails the required status check and blocks the release. diff --git a/scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in b/scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in index e7af3c5a5..3f9051ae0 100644 --- a/scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in +++ b/scripts/nvx_tools/microvm_test_scripts/snapshot-tier.sh.in @@ -28,4 +28,7 @@ echo @RELEASED_MARKER@ blockdev --flushbufs /dev/vda value="$(dd if=/dev/vda bs=1 count=1 2>/dev/null | od -An -tu1 | tr -d '[:space:]')" echo "@LAYER_MARKER@$value" +set +e +@STATUS_QUERY@ +dd if=/dev/hvc1 bs=1 count=1 >/dev/null 2>&1 nvx-exit 0 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index efb768c51..779216740 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -4338,6 +4338,13 @@ def _snapshot_tier_script(tier: str) -> str: REPAIR_ACTION=repair_action, RELEASED_MARKER=f"{prefix}-RELEASED", LAYER_MARKER=f"{prefix}-LAYER-", + # After the tier's assertions, the restored guest asks nvx-time status, + # which waits for the restore's deferred checks, the debug kernel's + # watchdogs among them, and prints the restore line; the script turns + # off set -e first, so a failing status still reports its exit status. + # It then waits for one byte from the host before nvx-exit, so the + # query's lines reach the virtio console before the VM stops. + STATUS_QUERY=status_script().rstrip("\n"), ) @@ -4507,6 +4514,11 @@ def _run_snapshot_tier( console.wait_for(workload_marker, timeout) console.wait_for(released_marker, timeout) console.wait_for(expected_layer, timeout) + # The guest's status query waits for the restore's deferred + # checks. It then waits for one byte from here before nvx-exit, + # so the query's lines reach the console before the VM stops. + console.wait_for_time_abi_status(STATUS_TIMEOUT_SECONDS) + console.send_bytes(b"Z") restored = process.wait(timeout) restore_console = console.finish() console = None @@ -4523,6 +4535,14 @@ def _run_snapshot_tier( raise RuntimeError(f"{tier} input crossed the restore gate") if restore_lines.count(expected_layer) != 1: raise RuntimeError(f"{tier} restore observed the wrong layer binding") + # The restored guest's status query, after the tier's assertions, must + # report a passing restore line, so the deferred restore checks ran. + _check_restore_status( + restore_console, + restore_command, + processors=int(restore_command[restore_command.index("--processors") + 1]), + context=f"{tier} restore", + ) if _snapshot_fingerprint(snapshot) != fingerprint: raise RuntimeError(f"{tier} restore modified snapshot payloads") diff --git a/scripts/nvx_tools/openvmm_process.py b/scripts/nvx_tools/openvmm_process.py index 79740827e..6787ea117 100644 --- a/scripts/nvx_tools/openvmm_process.py +++ b/scripts/nvx_tools/openvmm_process.py @@ -457,6 +457,31 @@ def wait_for_line(self, marker: bytes, timeout: float) -> None: raise RuntimeError(f"TCP console closed before line marker {marker!r}") self._consume(chunk) + def wait_for_time_abi_status(self, timeout: float) -> None: + """Wait until the guest's next ``nvx-time status`` query exits. + + The monitor validates what the query printed when its exit line + arrives, so a failed or pending check raises here. + """ + if self._monitor is None: + raise ValueError("TCP console has no time ABI monitor") + before = self._monitor.status_queries + deadline = time.monotonic() + timeout + while self._monitor.status_queries == before: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError( + f"nvx-time status did not exit within {timeout:g}s on the TCP console" + ) + self._connection.settimeout(min(remaining, 0.25)) + try: + chunk = self._connection.recv(4096) + except TimeoutError: + continue + if not chunk: + raise RuntimeError("TCP console closed before nvx-time status exited") + self._consume(chunk) + def finish(self, *, check: bool = True) -> bytes: """Drain the console until it goes quiet, close it, and return its output. diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index cf490c8a3..49dd4f03b 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -3,6 +3,7 @@ import argparse import ast +import inspect import io import json import os @@ -2397,6 +2398,33 @@ def test_snapshot_tier_scripts_preserve_tier_specific_policy(self): self.assertIn("captured-workload-id", checkpoint) self.assertNotIn("@CAPTURE_ACTION@", checkpoint) + def test_restored_tier_guests_report_their_restore_checks_before_exiting(self): + # The debug-kernel lane gates on snapshot-tiers, so every restored tier + # guest waits for its deferred restore checks before it exits. + status = microvm_tests.status_script().rstrip("\n") + for tier in ("platform", "workload-start", "instance-checkpoint"): + with self.subTest(tier=tier): + script = microvm_tests._snapshot_tier_script(tier) + self.assertNotIn("@STATUS_QUERY@", script) + self.assertLess( + script.index(f"NVX-TIER-{tier.upper()}-LAYER-"), + script.index(f"set +e\n{status}\n"), + ) + # The guest waits for the host's byte, so the query's lines + # reach the console before nvx-exit stops the VM. + self.assertTrue( + script.endswith( + f"{status}\ndd if=/dev/hvc1 bs=1 count=1 >/dev/null 2>&1\n" + "nvx-exit 0\n" + ) + ) + runner = inspect.getsource(microvm_tests._run_snapshot_tier) + self.assertIn("_check_restore_status(\n restore_console,", runner) + self.assertLess( + runner.index("console.wait_for_time_abi_status(STATUS_TIMEOUT_SECONDS)"), + runner.index('console.send_bytes(b"Z")\n restored = '), + ) + def test_snapshot_tier_entry_points_reject_unsupported_tiers(self): with self.assertRaisesRegex(ValueError, "unsupported snapshot tier 'invalid'"): microvm_tests._snapshot_tier_script("invalid") diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index cef581292..628b5e7ef 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -1076,6 +1076,29 @@ def test_tcp_console_queries_a_cold_boot_before_the_first_input(self): console.close() peer.close() + def test_tcp_console_waits_for_a_status_query_to_exit(self): + connection, peer = socket.socketpair() + monitor = TimeAbiMonitor(MSHV_RESTORE) + console = openvmm_process.TcpConsole(connection, monitor) + restore = RESTORE_LINE.replace("lapic_hz=1000000000", "lapic_hz=200000000") + peer.sendall((restore + RUNTIME_LINE + STATUS_OK).encode()) + console.wait_for_time_abi_status(1.0) + self.assertEqual(monitor.status_queries, 1) + self.assertEqual(len(monitor.restores), 1) + # A query that reports no restore line fails when it exits. + peer.sendall(STATUS_OK.encode()) + monitor.restores.clear() + with self.assertRaisesRegex(TimeAbiFailure, "restore marker"): + console.wait_for_time_abi_status(1.0) + with self.assertRaisesRegex(TimeoutError, "did not exit"): + console.wait_for_time_abi_status(0.2) + console.close() + peer.close() + unmonitored = socket.socket() + self.addCleanup(unmonitored.close) + with self.assertRaisesRegex(ValueError, "no time ABI monitor"): + openvmm_process.TcpConsole(unmonitored).wait_for_time_abi_status(1.0) + def test_tcp_console_queries_only_cold_boots_that_ask(self): for command, enabled in ( (MSHV_RESTORE, True), From 655692a93b72b609bf31aa13f5b111b5a95b22b8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 02:32:12 -0700 Subject: [PATCH 179/188] doc: name snapshot-tiers among the restores that CI queries 09ea1be4 makes every restored snapshot-tiers guest run nvx-time status before it exits, and the harness require a passing restore line. The one exception is the gate-timeout check, whose guest OpenVMM stops at the restore gate on purpose. The test matrix still counted those restores among the ones that CI only scans. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index aaf07da3c..fa04a08ec 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -2046,10 +2046,12 @@ every backend. Production paths print no marker, so the matrix driver runs `/sbin/nvx-time status` over the console after shell-ready and after every restore. CI runs it after every cold boot whose shell is on a console the harness reads, and after every restore in the `smp-snapshot`, -`restore-processors`, and `restore-downtime` scenarios. CI's other restores -and its guests without a shell, the managed lifecycle and one-shot workloads, -are only scanned for violation events; a failed check still powers the guest -off with status 193. Each query requires exit status 0 and, after a restore, +`restore-processors`, `restore-downtime`, and `snapshot-tiers` scenarios, +except `snapshot-tiers`' gate-timeout check, whose guest OpenVMM stops at the +restore gate on purpose. CI's other restores and its guests without a shell, +the managed lifecycle and one-shot workloads, are only scanned for violation +events; a failed check still powers the guest off with status 193. Each query +requires exit status 0 and, after a restore, a `phase=restore` line with `status=ok`, the restored `generation`, and `cpus` equal to the online CPUs; after a cold boot, a `phase=boot` line with `generation=0`. During fleet validation (`p6`), the matrix driver records From c143b4b030d7261cee879540deccfdf101d5c58c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 02:45:05 -0700 Subject: [PATCH 180/188] doctor: fail H7 when w32tm names no time source H7 parsed w32tm /query /status by its English Source and Leap Indicator labels and passed when neither was found: on a Windows display language other than English, the source became "unknown", the leap indicator empty, and nothing marked the host unsynchronized. H7 now fails, with the values it read, unless the output names a source and a leap indicator from 0 to 3, so qualification cannot certify an unverified time source. CI doesn't run H7; it runs in the doctor only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 2 +- scripts/nvx_tools/doctor.py | 12 ++++++++++-- scripts/test_time_abi.py | 18 ++++++++++++++++++ 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 07858a86e..6e75bfd81 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -300,7 +300,7 @@ ABI failure code starts its detail with the code in brackets, for example | H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, `CLOCK_MONOTONIC` on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | | H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | -| H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows | +| H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows. Output without the English `Source` and `Leap Indicator` lines, as on a Windows display language other than English, names no source and fails | H2, H4, and H5 use a dependency-free host probe, [`host_time_probe.rs`](../scripts/nvx_tools/host_time_probe.rs), which the diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 831c84469..19cf9de5b 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -862,14 +862,22 @@ def _windows_time_source() -> tuple[bool, str]: name, separator, value = line.partition(":") if separator: fields[name.strip()] = value.strip() - source = fields.get("Source", "unknown") + source = fields.get("Source", "") leap = fields.get("Leap Indicator", "") + # w32tm prints its labels in the display language. Output without both + # English labels and a 0-3 leap indicator names no source, so the check + # fails rather than certify an unverified one. + if not source or not leap.startswith(("0", "1", "2", "3")): + raise ScriptError( + "cannot verify the host time source: w32tm /query /status reported " + f"source={source or 'none'} leap_indicator={leap or 'none'}" + ) unsynchronized = ( leap.startswith("3") or "Local CMOS Clock" in source or "Free-running System Clock" in source ) - return not unsynchronized, f"source={source} leap_indicator={leap or 'unknown'}" + return not unsynchronized, f"source={source} leap_indicator={leap}" def check_utc(context: DoctorContext) -> CheckResult: diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 628b5e7ef..e96b47f87 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -1800,6 +1800,24 @@ def test_utc_check_reads_the_host_clock_discipline(self): ), ): self.assertFalse(doctor.check_utc(context).passed) + # Output that names no source, such as a localized w32tm's, fails H7 + # instead of passing as an unknown source. + for incomplete in ( + "Sprungindikator: 0(keine Warnung)\nQuelle: time.windows.com,0x9\n", + "Leap Indicator: 0(no warning)\nStratum: 2\n", + "Stratum: 2\nSource: VM IC Time Synchronization Provider\n", + status.replace("0(no warning)", "unknown"), + ): + with ( + patch.object(doctor, "host_is_windows", return_value=True), + patch.object( + doctor.subprocess, "run", return_value=completed(incomplete) + ), + contextlib.redirect_stdout(io.StringIO()), + ): + (result,) = doctor.run_checks(context, ("H7",)) + self.assertFalse(result.passed, incomplete) + self.assertIn("cannot verify the host time source", result.detail) def test_run_prints_lines_writes_the_summary_and_fails_closed(self): def passing(check: str): From 1941f6311a5748e83044e802506b020220f4fc5e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 03:15:09 -0700 Subject: [PATCH 181/188] doctor: fail closed on every host fact the gates can't read A follow-up to the H7 fix, from an audit of every parser of external tool output that the doctor and validate-runner gate on. - H7 starts every failure to read the host's state with "cannot verify host UTC synchronization": w32tm failing, timing out, or printing no English Source and Leap Indicator lines; adjtimex failing or missing; and a clock state above TIME_ERROR, which Linux doesn't define and which passed before. - H3 requires a cpu_profile that is a revision of a catalog profile even when H2 doesn't run in the same invocation; a missing or unknown one passed before. A revision is now the lineage, .v, and a number, for H2 too. - run_checks also turns arithmetic, index, and type errors into a failed check, so an unreadable value fails only its own check instead of stopping the doctor before the later checks and the summary. Checked, and already fail-closed: validate-runner's nonstop_tsc gate, H1, H2's CPU identity and OpenVMM's NVX-CPU-PROFILE line, H3's status, backend, rates, and LAPIC rate, H4's and H5's probe records, and H6's boot line and warp summaries. Evidence fields, such as the microcode and the invariant-TSC flags, may still read unknown; doc/ci.md states the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 9 +++-- scripts/nvx_tools/doctor.py | 62 +++++++++++++++++++++++++--------- scripts/test_time_abi.py | 67 ++++++++++++++++++++++++++++++------- 3 files changed, 108 insertions(+), 30 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 6e75bfd81..8aeed7c95 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -290,17 +290,20 @@ profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. H4 and H6 run on the spec's long qualification schedules, which take about three minutes; `--ci-schedule` selects CI's short ones. A failure that matches a time ABI failure code starts its detail with the code in brackets, for example -`[E_PROFILE_HOST_UNKNOWN]`. +`[E_PROFILE_HOST_UNKNOWN]`. A check fails, and never passes, when it can't +read a fact it gates on: missing or unrecognized tool output fails it, and an +error in one check fails only that check. Only evidence fields, such as the +microcode and the invariant-TSC flags, may read `unknown`. | Check | Implementation | | --- | --- | | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | | H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry, and the generation and the profile that `auto` selects from the spec's catalog, which shares one profile per generation across backends: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), or `emeraldrapids` (6/207, `intel.emeraldrapids.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. Then `openvmm --hypervisor --cpu-fingerprint ` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the generation's profile, and prints one `NVX-CPU-PROFILE:` line. H2 requires exit status 0, `status=pass`, the same backend and generation, and a revision of the generation's profile, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | -| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` | +| H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a `cpu_profile` that is a revision (`.v` and a number) of a catalog profile and, when H2 runs too, of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` | | H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, `CLOCK_MONOTONIC` on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | | H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | -| H7 | `adjtimex` reports no `STA_UNSYNC` on Linux; `w32tm /query /status` names a synchronized source on Windows. Output without the English `Source` and `Leap Indicator` lines, as on a Windows display language other than English, names no source and fails | +| H7 | `adjtimex` reports no `STA_UNSYNC` and no `TIME_ERROR` on Linux; `w32tm /query /status` names a synchronized source on Windows. When H7 can't read either, its detail starts with `cannot verify host UTC synchronization`: `adjtimex` fails or reports a clock state Linux doesn't define, or `w32tm` fails or prints no English `Source` and `Leap Indicator` lines, as in a Windows display language other than English | H2, H4, and H5 use a dependency-free host probe, [`host_time_probe.rs`](../scripts/nvx_tools/host_time_probe.rs), which the diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 19cf9de5b..0c1ec5621 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -29,6 +29,7 @@ from .common import ScriptError, artifact_path, openvmm_binary_path from .time_abi import ( CI_WARP_GAPS, + CPU_GENERATIONS, LAPIC_HZ, MAX_TSC_HZ, MIN_TSC_HZ, @@ -98,6 +99,10 @@ class Schedule: CPUINFO_PATH = Path("/proc/cpuinfo") ADJTIMEX_TIME_ERROR = 5 ADJTIMEX_STA_UNSYNC = 0x0040 +# Every H7 failure to read the host's synchronization state starts with this +# phrase; a clock that is not synchronized starts "host UTC is not +# synchronized" instead. +UTC_UNVERIFIED = "cannot verify host UTC synchronization" WHP_CAPABILITY_HYPERVISOR_PRESENT = 0 @@ -410,7 +415,8 @@ def _rust_unescape(match: re.Match[str]) -> str: def _same_profile_lineage(selected: str, expected: str) -> bool: """Whether ``selected`` is a revision of the profile ``expected`` names.""" - return selected.startswith(expected.rpartition(".v")[0] + ".v") + lineage = re.escape(expected.rpartition(".v")[0]) + return re.fullmatch(rf"{lineage}\.v\d+", selected) is not None def _check_cpu_profile( @@ -585,7 +591,16 @@ def check_openvmm_preflight(context: DoctorContext) -> CheckResult: ) expected_profile = context.facts.get("profile", "none") selected_profile = fields.get("cpu_profile", "") - if expected_profile != "none" and not _same_profile_lineage( + # A missing or unknown profile fails even when H2 doesn't run with H3. + if not any( + _same_profile_lineage(selected_profile, generation.profile_id) + for generation in CPU_GENERATIONS + ): + problems.append( + "OpenVMM verified no catalog CPU profile: " + f"cpu_profile={selected_profile or 'none'}" + ) + elif expected_profile != "none" and not _same_profile_lineage( selected_profile, expected_profile ): problems.append( @@ -833,11 +848,18 @@ def worst(name: str) -> int: def _linux_clock_state() -> tuple[int, int]: """Return adjtimex's clock state and status without changing the clock.""" - libc = ctypes.CDLL(None, use_errno=True) - buffer = ctypes.create_string_buffer(256) - state = int(libc.adjtimex(buffer)) + try: + libc = ctypes.CDLL(None, use_errno=True) + buffer = ctypes.create_string_buffer(256) + state = int(libc.adjtimex(buffer)) + except (AttributeError, OSError) as error: + raise ScriptError( + f"{UTC_UNVERIFIED}: adjtimex is unavailable: {error}" + ) from error if state < 0: - raise ScriptError(f"adjtimex failed: {os.strerror(ctypes.get_errno())}") + raise ScriptError( + f"{UTC_UNVERIFIED}: adjtimex failed: {os.strerror(ctypes.get_errno())}" + ) # struct timex: unsigned modes, then the long offset, freq, maxerror, and # esterror fields, then the int status. status = int.from_bytes(buffer.raw[40:44], sys.byteorder) @@ -845,16 +867,19 @@ def _linux_clock_state() -> tuple[int, int]: def _windows_time_source() -> tuple[bool, str]: - completed = subprocess.run( - ["w32tm", "/query", "/status"], - capture_output=True, - text=True, - timeout=30, - check=False, - ) + try: + completed = subprocess.run( + ["w32tm", "/query", "/status"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except (OSError, subprocess.SubprocessError, ValueError) as error: + raise ScriptError(f"{UTC_UNVERIFIED}: w32tm /query /status: {error}") from error if completed.returncode != 0: raise ScriptError( - "w32tm /query /status failed: " + f"{UTC_UNVERIFIED}: w32tm /query /status failed: " + (completed.stdout.strip() or completed.stderr.strip()) ) fields: dict[str, str] = {} @@ -869,7 +894,7 @@ def _windows_time_source() -> tuple[bool, str]: # fails rather than certify an unverified one. if not source or not leap.startswith(("0", "1", "2", "3")): raise ScriptError( - "cannot verify the host time source: w32tm /query /status reported " + f"{UTC_UNVERIFIED}: w32tm /query /status reported " f"source={source or 'none'} leap_indicator={leap or 'none'}" ) unsynchronized = ( @@ -889,6 +914,10 @@ def check_utc(context: DoctorContext) -> CheckResult: return CheckResult("H7", True, detail) state, status = _linux_clock_state() detail = f"adjtimex state={state} status=0x{status:04x}" + if state > ADJTIMEX_TIME_ERROR: + return CheckResult( + "H7", False, f"{UTC_UNVERIFIED}: unknown clock state; {detail}" + ) if state == ADJTIMEX_TIME_ERROR or status & ADJTIMEX_STA_UNSYNC: return CheckResult( "H7", False, f"host UTC is not synchronized (STA_UNSYNC); {detail}" @@ -916,10 +945,13 @@ def run_checks(context: DoctorContext, checks: Sequence[str]) -> list[CheckResul try: result = CHECKS[check](context) except ( + ArithmeticError, + IndexError, KeyError, OSError, RuntimeError, subprocess.SubprocessError, + TypeError, ValueError, ) as error: first = str(error).splitlines()[0] if str(error) else type(error).__name__ diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index e96b47f87..1178a6ff3 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -1431,6 +1431,15 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: completed(line.replace("backend=kvm", "backend=mshv")), "verified backend mshv", ), + # Without H2 in the same run, H3 still requires a catalog profile. + ( + completed(line.replace(" cpu_profile=intel.icelake-sp.v1", "")), + "verified no catalog CPU profile: cpu_profile=none", + ), + ( + completed(line.replace("intel.icelake-sp.v1", "interim.host.kvm.v1")), + "verified no catalog CPU profile: cpu_profile=interim.host.kvm.v1", + ), ) for result_value, message in cases: with self.subTest(message=message): @@ -1441,6 +1450,8 @@ def context_with_files(backend: str = "kvm") -> doctor.DoctorContext: for selected, passed in ( ("intel.icelake-sp.v2", True), ("intel.skylake-sp.v1", False), + # A revision is the lineage plus .v and a number. + ("intel.icelake-sp.v2-rc", False), # OpenVMM selects catalog profiles, so an interim one is a regression. ("interim.host.kvm.v1", False), ): @@ -1800,24 +1811,39 @@ def test_utc_check_reads_the_host_clock_discipline(self): ), ): self.assertFalse(doctor.check_utc(context).passed) - # Output that names no source, such as a localized w32tm's, fails H7 - # instead of passing as an unknown source. - for incomplete in ( - "Sprungindikator: 0(keine Warnung)\nQuelle: time.windows.com,0x9\n", - "Leap Indicator: 0(no warning)\nStratum: 2\n", - "Stratum: 2\nSource: VM IC Time Synchronization Provider\n", - status.replace("0(no warning)", "unknown"), + # Output H7 can't read, such as a localized w32tm's, fails it with a + # stable leading phrase instead of passing as an unknown source. + unverified = "cannot verify host UTC synchronization: " + for outcome in ( + completed( + "Sprungindikator: 0(keine Warnung)\nQuelle: time.windows.com,0x9\n" + ), + completed("Leap Indicator: 0(no warning)\nStratum: 2\n"), + completed("Stratum: 2\nSource: VM IC Time Synchronization Provider\n"), + completed(status.replace("0(no warning)", "unknown")), + completed("The service has not been started. (0x80070426)\n", 1), + subprocess.TimeoutExpired(["w32tm"], 30), ): with ( patch.object(doctor, "host_is_windows", return_value=True), - patch.object( - doctor.subprocess, "run", return_value=completed(incomplete) - ), + patch.object(doctor.subprocess, "run", side_effect=[outcome]), contextlib.redirect_stdout(io.StringIO()), ): (result,) = doctor.run_checks(context, ("H7",)) - self.assertFalse(result.passed, incomplete) - self.assertIn("cannot verify the host time source", result.detail) + self.assertFalse(result.passed, result.detail) + self.assertTrue(result.detail.startswith(unverified), result.detail) + # So does an adjtimex clock state Linux doesn't define, or no adjtimex. + with ( + patch.object(doctor, "host_is_windows", return_value=False), + contextlib.redirect_stdout(io.StringIO()), + ): + with patch.object(doctor, "_linux_clock_state", return_value=(6, 0)): + (unknown_state,) = doctor.run_checks(context, ("H7",)) + with patch.object(doctor.ctypes, "CDLL", side_effect=OSError("no libc")): + (no_adjtimex,) = doctor.run_checks(context, ("H7",)) + for result in (unknown_state, no_adjtimex): + self.assertFalse(result.passed, result.detail) + self.assertTrue(result.detail.startswith(unverified), result.detail) def test_run_prints_lines_writes_the_summary_and_fails_closed(self): def passing(check: str): @@ -1867,6 +1893,23 @@ def broken(context: doctor.DoctorContext) -> doctor.CheckResult: self.assertIn("**fail** | rustc is required", text) self.assertIn("Generation: `emeraldrapids`", text) + # Unreadable values that break a computation, such as a zero rate, + # fail only their own check, and the later checks still run. + def arithmetic(context: doctor.DoctorContext) -> doctor.CheckResult: + del context + raise ZeroDivisionError("float division by zero") + + stdout = io.StringIO() + with ( + patch.dict(doctor.CHECKS, {"H2": arithmetic, "H5": passing("H5")}), + contextlib.redirect_stdout(stdout), + ): + self.assertEqual(doctor.run(arguments), 1) + self.assertIn( + 'check=H2 status=fail detail="float division by zero"', stdout.getvalue() + ) + self.assertIn("check=H5 status=pass", stdout.getvalue()) + def test_run_selects_the_cpu_fingerprint_or_runs_without_openvmm(self): seen: list[Path | None] = [] schedules: list[doctor.Schedule] = [] From 8de6fd441f0584106e397d10f247bfcd81f6f3ef Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 05:18:28 -0700 Subject: [PATCH 182/188] openvmm: pin the time ABI v1 PR head with explicit test comparisons c61f60f14 makes three assertions in virt_support_x86emu's virt_tests target convert explicitly (u16::from, u8::from). The time ABI's cpu_profile crate brings serde_json into virt's dependency graph, and its PartialEq impls for the integer types made the bare .into() in those comparisons ambiguous (E0283), so CI's OpenVMM unit tests no longer compiled on any backend. The commit changes only those three test files; the OpenVMM sources and binaries are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 53c4d1ae2..c61f60f14 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 53c4d1ae2f1db503f59470418197a03fa186c0af +Subproject commit c61f60f14d25c37439eeab0328176480601960bd From aa61f941f9fdfdc8e1327de4fda69015d2eee713 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 06:16:57 -0700 Subject: [PATCH 183/188] test-microvm: keep smp-lapic as an explicit scenario (#286) #286 removes the duplicate counting-LAPIC precheck but keeps `test-microvm --scenario smp-lapic` for explicit local use, and lists removing the scenario as a non-goal. The time ABI hides TSC-deadline on every backend, so `smp` already runs on the one-shot counting LAPIC, and the scenario no longer boots with lapic=notscdeadline. smp-lapic now runs only when named: --scenario accepts it, and no default or debug-kernel suite includes it, so CI runs nothing twice. It runs smp and also asserts that no CPU lists tsc_deadline_timer, that each online CPU's clockevent device is the counting lapic, and that the boot line reports the backend's LAPIC rate for every CPU. The guest-isolation adversarial cases go back to smp-lapic, as on dev. doc/benchmarks.md lists the jobs that consume, gate, publish, or persist the KVM benchmark results, and doc/ci.md and doc/usage.md describe the explicit scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/benchmarks.md | 23 +++- doc/ci.md | 5 +- doc/usage.md | 4 +- scripts/nvx_tools/adversarial_broker.py | 2 +- .../adversarial_cases/guest-isolation.json | 4 +- scripts/nvx_tools/microvm_tests.py | 75 ++++++++-- scripts/test_microvm_tests.py | 129 +++++++++++++++++- 7 files changed, 223 insertions(+), 19 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index a5a754cb9..44f8b7221 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -323,8 +323,11 @@ LAPIC calibration is not success. A counter frozen at 12 can indicate Linux's `APIC timer disabled due to verification failure`; increasing the poll budget cannot repair it. The time ABI hides the TSC-deadline timer on every backend, so every guest uses the one-shot counting LAPIC, and the `smp` scenario that the microVM correctness jobs run at 1, 2, 4, and 8 -vCPUs covers it. There is no separate `lapic=notscdeadline` scenario or pre-benchmark LAPIC -gate. +vCPUs covers it. The benchmarks run no LAPIC gate of their own (#286), and no guest boots with +`lapic=notscdeadline`. `test-microvm --scenario smp-lapic` remains for explicit local use: it runs +`smp` and also asserts that no CPU lists `tsc_deadline_timer`, that each online CPU's clockevent +device is the counting `lapic`, and that the boot line reports the backend's LAPIC rate for every +CPU. No default suite or CI job runs it, so nothing runs twice. Captures do not wait for a clocksource: under the time ABI, Linux registers the `tsc` clocksource at `device_initcall`, before any guest work, so no capture can observe the transitional `tsc-early` window. @@ -522,6 +525,22 @@ metrics and every configured attempt. Each backend job publishes its p50 tables `$GITHUB_STEP_SUMMARY`. Pull-request regression checks compare KVM, MSHV, and WHP results with the latest base-branch history. +These jobs consume, gate, publish, or persist the KVM benchmark results (#286); MSHV and WHP +follow the same flow with their own platform jobs: +- `Platform / Linux / KVM / Virtual machine` (`platform-kvm`) runs the benchmarks and publishes the + results: the run-scoped `benchmark-linux-kvm-virtual-machine-` artifact, the per-attempt + diagnostics artifact, and the p50 tables in its step summary. +- `Performance regression gate` (`performance-gate`, pull requests only) consumes the three + platform artifacts once all three platform jobs succeeded and gates them against the base + branch's history. It publishes and persists nothing. +- `Persist performance baseline` (`performance-persist`, `dev` pushes only) consumes them and + persists them to `data/*.csv`. +- `Publish development release` (`release`, `dev` pushes only) doesn't read them, but it and + `performance-persist` run only when the three platform jobs succeeded and every correctness + job, `nvx-microvm-tests-kvm` included, succeeded or was skipped. +- `Required status check` fails unless every job that the change schedules, including + `platform-kvm`, `performance-gate`, and `nvx-microvm-tests-kvm`, has its expected result. + Linux CI uses a reduced device contract of zero warmups and one retained attempt. Windows CI discards one warmup and retains five attempts so file initialization cannot dominate its p50. Both CI contracts use one-second operation windows. Canonical baseline collection uses the diff --git a/doc/ci.md b/doc/ci.md index 8aeed7c95..0eea7157d 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -263,7 +263,10 @@ to succeed. `Publish development release` and `Persist performance baseline` run only on `dev` pushes in which every microVM test job succeeded or was skipped. The pull-request `Performance regression gate` reads only the platform jobs' results and publishes nothing. The counting LAPIC that the -benchmarks depend on is covered by the `smp` scenario of those jobs. The workflow uses the read-only OpenVMM deploy key stored in the +benchmarks depend on is covered by the `smp` scenario of those jobs. +`smp-lapic` repeats `smp` with the counting-LAPIC facts asserted, for explicit +local use only; no CI job runs it. [Benchmarks](benchmarks.md#ci-collection) +lists the jobs that consume, gate, publish, or persist the results. The workflow uses the read-only OpenVMM deploy key stored in the `OPENVMM_DEPLOY_KEY` Actions secret to fetch the private submodule at its pinned commit. Shared guest binaries and development release packages move through short-lived workflow artifacts alongside the OpenVMM handoff and benchmark diff --git a/doc/usage.md b/doc/usage.md index fc86728d9..271cc2234 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -250,7 +250,9 @@ python3 scripts/nvx.py test-microvm Runs NVX-owned Linux, SMP, virtio, sandbox, and snapshot correctness scenarios against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without -it, every scenario supported by the selected guest runs. Alpine remains the +it, every scenario supported by the selected guest runs, except `smp-lapic`, +which runs only when named: it repeats `smp` and also asserts that every CPU +uses the one-shot counting LAPIC. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and the sandbox-control-dependent `snapshot-tiers` scenario. Ubuntu also rejects diff --git a/scripts/nvx_tools/adversarial_broker.py b/scripts/nvx_tools/adversarial_broker.py index beb678bac..cccd151af 100644 --- a/scripts/nvx_tools/adversarial_broker.py +++ b/scripts/nvx_tools/adversarial_broker.py @@ -45,7 +45,7 @@ "restore-processors", "sandbox-blocks", "scratch-snapshot", - "smp", + "smp-lapic", "smp-snapshot", "snapshot-core", "snapshot-tiers", diff --git a/scripts/nvx_tools/adversarial_cases/guest-isolation.json b/scripts/nvx_tools/adversarial_cases/guest-isolation.json index ebfead2aa..90a57a3a6 100644 --- a/scripts/nvx_tools/adversarial_cases/guest-isolation.json +++ b/scripts/nvx_tools/adversarial_cases/guest-isolation.json @@ -43,7 +43,7 @@ }, { "id": "interrupt-topology-boundary", - "scenario": "smp", + "scenario": "smp-lapic", "description": "Exercise repeated interrupt and timer progress across the supported virtual processor topology.", "coverage": [ "lapic", @@ -56,7 +56,7 @@ }, { "id": "processor-interrupt-churn", - "scenario": "smp", + "scenario": "smp-lapic", "description": "Repeat bounded multi-processor scheduling, timer, and interrupt progress to expose CPU-side hangs or leaks.", "coverage": [ "cpu-churn", diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 779216740..788f4e75e 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -125,6 +125,12 @@ "restore-downtime", "snapshot-tiers", ) +# Scenarios that run only when named, never in a default suite. The time ABI +# hides TSC-deadline on every backend, so `smp` already runs on the one-shot +# counting LAPIC; `smp-lapic` repeats it and asserts the counting-LAPIC facts, +# for explicit local use (#286). +MICROVM_EXPLICIT_SCENARIOS = ("smp-lapic",) +SMP_LAPIC_COUNTING_MARKER = b"NVX-SMP-LAPIC-COUNTING-OK" MICROVM_PROCESSOR_COUNTS = (1, 2, 4, 8) MICROVM_TEST_SCRIPTS_DIR = Path(__file__).with_name("microvm_test_scripts") LIFECYCLE_COMPLETION_MARKER = b"NVX-LIFECYCLE-OK" @@ -254,8 +260,11 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: parser.add_argument( "--scenario", action="append", - choices=MICROVM_TEST_SCENARIOS, - help="scenario to run; repeat to select multiple (default: all)", + choices=(*MICROVM_TEST_SCENARIOS, *MICROVM_EXPLICIT_SCENARIOS), + help=( + "scenario to run; repeat to select multiple (default: all except " + "smp-lapic, which runs only when named)" + ), ) parser.add_argument( "--debug-kernel", @@ -1254,6 +1263,32 @@ def run_console_exit( ) +def counting_lapic_script(processors: int) -> str: + """Return a guest check that every CPU runs the one-shot counting LAPIC. + + The time ABI hides TSC-deadline, so no CPU lists ``tsc_deadline_timer``, + and each online CPU's clockevent device is ``lapic``, not + ``lapic-deadline``. + """ + return ( + "if grep -qw tsc_deadline_timer /proc/cpuinfo; then\n" + " echo SMP-LAPIC-FAIL tsc-deadline-exposed\n" + " exit 89\n" + "fi\n" + "cpu=0\n" + f'while [ "$cpu" -lt {processors} ]; do\n' + " device=$(cat /sys/devices/system/clockevents/clockevent$cpu/" + "current_device 2>/dev/null)\n" + ' if [ "$device" != lapic ]; then\n' + ' echo "SMP-LAPIC-FAIL cpu=$cpu clockevent=${device:-none}"\n' + " exit 89\n" + " fi\n" + " cpu=$((cpu + 1))\n" + "done\n" + f"echo {SMP_LAPIC_COUNTING_MARKER.decode()}\n" + ) + + def run_smp( executable: Path, kernel: Path, @@ -1264,6 +1299,7 @@ def run_smp( memory_mib: int, timeout: float, log_path: Path, + counting_lapic: bool = False, ) -> None: command = workload_boot_command( executable, @@ -1274,11 +1310,16 @@ def run_smp( "quiet loglevel=0", processors=processors, ) - result = run_guest_script( - command, + script = ( smp_probe_script(processors, exit_guest=False) + warp_probe_script() - + "nvx-exit 0\n", + + "nvx-exit 0\n" + ) + if counting_lapic: + script = counting_lapic_script(processors) + script + result = run_guest_script( + command, + script, WARP_PROBE_COMPLETION_MARKER, timeout=timeout, log_path=log_path, @@ -1289,6 +1330,21 @@ def run_smp( f"{processors}-vCPU guest finished without the SMP probe marker " f"{SMP_PROBE_COMPLETION_MARKER.decode()!r}" ) + if counting_lapic: + if not contains_output_line(output, SMP_LAPIC_COUNTING_MARKER): + raise RuntimeError( + f"{processors}-vCPU guest finished without the counting-LAPIC " + f"marker {SMP_LAPIC_COUNTING_MARKER.decode()!r}" + ) + # The boot line must also report the backend's LAPIC rate for all + # the CPUs; the status query alone doesn't check the CPU count. + monitor = TimeAbiMonitor(command) + monitor.feed(output) + monitor.finish() + try: + monitor.require_boot("the counting-LAPIC check", online_cpus=processors) + except TimeAbiFailure as error: + raise RuntimeError(f"{processors}-vCPU smp-lapic: {error}") from error check_warp_probe( result["text"], cpus=processors, @@ -4872,10 +4928,12 @@ def run(args: argparse.Namespace) -> int: timeout=args.timeout, output_dir=output_dir, ) - if "smp" in scenarios: + for scenario, counting_lapic in (("smp", False), ("smp-lapic", True)): + if scenario not in scenarios: + continue for processors in dict.fromkeys(args.processors): print( - f"Running microVM smp correctness ({processors} vCPU) " + f"Running microVM {scenario} correctness ({processors} vCPU) " f"on OpenVMM/{args.backend}" ) run_smp( @@ -4886,7 +4944,8 @@ def run(args: argparse.Namespace) -> int: processors, memory_mib=args.memory_mib, timeout=args.timeout, - log_path=output_dir / f"smp-{processors}.log", + log_path=output_dir / f"{scenario}-{processors}.log", + counting_lapic=counting_lapic, ) if "sandbox-blocks" in scenarios: print(f"Running microVM sandbox-block correctness on OpenVMM/{args.backend}") diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 49dd4f03b..554876710 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2604,8 +2604,10 @@ def test_smp_requires_the_probe_marker_and_a_passing_warp_probe(self): def test_smp_exercises_the_counting_lapic_without_a_duplicate_scenario(self): # The time ABI hides TSC-deadline on every backend, so the ordinary - # smp scenario already exercises the one-shot counting LAPIC (#286). + # smp scenario already exercises the one-shot counting LAPIC, and no + # default suite also runs smp-lapic (#286). self.assertNotIn("smp-lapic", microvm_tests.MICROVM_TEST_SCENARIOS) + self.assertNotIn("smp-lapic", microvm_tests.DEBUG_KERNEL_SCENARIOS) text = (b"NVX-SMP-PROBE-OK\r\n" + _warp_probe_output(4)).decode() with patch.object( microvm_tests, "run_guest_script", return_value={"text": text} @@ -2625,8 +2627,123 @@ def test_smp_exercises_the_counting_lapic_without_a_duplicate_scenario(self): self.assertTrue( script.startswith(benchmark.smp_probe_script(4, exit_guest=False)) ) + self.assertNotIn("tsc_deadline_timer", script) self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + def test_smp_lapic_remains_an_explicit_scenario(self): + # #286 keeps `test-microvm --scenario smp-lapic` for local use. + self.assertEqual(microvm_tests.MICROVM_EXPLICIT_SCENARIOS, ("smp-lapic",)) + args = nvx.parse_args( + ["test-microvm", "--backend", "kvm", "--scenario", "smp-lapic"] + ) + self.assertEqual(args.scenario, ["smp-lapic"]) + + def test_smp_lapic_asserts_the_counting_lapic_facts(self): + rates = f"tsc_hz=2793437000 lapic_hz={time_abi.LAPIC_HZ['kvm']}" + passing = ( + f"NVX-TIME-ABI: v=1 phase=boot status=ok cpus=4 {rates} " + "generation=0 elapsed_us=2390\r\n" + "NVX-TIME-ABI: v=1 phase=runtime status=synchronized generation=0 " + "discontinuities=0 offset_ns=-1200 uncertainty_ns=900 " + "rejected_samples=0 last_sample_error=none\r\n" + "NVX-TIME-STATUS-EXIT status=0\r\n" + "NVX-SMP-LAPIC-COUNTING-OK\r\n" + "NVX-SMP-PROBE-OK\r\n" + _warp_probe_output(4).decode() + ) + + def run_smp_lapic(text: str) -> MagicMock: + with patch.object( + microvm_tests, "run_guest_script", return_value={"text": text} + ) as run: + microvm_tests.run_smp( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + 4, + memory_mib=128, + timeout=60, + log_path=Path("smp-lapic-4.log"), + counting_lapic=True, + ) + return run + + command, script, marker = run_smp_lapic(passing).call_args.args + self.assertEqual(command[command.index("--cmdline") + 1], "quiet loglevel=0") + self.assertTrue(script.startswith(microvm_tests.counting_lapic_script(4))) + self.assertIn(benchmark.smp_probe_script(4, exit_guest=False), script) + self.assertEqual(marker, time_abi.WARP_PROBE_COMPLETION_MARKER) + for text, message in ( + ( + passing.replace("NVX-SMP-LAPIC-COUNTING-OK\r\n", ""), + "without the counting-LAPIC marker", + ), + (passing.replace("cpus=4 tsc_hz", "cpus=2 tsc_hz"), "cpus=2 is not 4"), + ): + with ( + self.subTest(message=message), + self.assertRaisesRegex(RuntimeError, message), + ): + run_smp_lapic(text) + + def test_counting_lapic_check_requires_the_counting_lapic_on_every_cpu(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + script = ( + microvm_tests.counting_lapic_script(2) + .replace("/proc/cpuinfo", "cpuinfo") + .replace("/sys/devices/system/clockevents/", "clockevents/") + ) + for name, flags, devices, status, expected in ( + ("counting", "fpu tsc apic", ("lapic", "lapic"), 0, "COUNTING-OK"), + ( + "deadline-flag", + "fpu tsc_deadline_timer apic", + ("lapic", "lapic"), + 89, + "SMP-LAPIC-FAIL tsc-deadline-exposed", + ), + ( + "deadline-device", + "fpu tsc apic", + ("lapic", "lapic-deadline"), + 89, + "SMP-LAPIC-FAIL cpu=1 clockevent=lapic-deadline", + ), + ( + "missing-device", + "fpu tsc apic", + ("lapic",), + 89, + "SMP-LAPIC-FAIL cpu=1 clockevent=none", + ), + ): + with self.subTest(name=name), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / "cpuinfo").write_text( + f"processor\t: 0\nflags\t\t: {flags}\n", encoding="ascii" + ) + for cpu, device in enumerate(devices): + directory = root / "clockevents" / f"clockevent{cpu}" + directory.mkdir(parents=True) + (directory / "current_device").write_text( + device + "\n", encoding="ascii" + ) + result = subprocess.run( + [shell], + cwd=root, + input=script, + text=True, + capture_output=True, + timeout=10, + check=False, + ) + self.assertEqual( + result.returncode, status, result.stdout + result.stderr + ) + self.assertIn(expected, result.stdout) + def test_virtio_net_uses_portable_endpoint_policy(self): with ( patch.object( @@ -4123,7 +4240,7 @@ def test_runner_dispatches_selected_scenarios_once(self): args = argparse.Namespace( backend="whp", guest="alpine", - scenario=["smp", "smp"], + scenario=["smp", "smp", "smp-lapic", "smp-lapic"], processors=[2, 2, 8], memory_mib=128, timeout=60.0, @@ -4148,11 +4265,15 @@ def require(path: Path, _description: str) -> Path: run_lifecycle.assert_not_called() self.assertEqual( [entry.args[4] for entry in run_smp.call_args_list], - [2, 8], + [2, 8, 2, 8], ) self.assertEqual( [entry.kwargs["log_path"].name for entry in run_smp.call_args_list], - ["smp-2.log", "smp-8.log"], + ["smp-2.log", "smp-8.log", "smp-lapic-2.log", "smp-lapic-8.log"], + ) + self.assertEqual( + [entry.kwargs["counting_lapic"] for entry in run_smp.call_args_list], + [False, False, True, True], ) def test_runner_dispatches_public_managed_exec_configuration(self): From 860710112ef14d3fae5fc081338e9b8eb77f453d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 06:20:09 -0700 Subject: [PATCH 184/188] ci: key both kernel caches on docker/Dockerfile The debug-kernel cache key, which this branch adds, and the production kernel's key, which predates it, omitted docker/Dockerfile, although both kernels build inside the image that file pins, with its compiler, toolchain, and the kernel and kernel-debug-artifacts targets. A Dockerfile-only toolchain or target change therefore restored an older vmlinux or vmlinux-debug instead of rebuilding it. The image keys already included it. The test now requires the Dockerfile in every guest cache key and checks that every docker build uses it. The first run after this change rebuilds both kernels once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/build-guest-artifacts/action.yml | 4 ++-- doc/ci.md | 11 +++++++---- scripts/test_nvx_tools.py | 13 +++++++++++-- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/.github/actions/build-guest-artifacts/action.yml b/.github/actions/build-guest-artifacts/action.yml index e28d4aecc..6366432c7 100644 --- a/.github/actions/build-guest-artifacts/action.yml +++ b/.github/actions/build-guest-artifacts/action.yml @@ -71,8 +71,8 @@ runs: ALPINE_VERSION: ${{ steps.guest-versions.outputs.alpine-version }} UBUNTU_VERSION: ${{ steps.guest-versions.outputs.ubuntu-version }} AZURELINUX_VERSION: ${{ steps.guest-versions.outputs.azurelinux-version }} - KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/patches/**', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} - DEBUG_KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/config-microvm-debug', 'kernel/patches/**', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} + KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/patches/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} + DEBUG_KERNEL_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'kernel/config-microvm', 'kernel/config-microvm-debug', 'kernel/patches/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py') }} ALPINE_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'guest/alpine/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py') }} UBUNTU_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'guest/ubuntu/**', 'ubuntu/**', 'docker/Dockerfile', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py', 'scripts/nvx_tools/ubuntu.py') }} AZURELINUX_INPUT_HASH: ${{ hashFiles('SOURCE-MANIFEST.json', 'guest/common/**', 'azurelinux/**', 'docker/Dockerfile', 'scripts/nvx_tools/azurelinux.py', 'scripts/nvx_tools/build.py', 'scripts/nvx_tools/build_config.py', 'scripts/nvx_tools/build_constants.py', 'scripts/nvx_tools/common.py', 'scripts/nvx_tools/guests.py') }} diff --git a/doc/ci.md b/doc/ci.md index 0eea7157d..8c3593886 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -242,11 +242,14 @@ carried in `OpenVmmBuildConfig`; the build workflow maps KVM, MSHV, or WHP to GNU, musl, or MSVC without probing runtime devices. CI therefore retains its musl build for MSHV without maintaining a separate shell build path. -The kernel and initramfs cache keys include +The kernel, debug kernel, and initramfs cache keys include +[`docker/Dockerfile`](../docker/Dockerfile), which pins the build image and its +toolchain and in which every guest artifact builds, and [`build_config.py`](../scripts/nvx_tools/build_config.py) and -[`build_constants.py`](../scripts/nvx_tools/build_constants.py), so shared build -configuration or constant changes invalidate cached guest artifacts and their -provenance. The Ubuntu distro layer shares the Ubuntu input hash. +[`build_constants.py`](../scripts/nvx_tools/build_constants.py), so a toolchain, +shared build configuration, or constant change invalidates cached guest +artifacts and their provenance. The Ubuntu distro layer shares the Ubuntu input +hash. The producer handoff uses one-day workflow artifacts rather than caches. Each consumer downloads both the normalized executable and its build provenance, diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 7443d94b7..6930d9aa7 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -5809,18 +5809,27 @@ def test_ci_guest_cache_keys_include_build_configuration(self): / "build-guest-artifacts" / "action.yml" ).read_text(encoding="utf-8") - for cache_name in ( + cache_names = ( "KERNEL_INPUT_HASH", + "DEBUG_KERNEL_INPUT_HASH", "ALPINE_INPUT_HASH", "UBUNTU_INPUT_HASH", "AZURELINUX_INPUT_HASH", - ): + ) + # Every artifact, both kernels included, builds inside the image that + # docker/Dockerfile pins, so a toolchain change must miss every cache. + self.assertEqual( + action.count("--file docker/Dockerfile"), + action.count("docker build"), + ) + for cache_name in cache_names: with self.subTest(cache=cache_name): cache_input = next( line for line in action.splitlines() if line.strip().startswith(f"{cache_name}:") ) + self.assertIn("'docker/Dockerfile'", cache_input) self.assertIn("'scripts/nvx_tools/build_config.py'", cache_input) self.assertIn("'scripts/nvx_tools/build_constants.py'", cache_input) From c2dbbb99f3acc1b0c63f436a0329d68271a23c1d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 06:27:34 -0700 Subject: [PATCH 185/188] doc: document doctor and every CLI option in the usage reference doc/usage.md is the CLI reference, but this branch added the doctor command without a row in its command table or a usage section, and added --debug-kernel to test-microvm and build without listing it in their synopses. It now documents doctor (the synopsis, each option with its default, and its fail-closed exit status), and both synopses list --debug-kernel, with test-microvm's debug-kernel scenarios and artifacts. A new test reads every command's --help usage, including performance's subcommands, and requires the command table and each section to name exactly the options that the parser accepts. doctor's --kernel and --initrd help now names H3, whose preflight boots them too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/usage.md | 49 ++++++++++++++++++++++++++++++++-- scripts/nvx_tools/doctor.py | 4 +-- scripts/test_nvx_tools.py | 53 +++++++++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 4 deletions(-) diff --git a/doc/usage.md b/doc/usage.md index 271cc2234..fba908d90 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -36,6 +36,7 @@ python3 scripts/nvx.py performance gate --help | `test-openvmm-unit` | Run the OpenVMM workspace unit and documentation tests. | | `test-openvmm` | Run self-contained OpenVMM microVM control-plane tests. | | `test-microvm` | Run NVX Linux and device correctness tests through OpenVMM. | +| `doctor` | Qualify this host for the NVX time ABI. | | `test-aci-edge-sandboxes` | Run the `aci_edge_sandboxes` Rust crate lifecycle test on a real hypervisor. | | `test-adversarial` | Run a brokered Copilot-driven adversarial campaign. | | `build` | Build the guest artifacts and OpenVMM. | @@ -203,6 +204,7 @@ prerequisite is missing. python3 scripts/nvx.py build [--guest {alpine,ubuntu,azurelinux,all}] [--native] + [--debug-kernel] [--skip-restore] [--backend {kvm,mshv,whp}] ``` @@ -242,6 +244,7 @@ python3 scripts/nvx.py test-microvm --backend {kvm,mshv,whp} [--guest {alpine,ubuntu,azurelinux}] [--scenario SCENARIO]... + [--debug-kernel] [--processors {1,2,4,8} ...] [--memory-mib MIB] [--timeout SECONDS] @@ -256,10 +259,52 @@ uses the one-shot counting LAPIC. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and the sandbox-control-dependent `snapshot-tiers` scenario. Ubuntu also rejects -the Alpine-prompt-specific `console-snapshot` scenario. The -command requires `build/vmlinux`, the selected initramfs, and +the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots +the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task +detectors the guest's time ABI watcher reports; without `--scenario`, it runs +only the same-host restore scenarios `smp`, `smp-snapshot`, +`restore-processors`, `restore-downtime`, and `snapshot-tiers`. The command +requires `build/vmlinux` (with `--debug-kernel`, `build/vmlinux-debug` and its +`build/vmlinux-debug.config`), the selected initramfs, and `openvmm/target/release/openvmm[.exe]`. +### `doctor` + +```text +python3 scripts/nvx.py doctor + --backend {kvm,mshv,whp} + [--checks ID ...] + [--openvmm PATH] + [--kernel PATH] + [--initrd PATH] + [--cpu-fingerprint PATH | --no-openvmm] + [--ci-schedule] + [--probe-dir PATH] + [--summary PATH] + [--timeout SECONDS] +``` + +Qualifies this host for the [time ABI](design/time-abi.md#host-qualification). +It runs the selected checks in spec order, prints one +`NVX-DOCTOR: check= status= detail="..."` line per check, and +exits with status 1 if any check fails. A check fails, and never passes, when +it can't read a fact it gates on. [CI](ci.md#host-qualification) describes each +check and the subset that CI runs. + +| Option | Default | Description | +| --- | --- | --- | +| `--backend {kvm,mshv,whp}` | required | Select the backend to qualify. | +| `--checks ID ...` | `H1` to `H7` | Run only these checks, from `H1` to `H7`, in spec order. | +| `--openvmm PATH` | `openvmm/target/release/openvmm[.exe]` | Select the OpenVMM binary for `H2`, `H3`, and `H6`. | +| `--kernel PATH` | `build/vmlinux` | Select the guest kernel for `H3` and `H6`. | +| `--initrd PATH` | `build/initramfs.cpio.gz` | Select the guest initramfs for `H3` and `H6`. | +| `--cpu-fingerprint PATH` | `nvx-cpu-fingerprint-.json` in the probe directory | Select where `H2` writes OpenVMM's CPU fingerprint. | +| `--no-openvmm` | off | Qualify without an OpenVMM binary. `H2` then checks the CPU identity and generation but not the CPU profile, and `--checks` must exclude `H3` and `H6`, which boot OpenVMM. | +| `--ci-schedule` | off | Run `H4` and `H6` on CI's short schedules: 3 rate samples 1 s apart instead of 13 samples 10 s apart, and two warp-probe runs instead of five. | +| `--probe-dir PATH` | `$RUNNER_TOOL_CACHE/nvx-host-time-probe`, or `build/host-time-probe` | Select the cache directory for the host probe, which the doctor builds with `rustc`. | +| `--summary PATH` | none | Append a Markdown summary, for example to `$GITHUB_STEP_SUMMARY`. | +| `--timeout SECONDS` | `120` | Set the seconds allowed for each probe or guest. | + ### `test-aci-edge-sandboxes` ```text diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index 0c1ec5621..62777be95 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -1071,8 +1071,8 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: parser.add_argument( "--openvmm", type=Path, help="OpenVMM binary for H2, H3, and H6" ) - parser.add_argument("--kernel", type=Path, help="guest kernel for H6") - parser.add_argument("--initrd", type=Path, help="guest initramfs for H6") + parser.add_argument("--kernel", type=Path, help="guest kernel for H3 and H6") + parser.add_argument("--initrd", type=Path, help="guest initramfs for H3 and H6") openvmm = parser.add_mutually_exclusive_group() openvmm.add_argument( "--cpu-fingerprint", diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 6930d9aa7..fedb7dfd8 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -428,6 +428,59 @@ def _write_release_fixture( class CliTests(unittest.TestCase): + def test_usage_documents_every_command_and_option(self): + # doc/usage.md is the CLI reference: every command has a row in its + # command table and a section that names exactly the options of the + # command's --help usage. + option = re.compile(r"(? str: + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout), self.assertRaises(SystemExit): + nvx.parse_args([*command, "--help"]) + return stdout.getvalue().split("\n\n", 1)[0] + + def subcommands(usage: str) -> list[str]: + # A subcommand list reads "{a,b} ...", which argparse may wrap + # before the dots; an option's repeated choices read + # "{a,b} [{a,b} ...]". + match = re.search(r"(? str: + match = re.search( + rf"^{'#' * level} `{re.escape(title)}`\n(.*?)(?=^#{{2,{level}}} |\Z)", + document, + re.MULTILINE | re.DOTALL, + ) + self.assertIsNotNone(match, f"doc/usage.md has no section for {title}") + return match.group(1) if match else "" + + table = document.split("## Commands\n", 1)[1].split("\n## ", 1)[0] + commands = subcommands(usage_of()) + self.assertEqual( + set(re.findall(r"^\| `([a-z0-9-]+)` \|", table, re.MULTILINE)), + set(commands), + ) + for name in commands: + usage = usage_of(name) + nested = subcommands(usage) + section(name, 3) + targets = ( + [((name, child), 4) for child in nested] if nested else [((name,), 3)] + ) + for command, level in targets: + title = " ".join(command) + with self.subTest(command=title): + self.assertEqual( + set(option.findall(section(title, level))), + set(option.findall(usage_of(*command))), + ) + def test_guest_selection_includes_azure_linux(self): args = nvx.parse_args(["build-initramfs", "--guest", "azurelinux"]) self.assertEqual(args.guest, "azurelinux") From 763abc12d4441166e022f41fbbefa69c80c9223b Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 06:30:36 -0700 Subject: [PATCH 186/188] run: state the guest memory defaults from the guest descriptors This branch raised the Ubuntu guest's default RAM to 512 MiB, at which its root unpacks completely, but `nvx.py run --help` still offered 256 MiB for Ubuntu, and doc/usage.md's run table said the same; at 256 MiB the Ubuntu root is truncated. The help now names each guest's default from its descriptor (alpine 128, ubuntu 512, azurelinux 512), the table reads 512 for Ubuntu, and a test checks both against the descriptors. No other doc or README names a stale Ubuntu default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/usage.md | 2 +- scripts/nvx.py | 9 ++++++++- scripts/test_nvx_tools.py | 20 ++++++++++++++++++++ 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/doc/usage.md b/doc/usage.md index fba908d90..644a3ba34 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -451,7 +451,7 @@ python3 scripts/nvx.py run | `--guest {alpine,ubuntu,azurelinux}` | `alpine` | Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. This option is not used for snapshot restore. | | `--hypervisor {auto,whp,kvm,mshv}` | `auto` | Select the OpenVMM hypervisor. `auto` chooses WHP on Windows and KVM elsewhere. | | `--machine {microvm}` | `microvm` | Select the fixed-topology microVM with shared-status edge interrupts. | -| `--memory-mib MIB` | guest-specific | Set guest memory in MiB. Defaults to 128 for Alpine, 256 for Ubuntu, and 512 for Azure Linux. | +| `--memory-mib MIB` | guest-specific | Set guest memory in MiB. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux. | | `--memory-capacity-mib MIB` | none | Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot. | | `--processors {1,2,4,8}` | `1` | Select the microVM processor count. | | `--mount GUEST_TARGET,HOST_PATH[,ro\|rw]` | none | Expose one host directory to the absolute guest target. An `rw` mapping accepts guest-created symbolic links, which the host never follows. Active snapshot restore requires the same canonical path, target, and mode; a dormant-slot restore may attach a new mapping that the resumed guest mounts explicitly. | diff --git a/scripts/nvx.py b/scripts/nvx.py index 7f20aa1cc..2977a53d2 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -863,7 +863,14 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: run.add_argument( "--memory-mib", type=int, - help="guest RAM; defaults to 128 MiB for Alpine and 256 MiB for Ubuntu", + help=( + "guest RAM in MiB (default by --guest: " + + ", ".join( + f"{name} {guest_descriptor(name).default_memory_mib}" + for name in GUEST_NAMES + ) + + ")" + ), ) run.add_argument("--memory-capacity-mib", type=int) run.add_argument("--processors", type=int, choices=(1, 2, 4, 8), default=1) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index fedb7dfd8..6437038ea 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -489,6 +489,26 @@ def test_guest_selection_includes_azure_linux(self): "initramfs-azurelinux.cpio.gz", ) + def test_run_help_and_usage_state_each_guest_memory_default(self): + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout), self.assertRaises(SystemExit): + nvx.parse_args(["run", "--help"]) + help_text = " ".join(stdout.getvalue().split()) + usage = (BuildConstants.REPO_ROOT / "doc" / "usage.md").read_text( + encoding="utf-8" + ) + row = next( + line + for line in usage.splitlines() + if line.startswith("| `--memory-mib MIB`") + ) + labels = {"alpine": "Alpine", "ubuntu": "Ubuntu", "azurelinux": "Azure Linux"} + for name in guests.GUEST_NAMES: + default = guests.guest_descriptor(name).default_memory_mib + with self.subTest(guest=name): + self.assertIn(f"{name} {default}", help_text) + self.assertIn(f"{default} for {labels[name]}", row) + def test_benchmark_exposes_device_restore_profile(self): args = nvx.parse_args( [ From b9f6e4b9d502c1685511ccfcb2c4e3abceb57e6a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 07:16:33 -0700 Subject: [PATCH 187/188] doc: say what the time ABI removes from smp-lapic 2f420aab keeps smp-lapic as an explicit scenario for local use (#286). No guest boots with lapic=notscdeadline, no CI job runs the scenario, and the benchmarks run no LAPIC gate of their own: smp covers the counting LAPIC that every guest now uses. The time ABI's removed list still said that the scenarios were gone. It now names what was removed, as doc/ci.md and doc/benchmarks.md describe it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/time-abi.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/design/time-abi.md b/doc/design/time-abi.md index fa04a08ec..a9bc2e4ba 100644 --- a/doc/design/time-abi.md +++ b/doc/design/time-abi.md @@ -2171,10 +2171,12 @@ Removed from NVX: - Harness: `tsc=reliable` and `no_timer_check` in `BASE_TUNING`; per-backend `clocksource=` tokens; `stable_clocksource_wait_script` and the capture-time clocksource waits; the KVM and WHP branches of - `snapshot-core`; the `smp-lapic` scenarios (`lapic=notscdeadline`), which - duplicate `smp`; `restore-tsc-sync`, `clearcpuid=tsc_adjust`, and the - fresh-boot TSC control, which the warp probe replaces; and the dmesg greps - in `restore-processors.sh`, which the watcher replaces. The + `snapshot-core`; `smp-lapic`'s `lapic=notscdeadline` boot, its runs in CI, + and the benchmarks' LAPIC gate, which `smp` makes redundant now that every + guest uses the counting LAPIC (#286; the scenario remains for explicit + local use); `restore-tsc-sync`, `clearcpuid=tsc_adjust`, and the fresh-boot + TSC control, which the warp probe replaces; and the dmesg greps in + `restore-processors.sh`, which the watcher replaces. The `cold_start_clocksource` benchmark scenario passes `clocksource=tsc` on every backend. From a453866e8337cf68f5902067c32831121fedb3ad Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 3 Oct 2026 07:21:46 -0700 Subject: [PATCH 188/188] openvmm: pin the time ABI v1 PR head with the KVM soundness fixes 974f35652 makes set_msr_filter hand KVM zero-padded copies of each MSR filter range's bitmap in whole 64-bit words, because KVM copies BITS_TO_LONGS(nmsrs) * sizeof(long) bytes and read past the one-byte bitmaps of the time ABI's ranges. Every bit stays where KVM tests it. It also replaces the safe Vp::enable_cap, which passed any capability and arguments to KVM_ENABLE_CAP, with enable_enforce_pv_feature_cpuid, its only use, and fixes two test-only issues in virt_mshv and virt_whp. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index c61f60f14..974f35652 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit c61f60f14d25c37439eeab0328176480601960bd +Subproject commit 974f356521829b96798ee6ac30cc5d1f0e1bb87b