[Pedro Henrique Penna (ppenna)]

Fixes #1008.

Summary

From 2026-10-28 23:00Z to 2026-10-29 15:00Z, H4 failed 9 of the 75 microVM and platform jobs on azure-mshv-1, and 8 CI runs with them. The runner's TSC keeps its rate, but the Hyper-V PTP clock that chronyd follows wanders. H4 gated the host's current clock correction, the TSC's error plus that wander, against 100 ppm. The time ABI v1 had set that bound for the TSC's error alone, and no short measurement can tell the two apart.

This PR implements the issue's proposed fix, one commit per item:

1. Gate on the guest's budget (doctor: gate H4 on the guest's 250 ppm clock correction budget). H4's limit is now 250 ppm: Linux's 500 ppm frequency limit less the time ABI's 250 ppm restore tolerance. The doctor derives RATE_TOLERANCE_PPM from those two constants. A new unit test reads FREQUENCY_LIMIT and RATE_DEVIATION_LIMIT from nvx-rs, and TSC_TOLERANCE_PPM from the OpenVMM gitlink's commit, as the CPU profile tests read OpenVMM's catalog, and checks that H4's limit is their difference.
2. Re-read chronyd before declaring a result inconclusive (doctor: re-read chronyd's estimate before H4 calls a rate inconclusive). When the range within chronyd's skew straddles the limit, H4 re-reads the estimate every 2 s for up to 60 s and uses the first conclusive one, whether it passes or fails, keeping the same samples. Otherwise it fails as inconclusive, as before.
3. Warn above 100 ppm (doctor: warn when H4's deviation exceeds 100 ppm). A passing deviation above 100 ppm passes with a warning. The warning gives the share of the budget that the host's correction uses and says that one run can't tell the TSC's error from the reference's wander. It leads the H4 detail, the step summary repeats it, and in GitHub Actions a ::warning:: annotation does too.
4. Use the kernel's correction without chronyd (doctor: read a kernel PLL's frequency in H4 without chronyd). Without chronyd's estimate, H4 corrects the rate by adjtimex's frequency plus its tick's, while the kernel reports the clock synchronized with its PLL on (STA_PLL), as systemd-timesyncd and ntpd run it. Otherwise it measures the CLOCK_MONOTONIC window as before. chronyd keeps the PLL off and slews through the frequency, so H4 never reads it from chronyd. H4 and H7 now share one ctypes struct timex, and H7's output is unchanged.
5. Report the average correction since boot, as evidence (doctor: report H4's average correction since boot as evidence). On Linux, H4 reports the rate of CLOCK_MONOTONIC_RAW against CLOCK_MONOTONIC over the uptime, and the deviation that it gives. It doesn't gate on it.

doc/ci.md's H4 row describes the limit with its rationale, the re-reads, the warning, the kernel PLL, and the since-boot evidence. The wall-clock discipline in doc/design/3.2-clock-and-time.md ties H4's limit to the budget, and doc/usage.md says how the doctor reports a warning.

These still fail closed as before: the interval checks against CLOCK_MONOTONIC_RAW, the rate's plausibility, an unreadable or unrecognized chronyd report, H6, and the time ABI's restore tolerance.

Validation

Local (Windows):

• python -m unittest scripts/test_time_abi.py passes 74 tests. The budget test reads OpenVMM at the pinned c32c57d12, and mutating either constant fails it.
• ruff check, ruff format --check, pyright --pythonplatform Linux and Windows, check_markdown_links.py, and compileall pass.
• The rest of validate-nvx's list passes except 27 shell-script failures in test_setup_azure.py and test_microvm_tests.py. Those fail identically on a clean dev worktree on Windows.

Linux (nvx-mshv-test, Azure Linux 3, Python 3.12):

• test_time_abi.py passes, including the budget test with the submodule and the Linux branch of the since-boot test.
• Specula, test_adversarial.py, and test_hosts.py pass.
• The full validate-nvx list passes once scripts: test the mutants merge against the shards' indexed outcomes #1023 fixed dev's mutants tests, which ci: fix nvx-rs mutants summarize miscounting shards #952 had broken.

Read-only H4 runs on the runners. I ran CI's H4 schedule from /tmp, with each runner's declared rate from its latest H3 line, alternating dev's doctor and this one:

Host	Rounds	chronyd's estimate (skew)	dev H4 failures	New H4 failures	Since-boot average
azure-mshv-1 (wandering)	24	+57.1 to +97.1 ppm (up to 26.6)	5, inconclusive at 100 ppm	0	+43.39 to +43.46 ppm over 40 h
azure-mshv-3 (steady)	12	+9.9 to +10.1 ppm (≤ 0.45)	0	0	+9.39 ppm over 40 h

I also drove the remaining paths on real hardware with synthetic declared rates:

• Warning (azure-mshv-3, 150 ppm): H4 passes, the detail leads with the warning, and the annotation prints with 60%25 escaped.
• Re-read that resolves (azure-mshv-3, 250 ppm): H4 re-read for 24 s until chronyd's range lay wholly above the limit, then failed conclusively.
• Re-read that stays inconclusive (azure-mshv-1, 250 ppm): no re-read in 58 s was conclusive, so H4 failed as inconclusive.
• Fallback (prometheus30): chronyd runs with cmdport 0, so unprivileged chronyc can't read it. adjtimex reports the PLL off (status=0x0000), so H4 measures the window and states both reasons as evidence.

Since-boot survey: on the 5 KVM runners and the 4 steady MSHV runners, the average matched chronyd's current estimate within 2.0 ppm after about 40 h, and within 0.2 ppm on prometheus32 after 63 days. On azure-mshv-1, it read +43.4 ppm while chronyd's estimate wandered from +21 to +97 ppm.

No configured host runs a PLL-mode time service; all run chronyd. NVX's own guest, though, drives the kernel PLL as systemd-timesyncd does. In an Alpine microVM on nvx-mshv-test, the PLL's frequency, which kernel_frequency() reads, changed only at the discipline's polls and held at −0.4 to −1.9 ppm. Over the same seconds, the 1 s CLOCK_MONOTONIC/RAW window swung between −13 and −42 ppm with the phase slews (details in the PR comments). Before the discipline synchronized, the kernel reported TIME_ERROR with STA_UNSYNC, which kernel_frequency() rejects.

Notes

• time ABI: publish the host's clock correction with each time sample #1018 (for time ABI: publish the host's clock correction with each time sample instead of having guests estimate it #1009, the structural fix) rewrites H4 in the same files. Whichever of the two merges second needs a rebase.
• This supersedes the unmerged branch agents/issue-1008-h4-guest-budget, an earlier attempt at the same plan.

[Copilot]

🔵 Needs a closer look

The kernel-PLL qualification path has only mocked coverage and needs validation on a synchronized Linux host with STA_PLL enabled to confirm that its selected frequency excludes phase slews.

No open issues 🎉

What changed in this PR

Updates the doctor’s H4 host qualification check to use the guest’s 250 ppm clock-correction budget, addressing failures caused by wandering reference clocks in #1008 without changing the time ABI or restore tolerance.

Changes:

• Derives H4’s limit from the guest’s frequency budget and warns on passing deviations above 100 ppm.
• Retries inconclusive chronyd estimates, adds a synchronized kernel PLL fallback, and reports since-boot correction as evidence.
• Adds tests and documents the qualification behavior and warning output.

File	Description
scripts/​test_time_abi.py	Tests budget constants, retries, PLL correction, since-boot evidence, and warnings.
scripts/​nvx_tools/​doctor.py	Implements the revised H4 gate, correction sources, retries, and reporting.
doc/​usage.md	Explains warnings in doctor output.
doc/​design/​3.2-clock-and-time.md	Connects host qualification to the guest’s correction budget.
doc/​ci.md	Documents H4’s limits, fallback behavior, evidence, and retry duration.

_{Review effort: Balanced}

[Pedro Henrique Penna (ppenna)]

Following up on the review's note that the kernel PLL path had only mocked coverage: I validated it on a real synchronized kernel PLL.

No configured host runs a PLL-mode time service; all run chronyd, which keeps STA_PLL off. NVX's own guest, though, disciplines its wall clock exactly as systemd-timesyncd does, with ADJ_OFFSET under STA_PLL|STA_NANO (nvx-rs/src/guest/sys/discipline.rs). So I booted an Alpine microVM on nvx-mshv-test (MSHV) with nvx-rs run and ran a static probe there every second for 150 s. The probe printed adjtimex's state and status, the remaining phase offset, and the frequency that kernel_frequency() reads (freq / 2^16 + (tick - 10000) * 100), beside the rate of CLOCK_MONOTONIC against CLOCK_MONOTONIC_RAW over that second.

Seconds after start	adjtimex	PLL frequency (doctor)	Remaining phase offset	CLOCK_MONOTONIC vs RAW over 1 s
0 to 16	state=5 status=0x2040 (unsynchronized)	—	0	±0.03 ppm
20	state=0 status=0x2001 (STA_PLL|STA_NANO)	+0.000 ppm	−0.97 ms	−15.4 ppm
50	same	−0.972 ppm	−2.25 ms	−37.0 ppm
70	same	−1.659 ppm	−2.52 ms	−42.3 ppm
120	same	−1.659 ppm	−1.15 ms	−20.2 ppm
150	same	−1.919 ppm	−3.89 ms	−17.2 ppm

• The PLL's frequency changes only at the discipline's polls, at about 17, 32, 48, 64, and 128 s, and holds between them. Over the same seconds the 1 s window swings by tens of ppm with the phase slew. So adjtimex's frequency excludes the PLL's phase slews, which is the premise of the fallback, while the window that H4 used before would have read up to −42 ppm against a −1.7 ppm frequency.
• Until the discipline synchronizes, the kernel reports TIME_ERROR with STA_UNSYNC, which kernel_frequency() rejects. That leaves H4 on the window, as designed.
• The probe's struct timex is 208 bytes in the guest too. On the host, whose chronyd holds tick=9999, the doctor's formula with the tick term gave −72.555 ppm, against a measured −72.558 ppm between slews.