Track doctor qualification bugs: optional clocksource read fails H4 #507

Hunt scope and revision

• Repository: microsoft/nvx.
• Latest origin/dev fetched for this hunt: a5df986e5e606a82add90646fb6a3f9de0a2ec69.
• Focus: the uncovered doctor qualification component, specifically prerequisite/error handling and the boundary between optional evidence and required measurements.
• Existing repo:microsoft/nvx label:bug-hunt issues were checked first. Host/storage telemetry and control-session timeout handling were excluded after reading Track benchmark result collection bugs: stale device-I/O logs survive suite changes #499 and Track managed sandbox lifecycle bugs: on Windows, provision waits out its timeout for a lock file that refuses access for good #464; no tracker for the doctor component was found.
• Tests and reads used a git archive of this exact revision outside the worktree. No repository files were edited and no commits were created. Investigation stopped at the first confirmed bug.

1. Optional clocksource evidence incorrectly gates H4

Severity: Medium. An unavailable evidence-only sysfs file rejects H4 without measuring the TSC at all. This can falsely reject Linux KVM/MSHV host qualification and fail the Linux validate-runner prerequisite gate. This is a confirmed diagnostic/error-path bug, not a claim of an observed CI outage or defective host clock.

Trigger

On the Linux branch, /sys/devices/system/clocksource/clocksource0/current_clocksource cannot be read, for example because the sysfs mount does not expose that file. The smallest reproduction uses an actually missing temporary file and otherwise passing, deterministic native-probe records. The missing-file case is confirmed; no live Linux sysfs or host configuration was changed.

Expected versus actual

• Expected: Report the unavailable clocksource as evidence and still run the TSC rate/stability measurements. H4's verdict should depend on the required measurements and thresholds, not this optional label.
• Actual: host_clocksource() raises FileNotFoundError; run_checks() turns it into an H4 failure. The native probe is never called, and the real CLI entrypoint returns 1. The same native-counter fixture passes when only the evidence file is made readable.

Trace and pinned permalinks

1. H4's documented contract explicitly says the host clocksource never gates.
2. host_clocksource() performs an unguarded read_text().
3. check_rate() reads that optional evidence before calling run_probe().
4. run_checks() catches the resulting OSError and emits status=fail. doctor.run() returns 1 for the failed check, and the CLI propagates that status.
5. The Linux runner action already tolerates an unreadable clocksource when reporting evidence, but then invokes doctor with H4 as a required qualification check.
6. Existing coverage: the H4 test helper always returns a successfully read clocksource; the evidence-only cases vary the successful value but do not test a read failure.

Smallest reproduction

Run this Python code from the repository root at the pinned revision, using python -B (or python3 -B). It needs no hypervisor, OpenVMM binary, Rust build, or Linux host. It exercises the actual CLI/parser/check/error-handling path, selects the Linux branch with one platform patch, supplies known-good native measurements, and performs real file reads against temporary files outside the repository.

import contextlib
import io
import sys
import tempfile
from pathlib import Path
from unittest.mock import patch

sys.path.insert(0, str(Path.cwd() / "scripts"))
import nvx
from nvx_tools import doctor

records = [
    ("clock", {"role": role, "name": name, "hz": "1000000000", "resolution_ns": "1"})
    for role, name in (("rate", "CLOCK_MONOTONIC"), ("stability", "CLOCK_MONOTONIC_RAW"))
]
for index in range(3):
    sample = {"index": str(index)}
    for role in ("rate", "stability"):
        sample.update({f"{role}_tsc": str(index * 3_000_000_000),
                       f"{role}_clock": str(index * 1_000_000_000),
                       f"{role}_bracket": "20"})
    records.append(("sample", sample))

with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    existing = root / "current_clocksource"
    existing.write_text("tsc\n", encoding="utf-8")
    statuses = []
    for label, clocksource in (("readable evidence", existing),
                              ("unavailable evidence", root / "unmounted-sysfs" / "current_clocksource")):
        output = io.StringIO()
        with (patch.object(doctor, "host_is_windows", return_value=False),
              patch.object(doctor, "CLOCKSOURCE_PATH", clocksource),
              patch.object(doctor, "run_probe", return_value=records) as probe,
              contextlib.redirect_stdout(output)):
            status = nvx.main(["doctor", "--backend", "kvm", "--checks", "H4",
                               "--no-openvmm", "--ci-schedule", "--probe-dir", str(root)])
        statuses.append(status)
        print(f"{label}: exit={status}, native_probe_calls={probe.call_count}")
        print(output.getvalue().strip())
    assert statuses[0] == 0
    assert statuses[1] == 0, "H4 must not fail solely because evidence-only clocksource is unreadable"

Observed output (temporary paths normalized):

readable evidence: exit=0, native_probe_calls=1
NVX-DOCTOR: check=H4 status=pass detail="tsc_hz=3000000000 against CLOCK_MONOTONIC from 3 samples 1 s apart; against CLOCK_MONOTONIC_RAW (resolution 1 ns) the interval rates agree within 0.000 ppm, largest interval uncertainty 0.021 ppm; not compared with the backend's rate (H3 did not run); host clocksource tsc (evidence)"
Time ABI host qualification on kvm: passed
unavailable evidence: exit=1, native_probe_calls=0
NVX-DOCTOR: check=H4 status=fail detail="[Errno 2] No such file or directory: '<temp>/unmounted-sysfs/current_clocksource'"
Time ABI host qualification on kvm: failed (H4)
AssertionError: H4 must not fail solely because evidence-only clocksource is unreadable

Suggested fix

Handle OSError narrowly around the optional clocksource read, retaining an explicit diagnostic/unknown-evidence value, and continue running the native probe and applying all H4 thresholds. Do not suppress measurement failures or make qualification succeed when required counters are unavailable. Add missing/unreadable clocksource regression cases with both passing and failing measurements.

Validation and limits

• Local environment: Windows, Python 3.14.7; CI's configured Python 3.10 was not exercised.
• python -B -m unittest scripts.test_time_abi.DoctorTests -q: 16 tests, OK.
• The reproduction above fails its final expected-behavior assertion, confirming the false negative; its control passes and isolates the evidence file as the only changed input.
• Real doctor --backend whp --checks H1 --no-openvmm: passed.
• Real doctor --backend whp --checks H4 --no-openvmm --ci-schedule --probe-dir <external-dir> --timeout 30: passed. Its unmodified native Rust probe reported 0.055 ppm interval agreement and 0.100 ppm uncertainty. Probe build/cache artifacts were placed outside the repository.
• python -B scripts\nvx.py verify in the original worktree returned 1 because the OpenVMM submodule is uninitialized (openvmm\Cargo.toml absent). It was not initialized or repaired because this hunt must preserve repository files.
• No live Linux execution, guest boot, OpenVMM build, full hardware qualification, benchmark, or complete cross-platform suite was performed. The confirmed finding is the Linux Python error path with deterministic native-counter fixtures, not hardware behavior.

Checked, not filed

• Host/storage telemetry: the existing 27 telemetry tests passed (one Linux-only case skipped on Windows); this area was excluded because Track benchmark result collection bugs: stale device-I/O logs survive suite changes #499 already records its exploration.
• Control-session framing/timeouts: excluded because Track managed sandbox lifecycle bugs: on Windows, provision waits out its timeout for a lock file that refuses access for good #464 already covers that component.
• Local WHP API detection and actual native H4 measurements passed; no bug filed for them.
• Doctor CPU-profile fallback/preflight, guest-warp scheduling, UTC synchronization, and check-error isolation: inspected through the existing doctor tests; no additional defect confirmed. In particular, conservative failure on localized/unreadable w32tm output is intentional and tested, not a new finding.
• Native affinity/skew error paths were inspected, but no independent bug was reproduced. Investigation did not continue after the confirmed H4 finding.