Repository navigation
release-check: assert the NPU is held awake (power/control=on, runtime_status=active) - #1231
Conversation
…e_status=active) Root cause this row exists for (c29 live-USB take, 2026-11-05 22:17:41): the NPU's driver runtime-PM suspends the array between requests, and the request that followed such a window died `npu runtime: invalid call (code 614): state 8, run 0 of 28` - the picture page's prompt turn. Lemonade answered 500 and the fallback model wrote the prompt, so the client saw a clean answer while the demo lost the beat, and dmesg carried one `aie2_legacy_tdr_detect` plus a power-cycle of the array. Owner decision (2026-11-05 23:4x ADT, "keep the npu awake on the stick"): the c30 fix is to never let the array runtime-suspend - a udev rule setting power/control=on for the XDNA2 NPU plus an early-boot fallback for a module that loaded before udev - and the release check gets a row for it. This commit is the row (the image half is PR #1230: 90-1bit-npu-power.rules and 04onboard-npu-keepawake). It reads the state THIS boot has rather than trusting a rule file, and asserts both facts, because power/control=on with runtime_status!=active is a different fault with a different message. The device is matched on PCI device id 0x17f0 - the same id the image's udev rule matches (pci 1022:17f0) and the firmware row tests for - so a machine without an NPU skips in the image's own words ("no XDNA2 NPU"). It waits up to 10 s for a just-resumed device to report active rather than reding a boot for the driver's workqueue, and it joins HW_SKIPPED so a VM footer names it instead of implying it ran. Measured: * `os/mini/release-check.sh --self-test` PASS, 73 cases (7 new: on/active is the PASS; auto/suspended is the c29 take's state; auto/active, on/suspended, on/unreadable and two no-device cases give FAIL/SKIP with distinct text). * the row block run against a real amdxdna device on hardware prints `PASS npu-power : the NPU is held awake (0000:c6:00.1): power/control=on, runtime_status=active`, with zero clients holding /dev/accel/accel0 - i.e. control=on keeps the array up when nothing is using it, which is the property the fix relies on. * board expectation changes from 25 to 26 checks (docs/run sheets that quote the count are the release lane's to update).
1b8d2b8 to
f8ebc74
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at f8ebc744.
Both measured claims reproduce here.
os/mini/release-check.sh --self-test -> release-check self-test PASS, and I counted the cases rather than taking the
number: 73 ok lines, exactly as the body says. The seven new ones fire with the verdicts you describe:
ok PASS power/control=on, runtime_status=active
ok FAIL power/control=auto (runtime_status=suspended): the array may runtime-suspend …
ok FAIL power/control=auto (runtime_status=active): …
ok FAIL power/control=on but runtime_status=suspended: held awake and not running
ok FAIL power/control=on but runtime_status is unreadable, so the array's state is not proven
ok SKIP no power/control to read (x2: device absent, both fields absent)
The live row, lifted out of the shipped script and run against this box's real sysfs (function + row block taken
verbatim from the diff, helpers stubbed) gives your exact line:
PASS npu-power : the NPU is held awake (0000:c6:00.1): power/control=on, runtime_status=active
with the device itself reporting on / active. bash -n rc 0; tools/copyright.py --check rc 0; no new files. The
shellcheck warnings that -S warning shows (SC1007, line 296) are pre-existing — origin/main's copy of the file
produces the same 5, and none is inside your hunks.
The design choices are the right ones and I checked them rather than nodding: npu-power joins HW_SKIPPED, so a VM's
footer names it instead of implying it ran; the device is found by id under /sys/bus/pci/devices/* rather than
0000:c6:00.1; and control=on + runtime_status!=active gets its own message, which is the distinction the c29 take
needed (it would have read auto/suspended).
Two nits, neither blocking:
- A kernel that reports
runtime_status=unsupportedwould red this row with "held awake and not running", when it
actually means the device cannot runtime-suspend at all — i.e. the property the row exists for already holds. Not
reachable on amdxdna as far as I can see (PM off means nopower/dir and the row skips), but it is a false red on a
release gate, so acasearm that PASSes with a note would cost nothing. - The device is matched on the PCI device id alone, where the image's udev rule matches vendor and device. That is
the same asymmetry I flagged on #1230's04onboard-npu-keepawake— worth making the three agree (0x1022check), so
the row cannot report a wrong-vendor0x17f0as the NPU. Here the loop also keeps the last match (nobreak),
which matters only in that impossible case.
Board count 25 -> 26 is consistent with what I read: the row is one pass/fail/skip on every boot that has an NPU,
and a skip on every VM boot.
I did not run it on the board — the on/active row is this box's, and the c29-shaped red (auto/suspended) is the
self-test's, which is the right instrument for it.
PR Reviewer Guide 🔍(Review updated until commit 4b7c37c)Here are some key observations to aid the review process:
|
|
Persistent review updated to latest commit 6316c7f |
|
Persistent review updated to latest commit 4b7c37c |
Owner decision (2026-11-05 23:4x ADT): "keep the npu awake on the stick" — c30's fix for the live-USB take's
code 614. This PR is the release-check half; the image half (the udev rule that setspower/control=onfor amdxdna, plus the early-boot fallback for a module loaded before udev) is the os/payload lane's.Root cause the row exists for
c29 take, 2026-11-05 22:17:41: the NPU's driver runtime-PM suspends the array between requests, and the request that followed such a window died
npu runtime: invalid call (code 614): state 8, run 0 of 28— the picture page's prompt turn. Lemonade answered 500, the fallback model wrote the prompt, and the client saw a clean answer while the demo lost the beat. dmesg in the same second:aie2_legacy_tdr_detect+power-cycling NPU to recover+1 context(s) skipped. Earlier NPU turns in that boot were fine (81.88 tok/s), so nothing about the model or the prompt was wrong.What the row asserts
Two facts, read from the state this boot has — not from the intent of a rule file:
power/controlison(the array cannot runtime-suspend), andruntime_statusisactive(it is actually running).They are separate messages on purpose:
control=onwithruntime_status!=activeis a different fault fromcontrol=auto. The device is found through/sys/bus/pci/drivers/amdxdna/[0-9]*, not a hardcoded PCI address, and the row waits up to 10 s for a just-resumed device to reportactiverather than reding a boot for the driver's workqueue. It joinsHW_SKIPPED, so a VM's footer names it instead of implying it ran.Measured
os/mini/release-check.sh --self-testPASS, 73 cases (7 new):on/activeis the PASS;auto/suspendedis the c29 take's state;auto/active,on/suspended,on/unreadable, and two no-device cases give FAIL/SKIP with distinct text.PASS npu-power : the NPU is held awake (0000:c6:00.1): power/control=on, runtime_status=active— with zero clients holding/dev/accel/accel0, which is the property the fix depends on (control=onkeeps the array up when nothing is using it).tools/copyright.py --checkrc 0; no new files.Merging is the owner's (overnight rule: no merges by lanes). If this does not land before the c30 cut, the image carries c29's 25-check release check and the board's
u4-release-checkrow will show it — the c30 hand-off says which it is.