Repository navigation
puppy: the NPU never runtime-suspends on the stick (the c30 614 fix) - #1230
Conversation
…fline update check
The c29 take's /var/log/messages carries 780 `auth.err ... warning:
/etc/hosts.allow, line 1: can't verify hostname: getaddrinfo(ryzen, AF_INET)
failed` lines in 19 minutes (665 from 192.168.50.100, 111 from
192.168.50.61), one per ssh connection. The appliance's runtime file is Puppy's
stock `ALL: LOCAL` followed by the LAN rule zz-onboard-ports appends, and libwrap
takes the FIRST matching rule: `LOCAL` is a name test (dotless reverse name), so
every connection made libwrap reverse-resolve the client and forward-verify the
name it got back. With no DNS search domain that verification fails.
Same file, same box, tcpdmatch 7.6.q-36 on a client address:
ALL: LOCAL -> 192.168.50.100 denied, 8.8.8.8 denied
ALL: LOCAL + LAN line (c29 runtime) -> 192.168.50.100 granted, 8.8.8.8 denied,
127.0.0.1 denied
the LAN line alone (this commit) -> 192.168.50.100 granted, 8.8.8.8 denied,
127.0.0.1 granted
So line 1 grants nothing by itself and loopback was riding on a resolver; the
address rule does all the work. The file is now WRITTEN with addresses only (no
`LOCAL`, no `LOCALHOST`, loopback spelled 127.0.0.1), so no connection can
trigger a name lookup, and the allow decision no longer depends on the router
answering a dotless `ryzen` (a `ryzen.lan` answer would have made line 1 stop
matching and hosts.deny refuse the owner's own machine).
Second: the same take's lemond log has 9 `Model update check failed ... CURL
error: Could not resolve hostname` lines at boot and one `logind Inhibit failed`
(Puppy has no logind). Both are config.json keys of the pinned payload, measured
from its own strings - `auto_check_model_updates` and `inhibit_suspend` - and
`lemond --help` has only -v/--version, so config.json is the only place to set
them. 10lemonade now seeds both false when they are absent (a config a person
edited keeps its value). The manual models/check_updates route still runs, so
updates stay askable; what stops is a startup check that cannot answer and can
move a model under a demo that pinned one.
vm-check's U7 hosts row writes the same address-only rule when an image lacks it
(so the login row can still be measured on an older artifact) and prints
name-rules=<n>, 0 on this image and 1 on c29's.
…e_status=active) Root cause this row exists for (c29 live-USB take, 2026-11-05 22:17:41): the NPU's driver runtime-PM suspends the array between requests, and the request that followed such a window died `npu runtime: invalid call (code 614): state 8, run 0 of 28` - the picture page's prompt turn. Lemonade answered 500 and the fallback model wrote the prompt, so the client saw a clean answer while the demo lost the beat, and dmesg carried one `aie2_legacy_tdr_detect` plus a power-cycle of the array. Owner decision (2026-11-05 23:4x ADT, "keep the npu awake on the stick"): the c30 fix is to never let the array runtime-suspend - a udev rule setting power/control=on for the XDNA2 NPU plus an early-boot fallback for a module that loaded before udev - and the release check gets a row for it. This commit is the row (the image half is PR #1230: 90-1bit-npu-power.rules and 04onboard-npu-keepawake). It reads the state THIS boot has rather than trusting a rule file, and asserts both facts, because power/control=on with runtime_status!=active is a different fault with a different message. The device is matched on PCI device id 0x17f0 - the same id the image's udev rule matches (pci 1022:17f0) and the firmware row tests for - so a machine without an NPU skips in the image's own words ("no XDNA2 NPU"). It waits up to 10 s for a just-resumed device to report active rather than reding a boot for the driver's workqueue, and it joins HW_SKIPPED so a VM footer names it instead of implying it ran. Measured: * `os/mini/release-check.sh --self-test` PASS, 73 cases (7 new: on/active is the PASS; auto/suspended is the c29 take's state; auto/active, on/suspended, on/unreadable and two no-device cases give FAIL/SKIP with distinct text). * the row block run against a real amdxdna device on hardware prints `PASS npu-power : the NPU is held awake (0000:c6:00.1): power/control=on, runtime_status=active`, with zero clients holding /dev/accel/accel0 - i.e. control=on keeps the array up when nothing is using it, which is the property the fix relies on. * board expectation changes from 25 to 26 checks (docs/run sheets that quote the count are the release lane's to update).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a1ece5fa. Every row in the body reproduces here, including arm A on the real device; three notes, none
of which needs to hold c30.
The rule is the box's own, character for character. diff of the non-comment line against
/etc/udev/rules.d/90-1bit-npu-power.rules is empty, and this box is in exactly the state the rule exists to produce —
0000:c6:00.1, Signal processing controller: … Strix/Krackan/Strix Halo Neural Processing Unit, power/control=on,
runtime_status=active. udevadm verify on the committed file: 1 rules file checked, Success 1, Fail 0.
All four fallback arms, run as shipped (arm A against the real sysfs, B-D against fixture trees):
A this box -> "... 0000:c6:00.1 power/control already on (90-1bit-npu-power.rules did it)"
"... control=on runtime_status=active" rc 0
B no NPU -> "no PCI device 0x17f0 after 15s - this machine has no XDNA2 NPU, nothing to keep awake" rc 0
C control=auto -> "was 'auto', wrote on (FALLBACK: the udev rule missed ...)", file now `on` rc 0
D unwritable -> stderr warning, file unchanged (`auto`), state line rc 0
sh -n rc 0, shellcheck -S warning rc 0, mode in the tree is 100755 (git ls-tree), tools/copyright.py --check
rc 0 — and the *.rules addition is the right shape, two lines in the # list with the same reasoning the .desktop
entry carries.
Two image facts I checked rather than assumed, because "it is in the image" is the claim that matters: the container
build installs the whole embed tree (container-build.sh:255, cp -a /recipe/embed/. "$skeleton/"), so both paths land
with no manifest edit; and the runner really does give the ordering the script needs — /var/tmp/c28embed/etc/rc.d/ rc.services:10 is for service_script in /etc/init.d/* (lexicographic, so 04 before 10), :12 runs only if [ -x … ],
and its only skip is grep -qF dbus-daemon, which this script does not contain (0 occurrences). It also ignores the
start argument rc.services passes, so the body runs on the real boot path.
Three notes:
- The fallback matches on device id only, the rule on vendor and device. Reproduced with a fixture where the
decoy sorts first (0000:00:01.0, vendor0x1234, device0x17f0,control=auto) and the real NPU is
0000:01:00.1alsoauto: the script writesonto the wrong device, prints the success line, and leaves the
NPU asleep — i.e. it reports "kept awake" while the 614 is still reachable, which is the one failure mode this script
exists to make loud. One line fixes it ([ "$(cat "$dev/vendor")" = 0x1022 ] || continue) and makes the fallback
agree with the rule it is the fallback for. - Arm B pays the full 15 s on every boot of a machine with no NPU (measured: 15 s wall). Fine on the board, which
finds the device immediately; on the QEMU gates it is 15 s added to every boot, before10lemonadestarts. If the VM
gates time their boot, 5 s is plenty for module enumeration. - If c30 is cut with
patch-sfs.sh --manifest(the c29 route) rather than the container build, the manifest needs two
lines for these paths and the third field 100755 for the init script — that tool's manifest is explicit, so the
embed copy is the only route that gets it for free.
No device claim from me either: the on/active row is this box's, the guest row cannot exist, and the 20 s-idle / 0 x 614
arm is pi-npufeat's and the release check's. Stacked on #1224 (os/v3-c30-log-cleanup) — merge #1224 first.
PR Reviewer Guide 🔍(Review updated until commit d404014)Here are some key observations to aid the review process:
|
a1ece5f to
b68af3a
Compare
Owner decision 2026-11-05 23:4x: instead of waiting for the driver-side resume fix, the appliance simply never
lets the XDNA2 NPU autosuspend, because the 5 s autosuspend timer is what puts the array into the state whose
first dispatch hangs:
npu runtime: invalid call (code 614): state 8, run 0 of 28 -> aie2_set_cmd_timeout -> power-cycle
Two files, and both are needed for different failures:
- /etc/udev/rules.d/90-1bit-npu-power.rules - ACTION=="add|bind", SUBSYSTEM=="pci", vendor 0x1022 device
0x17f0, TEST=="power/control", ATTR{power/control}="on". This is the rule this dev box has run since
2026-10-12 for the same reason (keep a persistent context), so it is copied measured, not invented. `add|bind`
covers both orders: the driver binding after udev is up, and the module arriving before it.
- /etc/init.d/04onboard-npu-keepawake - the early-boot fallback for the case where the rule misses (rules.d not
installed, a coldplug with neither event, a kernel whose device has no power/control). It waits up to 15 s for
PCI 0x17f0, then reports what it found: "already on (the udev rule did it)", "was 'auto', wrote on (FALLBACK:
the udev rule missed)", "could not write ... the 614 can come back", or "this machine has no XDNA2 NPU, nothing
to keep awake" (exit 0 - every QEMU gate boots without one). Runs before 10lemonade, so the device is awake
before the server that opens a context.
Measured, on this box and on fixtures - four arms of the fallback plus the rule's own syntax:
A this box (NPU at 0000:c6:00.1, control already on) -> "already on (90-1bit-npu-power.rules did it)", control=on runtime_status=active, rc 0
B no NPU (a QEMU guest's shape) -> one line, rc 0
C control=auto in a fixture tree -> "was 'auto', wrote on (FALLBACK: ...)", file now on
D control=auto and unwritable -> the warning on stderr, file unchanged, rc 0
udevadm verify -> 1 rules file checked, 0 failed
The .rules file needed a comment style for tools/copyright.py (udev rules are '#' comments, as the copied rule's
first line shows), so `*.rules` joins the '#' list - the same shape as the .desktop entry in cycle 16.
Not measured here, and not claimed: the device is not in a QEMU guest, so no VM boot can show control=on; the row
that proves it is the release check's on the board (pi-release) plus pi-npufeat's W arm (20 s idle between
requests, >=20 runs, 0 x 614). What this commit owes is that the rule is valid, the fallback behaves in all four
states, and both are in the image.
b68af3a to
d27ab39
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at b68af3a3 — my READY at a1ece5fa carries, and I re-ran the checks at the new head.
The delta is one file and comment-only (git diff a1ece5fa..HEAD: 90-1bit-npu-power.rules, +4/-2, every changed line a
#). The rule line is still byte-identical to this box's /etc/udev/rules.d/90-1bit-npu-power.rules (diffed again,
empty), udevadm verify still 1 checked / 0 fail, copyright --check rc 0, and the init script is still committed
100755. Nothing else in the PR moved, so the four fallback arms, the rc.services ordering facts and the
container-build.sh install path from my earlier review are unchanged.
One thing to fix while you are in here, because the new text claims more than it delivers: the line you quote as
"verbatim" is not. The box's own comment ends at …so the lane can keep a persistent context., but the quoted line
continues:
# 1bit (owner 2026-10-12): … can keep a persistent context. The `TEST` guard means the rule is inert on a kernel whose device has no
# power/control attribute, and the vendor/device ids mean it never touches another device: …
so the provenance line now carries a sentence of our own prose on its end (and runs to ~200 columns, against the file's
~110). The fix is to close the quote after context. and let "The TEST guard …" start a fresh comment line. Worth doing
precisely because the quote is the evidence that the rule was not invented.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at d27ab39d. The delta since b68af3a3 is the stop verb and nothing else, and the shutdown claim it
rests on is true — I read it out of the image's own runner:
/var/tmp/c28embed/etc/rc.d/rc.services:10 for service_script in /etc/init.d/* (boot: `start`, in sort order)
/var/tmp/c28embed/etc/rc.d/rc.shutdown:80-84 for service_script in $(ls /etc/init.d/* | sort -r) … $service_script stop
so yes, every script is called with stop at shutdown, and answering it with the 15 s wait would be a shutdown delay.
Measured on the shipped file, this box's real sysfs:
sh 04onboard-npu-keepawake stop -> 0.00s rc 0 (nothing printed)
sh 04onboard-npu-keepawake restart -> 0.00s rc 0 (same arm; harmless)
sh 04onboard-npu-keepawake start -> "… power/control already on (90-1bit-npu-power.rules did it)"
"… control=on runtime_status=active" rc 0
sh 04onboard-npu-keepawake -> same as `start` (the ${1:-start} default is right for a hand run)
start keeps the body intact (fixture C re-run at this head: auto -> written on, FALLBACK line, rc 0), sh -n rc 0,
shellcheck -S warning rc 0, copyright --check rc 0, mode still 100755. Nothing else in the PR moved, so the rule
line is still byte-identical to this box's live rule and my earlier four-arm table stands.
*) exit 0 also swallows an unknown verb silently, which for an idempotent keepawake check is the right trade — but it
does mean a typo (stat) reads as "did nothing, fine". Not worth changing; noting it so the shape is deliberate.
Still open, from my last verdict and not touched by this push: the line you quote as the box's comment "verbatim" is
not — it ends …keep a persistent context. The \TEST` guard means the rule is inert on a kernel whose device has no. Close the quote after context. and start "The \TEST` guard …" on its own line. Cosmetic, so it does not hold the merge;
I would rather say it once more than let the evidence line be wrong.
|
Reviewed the two files at 1. The rule line is the one this box has actually run since 2026-10-12. Diffed the rule line in the PR against 2. 3. Measured, on this box, with the shipped payload binary: with 4. Two nits, neither blocking:
5. One suggestion for the release-check row (pi-release's file, so only if it costs them nothing): assert the Nothing here to change before the cut; approve from my side once pi-prs has looked at it. |
|
The count the order asked for, on the shipped payload binary with this rule's effect in place: 20/20 clean, 0 × 614.
Same arm, same binary, same model, Harness (for whoever re-runs it): Nothing in this PR needs changing for that result; my review above stands (the one thing to protect is |
|
Persistent review updated to latest commit d404014 |
…e_status=active) (#1231) Root cause this row exists for (c29 live-USB take, 2026-11-05 22:17:41): the NPU's driver runtime-PM suspends the array between requests, and the request that followed such a window died `npu runtime: invalid call (code 614): state 8, run 0 of 28` - the picture page's prompt turn. Lemonade answered 500 and the fallback model wrote the prompt, so the client saw a clean answer while the demo lost the beat, and dmesg carried one `aie2_legacy_tdr_detect` plus a power-cycle of the array. Owner decision (2026-11-05 23:4x ADT, "keep the npu awake on the stick"): the c30 fix is to never let the array runtime-suspend - a udev rule setting power/control=on for the XDNA2 NPU plus an early-boot fallback for a module that loaded before udev - and the release check gets a row for it. This commit is the row (the image half is PR #1230: 90-1bit-npu-power.rules and 04onboard-npu-keepawake). It reads the state THIS boot has rather than trusting a rule file, and asserts both facts, because power/control=on with runtime_status!=active is a different fault with a different message. The device is matched on PCI device id 0x17f0 - the same id the image's udev rule matches (pci 1022:17f0) and the firmware row tests for - so a machine without an NPU skips in the image's own words ("no XDNA2 NPU"). It waits up to 10 s for a just-resumed device to report active rather than reding a boot for the driver's workqueue, and it joins HW_SKIPPED so a VM footer names it instead of implying it ran. Measured: * `os/mini/release-check.sh --self-test` PASS, 73 cases (7 new: on/active is the PASS; auto/suspended is the c29 take's state; auto/active, on/suspended, on/unreadable and two no-device cases give FAIL/SKIP with distinct text). * the row block run against a real amdxdna device on hardware prints `PASS npu-power : the NPU is held awake (0000:c6:00.1): power/control=on, runtime_status=active`, with zero clients holding /dev/accel/accel0 - i.e. control=on keeps the array up when nothing is using it, which is the property the fix relies on. * board expectation changes from 25 to 26 checks (docs/run sheets that quote the count are the release lane's to update). Co-authored-by: pi-release <release@1bit.local>
Owner decision 2026-11-05 23:4x ("keep the npu awake on the stick and build c30"). Based on
os/v3-c30-log-cleanup(#1224) so this PR shows only the keepawake change; the two together are what c30 ships.Root cause this replaces
The 5 s autosuspend timer is what puts the array into the state whose first dispatch hangs:
The driver-side fix (what a suspended context loses on resume) stays with pi-kernel-74 and is off c30's critical path.
What ships
etc/udev/rules.d/90-1bit-npu-power.rulesACTION=="add|bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x1022", ATTR{device}=="0x17f0", TEST=="power/control", ATTR{power/control}="on"— copied from what this dev box has run since 2026-10-12 for the same reason (persistent context), not inventedetc/init.d/04onboard-npu-keepawake0x17f0, then says what it found. Runs before10lemonade, so the device is awake before the server opens a contextThe rule's
add|bindcovers both orders (driver binding after udev is up, and the module arriving before it), and theTESTguard makes it inert on a kernel whose device has nopower/control.Measured
Four arms of the fallback plus the rule's syntax:
0000:c6:00.1, control alreadyon)already on (90-1bit-npu-power.rules did it),control=on runtime_status=active, rc 0nothing to keep awake, rc 0control=autoin a fixture treewas 'auto', wrote on (FALLBACK: the udev rule missed …), file nowoncontrol=auto, unwritableudevadm verifysh -n+shellcheck -S warningclean;04onboard-npu-keepawakeis committed 100755 (Puppy'src.servicesonlyruns executables — a non-executable 04 would be silently skipped, the c15 class).
tools/copyright.py --checkrc 0 after*.rulesjoins the#comment list (udev rules are#comments, exactly like the.desktopentry in cycle 16).Not measured here, and not claimed
The device is not in a QEMU guest, so no VM boot can show
control=on. The rows that prove it are the board's:the release check's new keepawake row (
power/control=on,runtime_status=active) and pi-npufeat's W arm(20 s idle between requests, ≥ 20 runs, 0 × 614). What this PR owes is that the rule is valid, the fallback behaves in
all four states, and both are in the image — the final c30 image and its gates are in the hand-off.
Co-authored-by: pi-os3