Repository navigation
npu plan P3: t2 on-core expansion kernel for the sealed M=16 family (XDNA) - halve the weight stream #802
Description
New private ELF (npu-kernels) that streams t2 and expands to int4 nibbles in local memory once per K slice, feeding the existing mmul path; target wiring. Gate: NPU_M16_CHECK byte-exact; window11 identical; decode tok/s before/after (12.9 -> ~6.5 GB/token; plain 2.1 -> ~5 tok/s expected). Lane: pi-bonsai after the 2026-11-02 08:00 MTP window. Depends on P2.
Part of #799 (the NPU plan layer). Gate and lane per the plan table in #799. Not a c20 blocker (bucket C: sequenced after the release freeze unless marked additive).
Activity
bong-water-water-bong commentedon Nov 2, 2026
Window data for P3 (t2 on-core expansion), measured 05:44-06:15 AST on the sealed M=16 pb16i family, pb16i-exact head
images (source codes + int16 magnitudes + host additive table), one
npu-holdticket per matrix,amdxdnacounter 0 afterevery ticket.
What the sealed family actually achieves for the head's shapes (M=16 rows, 5 iters, median of 4):
Two things for P3 from this:
47-49.75 GB/s the arena measures with pb16i, so the head would cost ~5 ms/token on the array (sum of the eight),
against 50 ms per draft for the same head on the host. That is the number P3 has to keep, and it is why halving
the wire (2.125 bpw vs 4.5) is worth doing on top: at 2.125 bpw the head's own bytes fall to ~2.3 MB/head-pass and
the expansion (530 ns per tile-class decode, ~103k tiles per core per token worst case) hides under the DMA of the
next slice, as the cost note in the state file sets out.
down(K=17408) mismatches every element ando/qkill the host runner before the dump (counter stayed 0 — no firmware event). The four that pass do sobyte-exactly, so the images and the tile order are right; the failures are BO-convention/geometry on those three
designs. Next step is to bind them the way
q4nx_forward.cppdoes (per-designpackB_blockedBO) and re-measure,before P3 builds a new kernel on the same conventions.
Head correctness is separately established, so this is a measurement gap and not a correctness doubt: the drafter run on
the same artifacts hit 69.4 % first-draft acceptance and 3.19 tokens/pass (see #805 comment).
bong-water-water-bong commentedon Nov 6, 2026
Implementation landed; the phase's measured row is what remains — parked on that.
On
origin/main(19486dcc): PRs #826/#830/#837/#843 merged the t2s stream + container launch path + v2 artifact family + sealed ELF (npu/q4nx_t2s.h,tools/q4nx_t2s.py,docs/npu.md:254-333); #837's body records the measured 12.93 → 6.11 GB/token stream, 40.2 GB/s, 1.76-1.96x, seeds byte-exact, and the synthetic gatetests/q4nx_t2s_test.pypasses.Remains: the issue's own gate — the model-level row (
NPU_M16_CHECK+ window11 identical + decode tok/s) is not recorded in the docs/leaderboard (there are no t2s rows there), and the issue still carries its old window note. That is a device/leaderboard measurement, not a code gap.Unpark: post the measured model-level t2s row (decode tok/s + window11 identity) to
docs/npu-leaderboard.mdand the issue; then this phase's acceptance is complete.bong-water-water-bong commentedon Nov 6, 2026
The phase's remaining item — the measured model-level row — is recorded: engine #1276 (
4deae1bb),Fixes #802.Same container file, same binary, two weight paths, adjacent windows, 7 steady-state tokens per arm from the serve's own
[ph] tokline (not request walls, which pay the per-serve image fill once):1.31x per token, 1.42x on the device wait, on 2.12x fewer bytes (431 images, 14.44 → 6.82 GB, checked on disk). Model keeps 1.31x of the kernel's 1.76x because the host half (66 ms) and the lm head (13.5 ms) do not shrink with the wire.
Acceptance: byte-exactness, not a text diff — every decode role matched the pb16i path byte for byte at layer 0 (6/6 roles) and layer 3 (8/8 roles) on three seeds, and greedy text is equal to the pb16i arm's on the shared prompt (
' Paris.\nThe capital of Germany is'). Threenpu-holdtickets on :18099 (2026-11-02 19:47/19:48/19:49),amdxdnacounter 0 in each, owner serve yielded and restarted by the keeper.Rows are in
bench/npu-leaderboard/bonsai2-27b-{t2s2,pb16i}-20261102.json,docs/npu-leaderboard.mdregenerated (--check: 21 records valid), and the t2s chunk ofdocs/npu.mdnow carries the model-level subsection.One item this does not claim: a same-day (2026-11-06) re-take of the three-prompt full-model text window. Both attempts today died on
[npu] waiting for the exclusive device lock— an unrelated lane's orphaned host-GEMM serve (pid 2231177,~/lb/1035-row, up since 11:28) holds/tmp/1bit-npu-device.lockexclusively while touching no array, and pi-moe-npu's ticket was stuck behind it too. Reported to its owner and to the orchestrator; the row above is the 2026-11-02 pair with its provenance in each record'ssource. The interesting bit for the lock's own design: aNPU_INFER_HOST_GEMM=1serve — a path that never opens the array — still takes the exclusive device lock, so a no-device probe can block every device lane.npu: record the t2s stream's model-level decode row, 313 ms/token vs …
bong-water-water-bong commentedon Nov 7, 2026
The acceptance step #1276's records name as open — the three-prompt full-model greedy-text window — is now measured. One
npu-holdticket (p3-accept-window, 06:42:38-06:48:01,:18099,amdxdnaerrors 0 for the window, owner serve yielded and restarted by the keeper,:18097never touched), both arms from the same binary~/wt/bonsai-speed-build2/1bit: pb16i (control) then t2s2 (2-bit), 3 prompts × 12 greedy tokens each.Acceptance: text agrees from the first token. PASS.
The history of the printing press begins with' the invention of movable type in China during the 11'' the invention of movable type in China during the 11'Write one sentence about the sea.''''window11quirk for this prompt, so it carries no weightIn one word, the capital of France is' Paris.'' Paris.'So the two prompts that produce text agree character for character, which is the claim the records were missing.
This window cannot also serve as the ratio, and I am not quoting it as one. It ran contended and with the arms sequential, which is the failure mode this lane's §6 records:
load15 29.58 / load5 45.71 / load1 45.58at 06:48 (it was 4.66 when the ticket was queued at 06:36). Other lanes' CPU work was the load —/tmp/bench_batchat 1024 % CPU, a 4-vCPU qemu guest, a python router bench.gemm host 111 vs 57,ql 74 vs 45, medians of 32 steady tokens), and the wall medians are 561 vs 528 ms/token with spreads of 485-711 and 419-782.So: the acceptance item is closed (text identity), the 1.31x row is untouched and remains a quiet-window figure (that pair was taken at load 4-9, one minute apart), and a contended re-take is on record here as inconclusive by construction rather than as a contradiction. If anyone wants a second ratio window it needs an idle box and interleaved arms, which is a scheduling question, not a measurement one.
For completeness on why this took a day: the 2026-11-06 attempts (15:10, 15:16) both died on
[npu] waiting for the exclusive device lock. The holder was an unrelated lane's orphanedNPU_INFER_HOST_GEMM=1serve (pid 2231177,~/lb/1035-row) — a path that provably never opens the array, yet it takesLOCK_EXon/tmp/1bit-npu-device.lockand so blocked every device lane, pi-moe-npu's ticket included. Reported three ways (pi-prs'sTPRS-STATE.md, my note inTMOSAIC-STATE.md, my state file), never touched by me; it exited on its own and this window ran at the first opportunity afterwards. If that lock is meant to protect the array, aNPU_INFER_HOST_GEMM=1serve should skip it.