[bong-water-water-bong]

Docs only (docs/npu.md, +35/-10). No code, no new files, no device use. Supersedes the conclusion of the merged #896 subsection; does not supersede its measurements.

Root cause

The "Routed expert stacks" subsection concluded: "MoE-on-NPU lands at 0.97-1.06x the Qwen3-4B-dense bar when down shares the dispatch, 0.68-0.80x when it cannot — so the deciding term is the down_proj A shape."

That table's per-dispatch cost is 0.086 ms, and 0.086 ms is a marginal cost measured inside one arena command list (two runs of the same 50 MB weight chunk: 1.051 ms / 2.016 ms). An engine decode step does not run that way — it submits one ELF per npu_gemm. The table therefore priced 385 submissions at a cost that only applies to chunk N of a dispatch, and then compared that against a 4B-dense bar priced the same way, which hid the error.

That per-invocation cost has since been measured on this box, in the engine's own route, by another lane: T1BP-ZYPHRA, 2026-11-04 20:1x-20:27, npu/forward device arm, M=16 family, 112 launches/token, 95.0 % of a token spent waiting on the array (host CPU 5.0 %) — three independent pairs fitting 2.8 / 3.7 / 5.91 ms per invocation.

Measured before/after

Both sides priced on the same path (this MoE 385 dispatches/token, Qwen3-4B dense 253; 34 GB/s in-loop; same host terms):

per-dispatch fixed cost	this MoE	Qwen3-4B dense, same path	ratio
0.086 ms (arena player)	11.05 tok/s	10.45	1.06x
2.8 ms (engine, driver-delta fit)	0.88	1.28	0.69x
3.7 ms (engine, two-point fit)	0.67	0.99	0.68x
5.91 ms (engine, launch-cost fit)	0.43	0.64	0.67x

Break-even: 0.126 ms per dispatch. Below it the MoE meets and beats the bar; above it the MoE loses by ~1.45x and stays there, because 385 submissions against 253 (1.52x) outweighs 1.70 GB against 2.26 GB of stream (1.33x). The engine's measured path is 22-47x over that threshold.

So on the decode loop that exists today the case misses its own bar, and the lever is submission granularity — one command list per token instead of one per npu_gemm (#799) — not the down_proj A shape, not the chunk size (1.18-2.13 MB against the arena's 50 MB is a 1.3 % effect), and not a gate/up-only MoE ELF.

Second correction in the same edit

The subsection said a chunk-list dispatch is "impossible for down". It is reachable by row placement: A is an M-row int8 buffer and the output is M rows per chunk, so with expert j's SwiGLU row in row j of A (rows 8-15 zeroed) and NCH=8, chunk j carries that expert's answer in row j — a flag, not a new @runtime_sequence signature. The 10-BO dispatch (A + 8 W + C, where only NCH=2 has run) and the WIN = 3 per-channel ordering are explicitly labelled unverified on the array.

Not claimed

Arithmetic on measured rates, not a kernel result — no MoE kernel has ever run on the device from this lane. The Qwen3-4B-dense geometry (2.26 GB/token, 253 launches) is arithmetic, not a measurement: no 4B container exists on this box. The park decision is unchanged (HRX decodes this model at 83-85 tok/s measured, above every rung here).

No closes: this is a correction to merged documentation, not a tracked issue. Related: #799 (the submission-granularity change this points at), #896 (the subsection being corrected).

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 1a39209)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 896 - Partially compliant Compliant requirements: Documentation updated to reflect the correct information about MoE decode step dispatch The deciding term is correctly identified as down_proj A shape Pricing is now done on the same submission path Non-compliant requirements: None Requires further human verification: None 799 - Partially compliant Compliant requirements: The PR focuses on documentation updates for MoE decode step pricing, which is a subset of the overall plan layer implementation The documentation correctly reflects the current state of MoE handling and dispatch costs Non-compliant requirements: The PR does not implement the full plan layer architecture (which includes code changes and new files) The PR does not cover the NpuTarget interface implementation or t2 chunk kind implementation Requires further human verification: None

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Documentation Accuracy The documentation correctly updates the MoE decode step pricing information, but it's important to ensure that all the numerical claims and performance data are accurately measured and reflect the actual hardware behavior. Specifically, the claim that the engine's measured path is 22-47x over the threshold should be verified against actual measurements from the hardware. Setting the two expressions equal puts the **break-even at 0.126 ms per dispatch**: below it the MoE meets and beats the bar, above it it loses by ~1.45x and stays there, because 385 submissions against 253 (1.52x) outweighs 1.70 GB against 2.26 GB of stream (1.33x). The engine's measured path is 22-47x over that threshold. So on the decode loop that exists today this case **misses** its own bar, and the lever is submission granularity — one command list per token instead of one per `npu_gemm` (engine#799) — not the chunk size (1.18-2.13 MB against the arena's 50 MB is a 1.3 % effect) and not a MoE-shaped ELF built for gate/up alone.

[bong-water-water-bong]

One correction to the bot's compliance table, for a human reading this: it credits the PR with "the deciding term for MoE cases is correctly identified as down_proj A shape" — that is the sentence this PR removes. The finding is the inverse: down_proj can share one chunk-list dispatch (row placement, NCH=8, expert j in row j of the M-row A), and the term that decides the case is submissions per token — break-even 0.126 ms per dispatch, with the engine's measured 2.8/3.7/5.91 ms per invocation putting this model at 0.67-0.69x the 4B-dense bar instead of 1.06x. Nothing else in the table needs action; no inline threads exist to resolve.

(Corrected repost: the first version of this comment lost the down_proj token to shell command substitution. This one is written from a file.)

[bong-water-water-bong]

VERDICT: READY at dfdec8fb — docs-only correction, and the arithmetic reproduces from the body's own anchors.

Recomputed the whole table from the two 0.086 ms cells (MoE 11.05 tok/s at 385 dispatches, 4B 10.45 at 253), which fixes both bases: MoE 57.4 ms, 4B-dense 73.9 ms per token. Those decompose as stream 50.0 / 66.5 ms at 34 GB/s (1.70 / 2.26 GB) plus an identical 7.4 ms host term, so "same host terms" is not a hand-wave. Then:

cost 0.086 ms -> MoE 11.05  4B 10.45  ratio 1.06     (table: 1.06x)
cost 2.800 ms -> MoE  0.88  4B  1.28  ratio 0.69     (table: 0.69x)
cost 3.700 ms -> MoE  0.67  4B  0.99  ratio 0.68     (table: 0.68x)
cost 5.910 ms -> MoE  0.43  4B  0.64  ratio 0.67     (table: 0.67x)
break-even (bases equal) = 0.1254 ms                 (stated: 0.126 ms)

Every cell matches. 48 x 8 + lm_head = 385 matches the dispatch count, 1,019,215,872 / 1,703,854,080 = 59.8 % matches "they are 60 % of it", 1.70/2.26 = 1.33x and 385/253 = 1.52x match, and 2.8/0.126 to 5.91/0.126 = 22-47x matches the stated over-threshold range.

The inherited term is real and correctly attributed: the T1BP-ZYPHRA state file carries 112 launches/token, 95 % waiting, and the 2.8 / 3.7 / 5.91 ms fits from three independent pairs (its line 1244 is this exact repricing), and its 5.91 ms fixed + 0.296 ms/row fit is the launch-cost arm. No number here is invented on this PR's side.

The old conclusion is gone rather than softened — deciding term, impossible for down and 721 without no longer appear in docs/npu.md, and #896's own measurement table is left intact. The two unverified pieces are labelled as such in the text (the 10-BO dispatch and the WIN = 3 ordering), which is the right way to publish a reachable-but-unrun design.

One precision nit, not worth a push on its own: "loses by ~1.45x and stays there" — 1.45x is the value at the 2.8 ms fit; the ratio asymptotes to 385/253 = 1.52x as the per-dispatch cost grows, so "1.45-1.52x" would be the tighter phrasing. It changes no conclusion.

CI on this head: build + all six CodeQL/Analyze jobs and pr_agent SUCCESS, hw_gate SKIPPED.

[github-actions]

Persistent review updated to latest commit 1a39209