From dfdec8fb992d4ca95ec1e25f8bfd31e1a17f3c1d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 4 Nov 2026 21:44:53 -0400 Subject: [PATCH] docs: price a MoE decode step on the submission path a decode step uses (#799) The "Routed expert stacks" subsection concluded that MoE-on-NPU lands at 0.97-1.06x the Qwen3-4B-dense bar, from a table whose per-dispatch cost is the arena player's 0.086 ms -- a marginal cost inside ONE command list. An engine decode step submits one ELF per npu_gemm, and that per-invocation cost has since been measured on this box in the engine's own npu/forward route (T1BP-ZYPHRA, 2026-11-04 20:1x-20:27, M=16 device arm, 112 launches/token, 95.0 % of the token waiting on the array): 2.8 / 3.7 / 5.91 ms per invocation from three independent pairs. Priced on the same path for both sides (385 dispatches/token for this MoE, 253 for the 4B dense; same 34 GB/s; same host terms): floor 0.086 ms (arena) -> 11.05 vs 10.45 tok/s = 1.06x floor 2.8 ms (engine) -> 0.88 vs 1.28 = 0.69x floor 3.7 ms (engine) -> 0.67 vs 0.99 = 0.68x floor 5.91 ms (engine) -> 0.43 vs 0.64 = 0.67x Break-even is 0.126 ms per dispatch: below it the MoE beats the bar, above it loses by ~1.45x and stays there, because 385 submissions against 253 (1.52x) outweighs 1.70 GB against 2.26 GB of stream (1.33x). So on the decode loop that exists today the case misses its own bar, and the lever is submission granularity (#799), not the down_proj A shape, not the chunk size (1.3 % effect), and not a gate/up-only MoE ELF. Also corrects the down_proj claim: a chunk-list dispatch IS reachable for down by row placement (A is an M-row int8 buffer, output is M rows per chunk, so expert j's SwiGLU row in row j with NCH=8 puts chunk j's answer in row j) -- a flag, not a new @runtime_sequence signature. The 10-BO dispatch and the WIN=3 per-channel ordering are labelled unverified. Arithmetic on measured rates, not a kernel result: no MoE kernel has run on the device. Docs only, no code, no new files. --- docs/npu.md | 45 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/docs/npu.md b/docs/npu.md index 56da43eb7..57077f8b7 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -436,12 +436,15 @@ a kernel with the **expert ids held fixed** (A row + the router's own top-k + th expert, which is what `tools/moe_expert_chunks.py gate` emits), so the gate measures the kernel and not the model's own sensitivity. `verify` result: see the TMOE-NPU lane's state file. -**A chunk-list dispatch shares ONE A across its chunks: exact for gate and up, impossible for down.** -At batch 1 every selected expert of a layer sees the same activation row, so gate and up are one -dispatch each — but `down_proj`'s input is each expert's own SwiGLU output, so it needs a per-chunk A -or one dispatch per expert. On this model (48 layers; per layer q, gate, k, v, o in the pb16i split, -then gate|up|down) that is **385 dispatches per token with the shared A and 721 without**, and that -gap decides whether the case meets its own bar: +**A chunk-list dispatch shares ONE A across its chunks: exact for gate and up, and reachable for down +by row placement.** At batch 1 every selected expert of a layer sees the same activation row, so gate +and up are one dispatch each. `down_proj`'s input is each expert's own SwiGLU output — but `A` is an +`M`-row int8 buffer and the output is `M` rows per chunk, so with expert `j`'s row in row `j` of `A` +(rows 8-15 zeroed) and `NCH=8`, chunk `j` carries that expert's answer in row `j`: a flag, not a new +signature (the 10-BO dispatch and the `WIN = 3` per-channel ordering are unverified on the array). +On this model (48 layers; per layer q, gate, k, v, o in the pb16i split, then gate|up|down) that is +**385 dispatches per token**, against 721 if down cannot share one — and, two paragraphs down, the +dispatch *rate* rather than the dispatch *shape* is what decides whether the case meets its own bar: | per-dispatch fixed cost | down shares one A | down per expert | Qwen3-4B dense, same arithmetic | |---|---|---|---| @@ -451,10 +454,32 @@ gap decides whether the case meets its own bar: That is at 34 GB/s in-loop over a **1,703,854,080 B/token** stream — routed experts 1,019,215,872 (8 of 128 x 3 projections, pb16 NW=16) + attention 509,607,936 + `lm_head` 175,030,272, with `--pad-n 1024` on gate/up taking it to 1,930,346,496 — and the ratio holds across 34-43.4 GB/s: -**0.97-1.06x the 4B-dense bar when down shares the dispatch, 0.68-0.80x when it cannot.** So the -deciding term is the down_proj A shape, not the expert stream: sizing a MoE case on the expert -bytes alone understates the stream by 40 % (they are 60 % of it) and misses the term that decides -it. +**0.97-1.06x the 4B-dense bar when down shares the dispatch, 0.68-0.80x when it cannot.** So sizing +a MoE case on the expert bytes alone understates the stream by 40 % (they are 60 % of it) and misses +a term that matters — but it is not the term that decides the case. That is the dispatch *rate*: + +**Which submission path runs the loop decides the answer, and the bar pays the same term.** The +0.086 ms in that table is a *marginal* cost inside one command list, measured on the arena player. An engine +decode step submits one ELF per `npu_gemm`, and that per-invocation cost has since been measured on +this box in the engine's own `npu/forward` route: the T1BP-ZYPHRA lane's device arm (M=16 family, +112 launches/token, 95.0 % of the token spent waiting on the array) fits **2.8 / 3.7 / 5.91 ms** per +invocation from three independent pairs (2026-11-04 20:1x-20:27). Pricing this model and the +4B-dense bar on the *same* path — 385 and 253 dispatches/token, same 34 GB/s, same host terms: + +| per-dispatch fixed cost | this MoE | Qwen3-4B dense, same path | ratio | +|---|---|---|---| +| 0.086 ms (arena player) | **11.05 tok/s** | 10.45 | **1.06x** | +| 2.8 ms (engine, driver-delta fit) | 0.88 | 1.28 | 0.69x | +| 3.7 ms (engine, two-point fit) | 0.67 | 0.99 | 0.68x | +| 5.91 ms (engine, launch-cost fit) | 0.43 | 0.64 | 0.67x | + +Setting the two expressions equal puts the **break-even at 0.126 ms per dispatch**: below it the MoE +meets and beats the bar, above it it loses by ~1.45x and stays there, because 385 submissions against +253 (1.52x) outweighs 1.70 GB against 2.26 GB of stream (1.33x). The engine's measured path is 22-47x +over that threshold. So on the decode loop that exists today this case **misses** its own bar, and the +lever is submission granularity — one command list per token instead of one per `npu_gemm` (engine#799) +— not the chunk size (1.18-2.13 MB against the arena's 50 MB is a 1.3 % effect) and not a MoE-shaped +ELF built for gate/up alone. The fixed cost above is measured, not assumed. Two runs of the same 50,135,040-B weight chunk in the arena's chunk-list design — 1.051 ms for one chunk, 2.016 ms for two — fit a marginal rate of