[bong-water-water-bong]

Refs #754 (a plan note, not a fix — #754 is closed; this records what the 2026 survey changes about why and what next).

Read for this: ~/amd-oss/npu-re/NPU-PAPERS-AMD.md #1-#4 and NPU-PAPERS-GENERAL.md #1, #13, #19, #24. The paper figures below are the
papers' own claims, cited so they can be checked; the numbers they are compared against are already on this page (and were measured by
this lane).

Three things that change the plan

1. The NPU prefill GEMM's target is published, and it is 6-8x above ours. TileFuse (arXiv 2606.11357): XDNA2 W4A16 prefill at
~12 TOPS (Krackan) / ~9 TOPS (Strix Point), Llama3-8B 2048 tokens in 4.13 s on the NPU against 8.25 s on the iGPU, 501.9 tok/s
at 1K context vs FastFlowLM's 345.9. Our gemm_q4_prefill is 1.2-1.5 TFLOP/s. So the bar this page derives — "an NPU prefill GEMM
~4x faster" — sits inside a published result on the same silicon class; the gap is a kernel project with a known technique
(interleaved column-major pre-tiling past 8K, fused unpack+dequant+matmul, 128 B-padded payload), not a property of the part. Their
decode agrees with our route: GEMV at 181-206 GOPS, slower than the iGPU, which is why decode stays on HRX.

2. Attention is a second splittable share. Every ceiling on this page assumes a split touches matmuls only, because attention is
57.1 % of our post-fix pp16384 device time. FlashAttention-on-XDNA (arXiv 2609.21264) reports 3.62 TFLOP/s at 32K with the
scores kept in compute-tile L1 (ridge: MemTile 33.7 vs compute 25.2 FLOP/B), and 5.3-7.2x the iGPU's energy efficiency from 2K
tokens up. If that kernel shape lands here, the split's end-to-end factor has two shares to spend and the 1.035x on this page becomes
the matmul-only answer.

3. Two house rules and one architecture target. The ANE-prefill hybrid gets 11.3x by dispatching 32 tokens per call against
a flat ~42 ms sequential dispatch cost → no NPU path should dispatch fewer than 32 rows (which also retires the one-token handoff
and M=16 twins as batch shapes). Pipeline Shards (arXiv 2608.19147) is the shape the shared-buffer work wants: a static NPU graph with
KV as explicit graph I/O in a host-managed bounded ring — the object a dma-buf would be made of, and a better long-term shape than
this route's per-request state file. HeteroInfer (SOSP'25) supplies the split shape that matches what was measured here: NPU takes the
static aligned part, GPU the ragged tail (our optimum was already 5-9 of 64 row-groups), with a ~400 us GPU-NPU sync as a
first-class cost — the bar any per-layer handoff would have to clear.

Licence check (the owner asked for it before any code reading)

The one paper with code is TileFuse, whose mlir-aie fork's LICENSE reads Apache License v2.0 with LLVM Exceptions (GitHub's API
reports NOASSERTION/"Other", so the file is the authority). Permissive: reading it for understanding is fine, and any port keeps the
notice and lands in the private kernel repositories, not in this tree. No code was read for this PR — the papers were enough.

Scope

Docs only, no behaviour, no device work. It is a dated section rather than a verdict change: the measurements on this page stand, and
what is new is the plan around them — attention as a lever, a published prefill target, and the 32-row floor.

[bong-water-water-bong]

VERDICT: READY at 31827c272ead687bedfc672bb1d6869fe4272351.

A plan note, so the check is that it distinguishes the papers' claims from this page's measurements and does not quietly contradict what is already there. Both hold in the added section: the paper figures are cited as the papers' targets, and the three consequences are derived from rows the page already carries rather than from new numbers.

The useful part is the reframing of attention as a second share: the page's own 2026 capture moved attention to 57.1 % of busy time (engine#1369), which is why a prefill split's ceiling fell from 1.164x to 1.085x — so a survey entry that treats attention as a fixed 47.7 % would have been the pre-#1369 reading. Recording "attention as a second share, and a published target to compare against" is the right way to keep #754's closed verdict usable as a live reference.

[bong-water-water-bong]

VERDICT: READY at 31827c272ead687bedfc672bb1d6869fe4272351

A "what the survey changes" section on a page whose whole method is measured numbers, and it keeps the two apart properly: the papers' claims are cited so a reader can check them (arXiv id, hardware, model, context), and the page's own figures stay the ones already on it.

What I could check holds: the 8.61 ms per 256-row tile is the same NPU tile figure my earlier reviews of this page used (#1306/#1369 derived the 1.03-1.08x ceilings and the 2.15 ms "4x bar" from it), and "our gemm_q4_prefill is 1.2-1.5 TFLOP/s" is the order that tile implies for a 4B-class up_gate shape — consistent with the numbers already on the page rather than a new measurement smuggled in.

What I did not check, and it is the substance: the cited results (TileFuse arXiv 2606.11357 — ~12 TOPS on Krackan, ~9 on Strix Point, 4.13 s vs 8.25 s for Llama3-8B at 2048 tokens, 501.9 vs 345.9 tok/s at 1K). I did not fetch the paper; the section is explicit that these are the papers' claims, so a reader can weigh them as such, and that is the right way to bring an external result into a page like this — as a target with a citation, not as a number the page owns.

The framing is also the useful one for the next lane: if the 4x bar "sits inside a published result on the same silicon class", then the gap is a kernel project with named techniques (pre-tiling past 8K, fused unpack+dequant+matmul, padded tile payload), not a property of the part. That is a different instruction to the next reader than "the split does not pay", and both are true at once — the split does not pay today, and the bar is a knowable distance away.

One suggestion: the section says "Three readings … change the plan", and I can see 1 (the prefill target); if readings 2 and 3 are the ones that follow, a one-line summary at the top ("1 changes the bar, 2 changes X, 3 changes Y") would let a reader who only wants the plan skip the paper discussion.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 754 - Partially compliant Compliant requirements: Update documentation to reflect the new performance targets and hybrid routing strategy for prefill operations Document the 6-8x performance improvement target for NPU prefill GEMM Document that attention can be a second splittable share Document house rules from hybrid designs (dispatch size, buffer sharing, sync costs) Reference academic papers and their findings to support the claims Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Documentation Clarity The documentation updates introduce several new claims and references to academic papers, but it's unclear if all the performance figures and claims are fully validated against the actual hardware measurements. Specifically, the claimed 6-8x performance improvement for NPU prefill GEMM and the 3.62 TFLOP/s for attention need to be confirmed with actual measurements on the target hardware. **1. The NPU prefill GEMM's target is published, and it is 6-8x above ours.** TileFuse (arXiv 2606.11357) reports XDNA2 W4A16 prefill at **~12 TOPS (Krackan) / ~9 TOPS (Strix Point)**, Llama3-8B at 2048 tokens in **4.13 s on the NPU against 8.25 s on the iGPU**, and **501.9 tok/s** at 1K context against FastFlowLM's 345.9. Our `gemm_q4_prefill` is **1.2-1.5 TFLOP/s** (the 8.61 ms per 256-row tile above). So the bar this page derives for a split - "an NPU prefill GEMM ~4x faster" - sits *inside* a published result on the same silicon class: the gap is a kernel project with a known technique (interleaved column-major pre-tiling past 8K, fused unpack+dequant+matmul, 128 B-padded tile payload), not a property of the part. Their decode is the counterpoint and it agrees with our route: **GEMV at 181-206 GOPS, slower than the iGPU**, which is why decode stays on HRX here. Paper References The documentation includes references to academic papers, but it's not clear if the specific performance numbers cited from these papers (e.g., TileFuse's 12 TOPS, FlashAttention's 3.62 TFLOP/s) are directly measurable on the current hardware setup or if they represent theoretical performance that may not translate to real-world scenarios. **1. The NPU prefill GEMM's target is published, and it is 6-8x above ours.** TileFuse (arXiv 2606.11357) reports XDNA2 W4A16 prefill at **~12 TOPS (Krackan) / ~9 TOPS (Strix Point)**, Llama3-8B at 2048 tokens in **4.13 s on the NPU against 8.25 s on the iGPU**, and **501.9 tok/s** at 1K context against FastFlowLM's 345.9. Our `gemm_q4_prefill` is **1.2-1.5 TFLOP/s** (the 8.61 ms per 256-row tile above). So the bar this page derives for a split - "an NPU prefill GEMM ~4x faster" - sits *inside* a published result on the same silicon class: the gap is a kernel project with a known technique (interleaved column-major pre-tiling past 8K, fused unpack+dequant+matmul, 128 B-padded tile payload), not a property of the part. Their decode is the counterpoint and it agrees with our route: **GEMV at 181-206 GOPS, slower than the iGPU**, which is why decode stays on HRX here. **2. Attention is a second splittable share, not untouchable overhead.** This page's ceilings assume a split can only touch the matmuls, because attention is **57.1 %** of the post-fix pp16384 device time on the 4B. FlashAttention-on-XDNA (arXiv 2609.21264) reports **3.62 TFLOP/s** for a fully fused XDNA2 attention at 32K - scores kept in compute-tile L1, reduced over the cascade - against 1.81 for a streamed design and 0.455 layer-by-layer, at **5.3-7.2x the iGPU's energy efficiency from 2K tokens up** (ridge points: MemTile 33.7 FLOP/B, compute tile 25.2 FLOP/B, so QK^T belongs in L1 and not the MemTile). If a kernel of that shape lands here, the split's end-to-end factor has *two* shares to spend and must be re-derived; the 1.035x end-to-end above is the matmul-only answer.