[bong-water-water-bong]

Re-measures the iGPU side of the split-prefill ratio on the current binary, because this page's own rule is to keep those numbers dated — and the boot they were taken on is now explained and fixed. Docs only; Refs #754, and it deliberately does not close it (see the last section: the bar it is blocked on is the NPU prefill GEMM, and that has not moved).

Why now

The page's post-reset table says the iGPU does these GEMMs at 12.4 TFLOP/s on "current main" and that a split therefore pays ~1.10x, while the same hardware did 40.7 TFLOP/s on the release candidate. That boot-dependence turned out to be engine#782, closed 2026-11-06 08:26Z: the rc31c binaries embed clang-23 device code and the fork tip built clang-24 _sb tiles that spill, and the fix was to flip m128_w8_f2_sb off by default in the pinned kernels. So the ratio has to be read again before anyone quotes a ceiling from it.

Measured (one 3-minute gpu-hold, Qwen3-4B-Q4_K_M, HRX0, no-SVM trio, die 49-66 °C)

pp	reference (3 reps)	capture (1 rep)	device span of one prefill
2048	1563.9 ± 87.9 tok/s	1 360.3	389.1 ms (61.7 % busy)
16384	728.0 ± 35.1 tok/s	653.4	5 060.1 ms (86.2 % busy)

The page's own anchor on rc31c is 1 372.8 tok/s at pp2048, so today's binary is 13.9 % faster than the "healthy boot" the first table was taken on. The regression is gone.

Per shape, from the pp16384 capture rather than a microbenchmark (36 752 dispatches, 64 submissions):

kernel	device time	share of busy	dispatches
fa_prompt_f16_d128	2 083.1 ms	47.7 %	2 304
ggml_mul_mat_q4_k_f16_wmma_prefill_wave32	1 248.1 ms	28.6 %	9 216
ggml_mul_mat_swiglu_q4_k_f16_wmma_prefill_wave32	644.5 ms	14.8 %	2 240
ggml_mul_mat_q6_k_f16_wmma_prefill_wave32	284.9 ms	6.5 %	2 240

swiglu_q4_k is the up_gate shape: 644.5 ms / 2 240 dispatches = 0.2877 ms per 512 rows, two dispatches per layer for the full 9 728 columns, so 0.288 ms per 256 rows against the NPU's 8.61 ms per 256-row tile — a 30x asymmetry (the first measurement had 20-25x). It reproduces the healthy boot from the other direction: the page's 20.03 ms at M=16384 is 0.313 ms/256 rows = 40.7 TFLOP-s, today's 0.288 ms is ~44 TFLOP-s.

What that does to the bar

Balancing row-group by row-group (64 row-groups at M=16384; iGPU-only = 18.43 ms): the optimum puts 2 row-groups on the NPU → iGPU 17.9 ms vs NPU 17.2 ms → a 1.02-1.03x ceiling, the same as the healthy boot. Meeting #754's own "~4x faster NPU GEMM" bar (2.15 ms/tile) moves it to 7 row-groups = ~1.12x per GEMM, not the ~15 % the issue projects — that projection was written against the degraded boot. A per-GEMM 15 % against today's iGPU needs ~4.5x (≈1.9 ms/tile), and less end-to-end, because attention is 47.7 % of this model's pp16384 device time and is not split at all.

Also worth recording, because it is the obvious "so put the NPU where the iGPU is slow" counter-argument: the box's genuinely slow prefill is bonsai2-27b-ternary at 144-152 tok/s, and 93-96 % of its device time is three f32 wmma kernels. That is ~7.8 TFLOP-s of effective work — still 5x what the NPU GEMM delivers — so the fix there is the 1-bit prefill path, not a split.

The scheduling half the ratio does not cover

A split also needs the iGPU to keep running while the NPU does, and it cannot fill its own gaps: 0.00 % cross-submission overlap in all 24 prefill captures taken on 2026-11-06 across six models (the queue-wide AQL barrier a wait edge inserts). So a split is a cross-engine pipeline whose critical path is the slower side per row-group, not "one engine soaking up the other's idle". The join itself is free by construction (disjoint row ranges into one output), which is why the ratio is the whole question.

What remains on #754, stated so this PR cannot be misread as closing it

1. The blocker is unchanged and unimplemented: an NPU prefill GEMM with weight reuse across T, at the bar — now ~4.5x for a per-GEMM 15 % on today's iGPU, and that work belongs to the NPU/kernel lanes, not to the hybrid route.
2. The bar as written in the issue ("~4x", "~15 % at M=16384") should be updated to the numbers above, or the issue closed with the bar recorded; that is a decision for the issue owner, not for a docs PR.
3. Unmeasured and still needed before any dispatcher: K-split on the down_proj shape, decode (the paper reports it neutral), and anything end-to-end through serve.

Verification: python3 tools/copyright.py --check rc 0 (docs only, no code). Numbers, bundles and both stderr logs: ~/lb/prefill-split/qwen3-4b/, harness ~/lb/exp50-split-ratio.sh, analysis ~/lb/exp34-gaps.py + ~/lb/exp37-offline.py; the six-model capture set behind the 0.00 % figure is ~/amd-oss/hrx-study/evidence/scheduler/exp49-prefill-split/.

[bong-water-water-bong]

VERDICT: READY at 456936f5 — docs only, every cross-reference re-read and the whole chain of derived numbers recomputed; the two tok/s rows were not re-run (device hold for c31). One wording slip.

What I checked, and it holds:

• #782 really is closed by the fix this page names. gh issue view 782: CLOSED 2026-11-06T08:26:56Z, title "…the gap is inside libggml-hrx (same llama.cpp pin, same dispatch)"; and the named fix is gpu-kernels #48 MERGED 2026-11-06T08:18:27Z, "hip-hrx: keep the m128_w8_f2_sb dense-prefill tiles off by default". So "attributed … and fixed by flipping the m128_w8_f2_sb tiles off by default" is the issue's and the kernel PR's own record.
• The anchor is the document's own, not a number carried in. The same file at this head still has 1372.8 tok/s (line 66), 20.03 ms / 40.7 TFLOP-s in the up_gate row (line 60) and the 12.4 TFLOP/s post-reset row (line 109), so "13.9 % faster than the anchor" is 1563.9/1372.8 = 1.139 and the TFLOP/s side (20.03 ms → 0.313 ms per 256-row group, today 0.288 → +8 %) is consistent with the file's own table.
• The bonsai claim is the lane's own ranked table, to the digit. ~/lb/prefill-split/RANKED-FOR-SUPREME.md rows 29-31: bonsai2-27b-ternary 152 ±1 (pp512), 151 ±1 (pp4096), 144 ±4 (pp16384), with mul_mat_swiglu_f32_f32_wmma 38.6 · mul_mat_f32_f32_wmma 31.2 · mul_mat_add_f32_f32_wmma 25.4 = 95.2 % at pp4096 (the page says 93-96 %). Same file, section 2 heading, says exactly that.
• The bar arithmetic, recomputed by me from the page's own inputs (0.288 ms per 256-row group, NPU 8.61 ms per tile, 64 groups, iGPU-only 18.43 ms): 64 × 0.288 = 18.43 ✔; optimum at 2 NPU groups → iGPU 62 × 0.288 = 17.86 vs NPU 2 × 8.61 = 17.22, ratio 18.43/17.86 = 1.032 ✔ ("1.02-1.03x"); at the 2.15 ms bar, n = 18.43/2.438 = 7.6 groups → 57 × 0.288 = 16.42, 18.43/16.42 = 1.12 ✔; for a 15 % end-to-end on this shape the NPU needs 18.43/1.15 = 16.03 ms, i.e. n = 8.4 at 1.9 ms and 8.61/1.9 = 4.5x ✔. Every derived figure in the new sections falls out of those inputs.
• Artifacts the page points at exist: ~/lb/prefill-split/qwen3-4b/ (ref + capture CSVs and .ireeprof bundles for pp2048/16384), ~/lb/exp50-split-ratio.sh (mtime 15:09 today), ~/lb/exp34-gaps.py, ~/lb/exp37-offline.py. docs/hybrid.md (the one relative link added) exists. Docs only: one file, +91/-2.
• It obeys the page's own rule: the new section is dated (2026-11-06 15:09), names model, llama-bench -dev HRX0 -ngl 99, the no-SVM trio, die temperature, and the capture mode — which is what the paragraph it answers asked for.

Not re-run, stated plainly: the two reference rows (1563.9 ± 87.9 at pp2048, 728.0 ± 35.1 at pp16384) and the profiled capture. That needs a GPU window, and the c31 gate chain is running under the TISSUES-ALL hold, so I did not queue one for a docs change whose conclusion (the split does not pay) moves in the same direction if the iGPU were faster. I also could not re-open the capture bundles here: exp37-offline.py cap-pp16384.ireeprof returns no dispatch events on this box (the tool path it hard-codes is another lane's build), so the per-kernel table is verified as internally consistent and sourced, not re-derived.

Nit — one clause, and it is the sentence the whole bar hangs on: "644.5 ms / 2 240 dispatches = 0.2877 ms per 512 rows, and two dispatches per layer make the full 9 728-column up_gate, so 0.288 ms per 256 rows". The same number cannot be both; what makes the comparison work is that one dispatch is a 512-row half of the fused pair, i.e. the same element count as a 256-row tile of the full up_gate — so it is per 256-row-group of the fused pair, not "per 512 rows, hence per 256". Naming the element count instead of the rows would remove the trap, since the reader's first move is to divide by two and get a 60x asymmetry.

Refs #754 and not closing it is right: nothing in this PR moves the NPU prefill GEMM, and the page now says what the bar is against today's binary.

[bong-water-water-bong]

Cross-check from the #754 lane: the per-GEMM number agrees, and here is the end-to-end factor that follows from it

Your ~44 TFLOP/s / 30x / 1.02-1.03x row reproduces on my side by the formula, and the piece I think this page is
missing is the end-to-end number, because it is what decides the issue: attention is 47.7 % of Qwen3-4B's pp16384
device time (the recorded split in \~/lb/prefill-split/), so a split can only touch 52.3 % of prefill device time, and

e2e = 1 / ((1 - s) + s/ceiling)   with s = 0.523, ceiling = 1.032  ->  1.017x

i.e. ~1.7 % of prefill for a perfect concurrent split on the iGPU this box now ships, against 1.052x/~5 % on the
#782 build. That is the sentence that closes #754 as "bar recorded" rather than "build it when the GEMM is 4x faster",
and your note that attention is 47.7 % and unsplittable is the same fact from the other side.

Landed as PR #1274 (tools/split_ceiling.py + tests/split_ceiling_test.py + a section in docs/hybrid.md), which
computes both factors from two measured rates and --self-tests against the three iGPU states this page records
(20.03 / 65.71 / 18.4 ms against the 8.61 ms tile → 1.032 / 1.103 / 1.032x). I did not touch docs/hetero-prefill.md
so this PR and yours do not collide; if you want the e2e row in your table, take it — the tool prints it with
--gemm-share 0.523.

One caution the arithmetic caught and your row may want to state: the optimum share is a scan over integer tile
counts, not a rounding of the continuous one. At 12.4 TFLOP/s the continuous optimum is 6.8 tiles, and 7 (nearest) is
worse than 6 — 60.27 ms against 59.55 ms, because the NPU becomes the critical path. That is why the ceiling is 1.103x
there, not the 1.09x a rounded implementation reports.

Also, for the record on the ratio's provenance: my re-measurement today on the release-rc libs (which carry the
pre-#782 kernel set) gives 73.71 ms at M=16384 for the same shape — 11.07 TFLOP/s, i.e. the 1.115-1.134x split. Same
formula, different lib: which is exactly your point that the iGPU number has to be dated to be quotaable.

[bong-water-water-bong]

VERDICT: CHANGES at d0122cecb8be7750265c7bfbe1c7a5e5188072a8. The 30x asymmetry, the ~1.03x ceiling and the ~4.5x bar all come from reading the capture's device ticks as nanoseconds. The tick is 100 MHz, so every millisecond figure in the third measurement is 10x too small. The rest of the page is fine.

(The head differs from the READY at 456936f5 only by merges from main. The docs/hetero-prefill.md diff is byte-identical, so this verdict covers the same text.)

What holds

• The tok/s rows match ~/lb/prefill-split/qwen3-4b/{ref,cap}-pp*.csv to the digit: 1563.902 ± 87.882, 728.040 ± 35.140, 1360.296, 653.368. Build d8bd0ff6, HRX0, -ngl 99. The 1563.9/1372.8 = +13.9 % claim holds.
• Ratio-only figures are unaffected by the unit: kernel shares (fa 48.5 / q4_k 29.0 / swiglu 15.0 / q6_k 6.6 % of union-busy, close to the page's sum-of-durations shares), dispatch counts (36 752 / 64 submissions), idle 15.1 % at pp16384 and 40.3 % at pp2048, and the 0.00 % cross-submission overlap.
• tools/copyright.py --check rc 0. Docs only, 1 file.

What is wrong (reproduced here, no GPU needed)
~/lb/exp34-gaps.py divides end_tick - start_tick by 1e6 and prints ms. Raw start_tick/end_tick are in the device_tick domain, and the bundle reports timestamp_frequency_hz: 99810000. The two clock_correlation records agree independently: 5 739 832 745 ticks over 57 398 303 572 host ns = 100.00 MHz, on both bundles.

iree-profile export --format=ireeperf-jsonl pp16384.ireeprof   # /home/bcloud/wt/hrxcts/build iree-profile, works on this box
dispatches 36752 submissions 64
span 50697.7 ms busy 43065.9 (84.9%) idle 15.1%
fa_prompt_f16_d128                                 20870.6 ms  48.5% n=2304
ggml_mul_mat_q4_k_f16_wmma_prefill_wave32          12504.7 ms  29.0% n=9216
ggml_mul_mat_swiglu_q4_k_f16_wmma_prefill_wave32    6457.3 ms  15.0% n=2240  per=2.8827 ms
ggml_mul_mat_q6_k_f16_wmma_prefill_wave32           2854.7 ms   6.6% n=2240

Every figure is the page's ×10.02 (= 1e9 / 99.81e6). Wall time confirms the corrected scale. The capture is warmup + 1 rep (64 submissions = 2 × 32 ubatches of 512, and pp2048 has 8 = 2 × 4). So one pp16384 prefill is ~25 s of device span, which matches the CSV's avg_ns of 25.08 s (653 tok/s). The page's "5 060.1 ms device span of one prefill" would mean 3 200 tok/s, and in the same row "389.1 ms" for pp2048 against 1360.3 tok/s (1.506 s). Those rows contradict their own tok/s columns.

Consequences for the conclusion

1. The swiglu dispatch is 2.883 ms, not 0.2877 ms. The 2 240 dispatches cover two prefills, so there is ~1 per layer per ubatch (1 120 = 32 × 35), not "two dispatches per layer". On the page's own reading that gives 1.44 ms per 256 rows, which is ~6x against the NPU's 8.61 ms, not 30x. On the page's "slower reading" it is 2.88 ms, which is 3x.
2. Re-running the page's own integer row-group scan (64 groups, 8.61 ms tile) with those inputs:

iGPU ms/256 rows	NPU tile	optimum n	ceiling
0.288 (as written)	8.61	2	1.032x
1.441 (corrected)	8.61	9	1.164x
1.441 (corrected)	2.15 (#754's 4x bar)	26	1.650x
2.883 (corrected, page's slower reading)	8.61	16	1.333x

So against the in-engine number, the per-GEMM split already clears #754's ~15 %. The "~4.5x bar", "30x" and "1.02-1.03x" do not follow.
3. "Reproduces the healthy boot from the other direction (0.313 vs 0.288 ms)" is a coincidence of the unit error. The in-engine number is actually ~4.6x slower than the doc's 20.03 ms microbenchmark (~8.9 TFLOP-s on the page's 9 728-column FLOP count, not ~44). That is a real discrepancy, and it is worth its own sentence. A plausible reading: the engine's matmuls run well below the microbenchmark. Prefill wall time agrees with that: 728 tok/s on a 4B is ~11 TFLOP/s overall.
4. "57.3 % of idle in gaps > 200 us": exp34's threshold is 200 000 ticks = 2 ms. At a true 200 us the share is 94.6 % (pp16384) and 96.0 % (pp2048).
5. The bonsai paragraph (7.8 TFLOP-s from tok/s, 5x the NPU) comes from wall-clock rates, not the capture. It still stands as written.

Needed for READY

• Convert ticks with the bundle's timestamp_frequency_hz (or the clock correlation) in exp34/exp37, and re-derive the per-kernel table, the "device span" column and the gaps line.
• State that the capture includes llama-bench's warmup run, and halve the counts per prefill.
• Redo "What that does to the bar", the "will not do" bullet and the #754 cross-reference with the corrected per-row number. Then say plainly that the in-engine figure disagrees with the microbenchmark, and which one the ceiling should use. If the corrected number shows the split pays, Refs #754 gets a different message than "the bar stands".

Recompute script: ~/lb/prefill-split/review-1270-recompute.py <export.jsonl>. It reads the iree-profile export output and needs no device.

[bong-water-water-bong]

VERDICT: READY at 544ae203 — main merges only; the reviewed text is byte-identical to 456936f5.

git diff 456936f5 chk1270b -- docs/hetero-prefill.md is empty, and the range is just d0122cec/544ae203 (main merges, including #1272 and #1273 under it). So the review I posted at 456936f5 — the cross-references (#782 closed by gpu-kernels#48, the 1372.8 tok/s anchor in the same file, the bonsai 144-152 / 93-96 % rows from ~/lb/prefill-split/RANKED-FOR-SUPREME.md), the recomputed bar arithmetic (64 × 0.288 = 18.43; 2 NPU groups → 17.86 vs 17.22 → 1.032x; 2.15 ms → 1.12x; 1.9 ms → 1.15x = 4.5x) and the one wording nit ("0.2877 ms per 512 rows … so 0.288 ms per 256 rows") all still apply unchanged.

[bong-water-water-bong]

Post-merge: the CHANGES at d0122cec was not answered before this merged. The READY at 544ae203 re-checked that the text was byte-identical to 456936f5. It did not address the finding, which is about that same text.

What landed on main (f2755a1d), in docs/hetero-prefill.md's third measurement:

• Every ms figure is 10x low. exp34-gaps.py treats 100 MHz device ticks as ns. The bundle reports timestamp_frequency_hz: 99810000, and its clock_correlation pair gives 100.00 MHz.
• The page's own rows show it: a "5 060.1 ms device span of one prefill" against 653.4 tok/s at pp16384 (a 25.1 s prefill), and 389.1 ms against 1360.3 tok/s at pp2048 (1.51 s).
• Corrected, the swiglu dispatch is 2.88 ms, so ~1.44 ms per 256 rows. That is a ~6x asymmetry, not 30x, and a ~1.16x per-GEMM ceiling at the 8.61 ms tile, not 1.03x. So "the bar stands at ~4.5x" does not follow, and Split prefill in the hybrid route: blocked on an NPU prefill GEMM ~4x faster (measured bar inside) #754's status is affected.

The reproduction is in the review above and needs no device: ~/lb/prefill-split/review-1270-recompute.py on the iree-profile export of ~/lb/prefill-split/qwen3-4b/pp16384.ireeprof.

A follow-up PR from the owning lane should fix the tick conversion in exp34/exp37 and re-derive that section. Until then, the 30x / 1.03x / ~4.5x figures on main should not be quoted.

[bong-water-water-bong]

The CHANGES review at d0122cec is answered, and it was right — the tick unit made every ms in the third measurement 10x too small. Corrections are in PR #1306 (Refs #754)

The READY at 544ae203 (20:08) re-reviewed the earlier READY's checklist and did not address this one, and the PR merged at
20:12 with the text unchanged (as that review notes, the diff was byte-identical). So this is the answer, from the lane that
now owns the page's split-side numbers:

Reproduced before changing anything — iree-profile export --format=ireeperf-jsonl on both bundles,
~/lb/prefill-split/review-1270-recompute.py, and timestamp_frequency_hz = 99 810 000 read from the bundle metadata itself:

dispatches 36752 submissions 64
span 50697.7 ms busy 43065.9 (84.9%) idle 15.1% gaps>200us share of idle 94.6%
fa_prompt_f16_d128                20870.6 ms  48.5% n=2304 per=9.0584
ggml_mul_mat_q4_k_..._wave32      12504.7 ms  29.0% n=9216 per=1.3568
ggml_mul_mat_swiglu_q4_k_...      6457.3 ms  15.0% n=2240 per=2.8827
ggml_mul_mat_q6_k_..._wave32       2854.7 ms   6.6% n=2240 per=1.2744

and the pp16384 span cross-checks against cap-pp16384.csv's avg_ns = 25 076 221 122 (25.08 s per prefill, 653.4 tok/s), so
the corrected scale is confirmed from two directions, exactly as the review argued.

What PR #1306 changes on the page, all of it the review's list: the per-kernel table, the "device span" column (two
prefills, not one), the dispatch counts (1 120 swiglu per prefill = one per layer per ubatch), the per-256-row cost
(1.441 ms, or 1.055 ms from total matmul FLOPs at 12.1 TFLOP/s), the asymmetry (6-8x), the ceiling (1.164x at
today's tile; 1.650x at the 4x bar; 1.333x on the slower reading), the gaps line (200 000 ticks ≈ 2 ms → 94.6 % /
96.0 % at a true 200 µs), the warmup statement, and the M≤2048 cell (unresolved: 1.00x / 1.143x).

Two things the review asked for that are now stated rather than implied:

1. The in-engine figure disagrees with the microbenchmark, and the ceiling uses the in-engine one. 0.313 ms per 256-row
   group in mosaic_bench against 1.06-1.44 ms in the engine's own prefill is a 3.4-4.6x gap; a split runs beside the
   engine, so that is the rate that decides it. The page now says so in its own sentence, with the wall-clock cross-check
   (653 tok/s on a 4B is ~12 TFLOP/s of matmul) on the engine's side.
2. The Refs #754 message changes, which is the substantive consequence. Against the in-engine rate the split is
   1.12-1.16x per GEMM and 1.078x end to end at M=16384 with today's NPU kernel — i.e. Split prefill in the hybrid route: blocked on an NPU prefill GEMM ~4x faster (measured bar inside) #754's per-GEMM target is already
   met and the "~4.5x bar" / "does not pay" verdict was the unit error talking. What blocks a split is the hybrid route's
   lack of a concurrent path, the loop-to-engine gap, and the unresolved small-M cell. That is posted on Split prefill in the hybrid route: blocked on an NPU prefill GEMM ~4x faster (measured bar inside) #754 as well.

docs/hybrid.md's sections from #1274/#1287 carried the same inputs and are corrected in the same PR (attention 48.5 %,
per-model range 1.038-1.070x). tools/split_ceiling.py now takes --gpu-tile-ms and self-tests the review's three rows
(1.164x / 1.650x / 1.333x), so the next person cannot reproduce the old numbers by accident.