HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines)

Split out of #390 (whose reporting half is done). With the HIP add-on the Flash-Next experts run on HRX (fork #142/#144, gpu-kernels #38/#39), and the file now measures PPL 1.8090, prefill 287 tok/s, decode 24.0 (UD-IQ4_XS, HRX0, no drafter; UD-IQ3_XXS 297 / 26.9). The field on the same chip reports prefill 510 (tuned Vulkan llama.cpp at 8k), 1,175-1,309 (halogen), 1,679-2,013 (gufo), 2,372 (hipfire), and decode 33-60 with drafters.

What gufo does that we do not (from the 2026-10-11 comparison, memory gufo-benchmark): prefill batches up to 4096 (dense) / 8192-token chunks (MoE), WMMA quant GEMMs or dequant-to-BF16 hipBLASLt for big shapes, FP16 activations, WMMA attention, MMQ experts with fitted tile widths, int8 WMMA for Q8_0 dense. Our expert prefill path is the HIP mmid GEMM at m64 tiles; the IQ2_S kernel GEAK authored is latency-bound at ~135 GB/s (52%).

Ask: a Flash-Next prefill plan on HRX: (1) dispatch profile of pp512 / pp4096 on UD-IQ4_XS with owners and the CPU splits (engine #389: -ub > 512 fails res=-3 on HRX, so chunking is capped), (2) expert-GEMM tile width fitted to the 512x640x2560 routed shapes, (3) the ub cap, (4) an MTP head on HRX for Flash-Next for the decode side. Numbers to beat: 1,679 prefill (gufo, same file), 33 decode (Vulkan llama.cpp with MTP).

🤖 Generated with Claude Code

docs: where Flash-Next prefill's 287 tok/s goes, and the ubatch cap t…

The claim still holds at the current pin, and the numbers are now in-tree rather than in a memory file — docs/npu-leaderboard.md:27-29:

Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp512  | 286.63 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846
Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill p1024 ub576 | 269.34 tok/s | ngl=99, fa=1, ub=576, engine=0d0c846
Flash-Next (IQ4_XS 4.25 bpw) | hrx | prefill pp4096 | 260.59 tok/s | ngl=99, fa=1, ub=512,  engine=0d0c846

So 287 tok/s vs gufo's 1,679 is real and reproducible on paper, and nothing in the tree since has moved it: the #622 tiled transpose and the #623 first-token pair are 27B NPU rows (435.9 s -> 145.7 s, docs/npu-leaderboard.md:34-35), not this prefill path. The gap analysis in the issue body (batch size, WMMA/dequant GEMM, FP16 activations, MMQ experts, int8 WMMA for Q8_0, latency-bound IQ2_S expert kernel at ~135 GB/s) is the right shape and is unimplemented.

Classification: needs GPU measurement — specifically llama-bench -fa 1 -ngl 99 -ub 512 -p 512 (3 repeats, and -ub 576 -p 1024 plus -p 4096 to match the three rows) on UD-IQ4_XS with the HIP add-on, on the current pin and again on 40c4ccb76e48 (the #610 bump target), reported as a table against 286.63 / 269.34 / 260.59. Without that row the next change to the expert GEMM has no before-number to beat. Left to the measurement lane — this box is CPU-only today.

Two things, and the second one is a retraction.

(a) Where this should look next. ggml_mul_mat_f32_f32_wmma is 8.6% of device time at pp512 — 56.8 ms of the pass — over 522 dispatches in the profiled bundle (docs/hrx.md:414-422), inside a 4.25 bpw model. That is a defect regardless of the per-layer accounting. One correction to the mechanism, though: it is not "some projection is falling back to an unquantized matmul" (docs/hrx.md:429-430). The operands are the file's own F32/BF16 tensors — 48 ffn_gate_inp (F32 [2560,512] router), 96 hc_{attn,ffn}_inject (F32 [10240,4]; N=4, so every quantized matcher declines), 72 ssm_alpha/ssm_beta (F32 [2560,48]), 24 indexer.{q,k}_proj (BF16) — 240 of the 261 per-forward f32 matmuls (perf-572-findings.md). ~95 GFLOP inside 56.8 ms ≈ 1.7 TFLOPS against ~9.5 for the expert GEMM, so the cost is dispatch overhead on tiny shapes, not a bad matmul. The lever is fusing or dispatch-batching those, not quantization.

(b) Retraction. An earlier reading today held that the bundle contains two forwards (92 swiglu dispatches = 2 × 46 layers) and therefore that device time was 37% rather than 76% of the pass. That does not survive checking. The GGUF says qwen4exp.block_count = 48 (header read of Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf, also qwen4exp.expert_count = 512), and the dispatch counts divide by neither 48 nor 46 — 92, 86 and 522 are 4×23, 2×43 and 2×261. So docs/hrx.md:409-411 stands as written and the 76% (1,355 ms of the 1,786 ms pp512 pass) is not being changed. The per-layer attribution inside the bundle is unresolved, not settled in the other direction either.

Next: the profiler run that settles per-layer counts — dispatch counts resolved against block_count with the bundle's own op names, at ub 512 and again at ub 2048, so kernel counts can be attributed per layer instead of divided by assumption.

Item 1: the host gap, measured. It is not there on any model that fits on this box, and the ~1.0 s/forward premise does not survive either instrument I could bring to it.

First the premise. The brief takes device time as 755.7 ms of the 1,786 ms pp512 pass (41.9 %) and the rest as host. This file's own profile records 1,355 ms of 1,786 ms as device (76 %) (the section merged as #616), and comment (b) above re-affirmed that 76 % after retracting the two-forwards reading. Those two splits cannot both be quoted: at 76 % the host leg is 431 ms, not 1,030 ms. The 755.7 ms figure is not in this issue or in docs/hrx.md.

Two instruments, both on engine 0d0c846 + llama.cpp 8ee20a0 + gpu-kernels 2b626798a.

A host-path profiler (GGML_HRX_HOST_GAP=1, llama.cpp ggml-hrx: an env-gated host-gap profiler (engine #572) llama.cpp#154, our own new file, off by default) that times every phase of a graph compute. Fully offloaded, llama-bench -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0, steady-state forward:

model	forward	graph build (outside the backend)	lookup	bind	upload	flush	wait (device)
Qwen3-Coder-30B-A3B Q4_K_M, 1,161 dispatches	263.5 ms	1.5 ms	0.004 ms	0.002 ms	0.005 ms	0.001 ms	263.46 ms
Qwen3.8-27B UD-Q4_K_XL, 1,519 dispatches	1369.0 ms	0.9 ms	0.003 ms	0.003 ms	0.004 ms	0.001 ms	1368.89 ms

Every item the brief names is in that row: graph build 1.5 ms, program-cache lookups 4 µs, host copies 5 µs, GET_ROWS fallback absent (one graph, no split, no CPU node), wait spin inside replay.wait where the device leg is. Total host: 1.5 ms of 263.5 ms, 0.6 %.

libhrx's own dispatch profile (HRX_PROFILE_FILE=<bundle> HRX_PROFILE_MODE=dispatch, then iree-profile statistics) — the only profiler that sees HRX dispatches:

model	dispatches in bundle	device kernel time	wall for the same 2 forwards	device/wall
Qwen3-Coder-30B-A3B Q4_K_M	2,322 = 2 × 1,161	538.14 ms	523.8 ms	103 %
Qwen3.8-27B UD-Q4_K_XL	3,038 = 2 × 1,519	1,871.15 ms	1,657.2 ms	113 %

Both instruments agree on the dispatch counts to the unit (1,161 and 1,519), and device kernel time is the whole forward; the >100 % is the profiler writing records.

rocprofv3 cannot do this job, and that may be where the 41.9 % came from. rocprofv3 --kernel-trace --stats on an HRX run exits 0 and writes no trace file: HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets, so the profiler's HSA interposition never sees a dispatch. Anything summed from a "device trace" that returns nothing, or from a partial one, gives an inflated host leg.

What a host leg looks like when there is one. The same 27B at -ngl 20 (a third of the layers on the CPU) runs 3,746 graph computes per forward instead of 2, and its host-staging upload alone costs 145.7 ms per forward (39 µs per graph), with the CPU backend's own compute on top. That is the shape of a real host component: a graph that is not fully on HRX.

Counts the last comment asked for (GGML_HRX_HOST_GAP=1, dispatches per forward):

model	p/ub 512	p/ub 1024	p/ub 2048
Qwen3.8-27B UD-Q4_K_XL	1,519	1,616	1,813
Qwen3-Coder-30B-A3B Q4_K_M	1,161	—	1,137

Resolved against block_count: 27B 62 layers → 24.5 dispatches/layer at ub 512, 29.2 at ub 2048; 30B-A3B 48 layers → 24.2 and 23.7. The counts are stable, so per-layer accounting can be done from them, but the timings are not: the same 27B pp512 read 828.6 / 870.2 / 1120.8 / 1369.9 / 1518.2 ms across five runs on this box today (±57 %). Any before/after on this lane needs repeats inside one gpu-hold claim.

What I could not measure. Flash-Next UD-IQ4_XS is 93.7 GB (87.3 GiB) of weights; this box has 75–82 GiB available with the owner's 27B NPU job resident, so loading it would swap and every number from it would be a swap number. UD-IQ3_XXS (76.4 GiB) is the one Flash-Next file that could be tried here, and only with that job stopped — your call, not mine. One run of the profiler on either file answers this issue's item 1 for Flash-Next itself.

On your item (a) (the f32 matmuls), from the same profile: on the 27B they are fused — common.mul_mat.f32_f32_wmma appears in the matcher census as fused and no ggml_mul_mat_f32_f32_wmma kernel row appears at all — so the 522-dispatch / 8.6 % shape is Flash-Next's own (hc_*_inject N=4 each getting a dispatch). The batching fix already has a precedent in this tree on a file that fits: qwen3_moe_router_projection_f32_four_row_wave32 batches the 30B-A3B's f32 router projection — 16 dispatches per forward, 3.3 % of its device time, instead of one per token.

PRs, neither merged: #646 (docs/hrx.md, the whole profile above) and 1bit-MONSTER/llama.cpp#154 (the profiler).

Follow-up with the long-prompt case closed, since it was the one place the host path did grow.

-p 4096 -ub 2048 on Qwen3-Coder-30B-A3B Q4_K_M, steady-state graph, every step named (llama.cpp #154 gained the remaining scopes: cache.key, replay.total, replay.rebind, replay.transient, replay.launch, replay.download):

wall 1158.581 ms = graph.lookup 0.179 + graph.bind 0.016 + graph.execute 1158.385
  execute          = cache.key 0.007 + replay.total 1145.534 + ~12.8 ms (cache mutex/prologue)
  replay.total     = replay.wait 1143.718 + rebind 0.002 + transient 0.001 + upload 0.004
                     + launch 0.042 + flush 0.001 + download 0.002

so host time outside the device wait is ~15 ms of a 1158.6 ms forward (1.3 %), and at ubatch 512 it is 0.03 ms of 263.5 ms (0.01 %). The 145 ms/forward I first saw at this ubatch was the one-time cost of the ubatch's first graph — prepare plus kernel JIT for the new shapes, 4.5 s above the following graphs — not a per-forward path.

Summary of what item 1 asked for, for every model that fits on this box:

	ub 512	ub 2048
host per prefill forward outside the device wait	0.03 ms (0.01 %)	~15 ms (1.3 %)
graph build (llama.cpp side)	1.5 ms	~2 ms
program-cache lookup	4 µs	0.18 ms
host copies (bind + staging)	7 µs	25 µs
GET_ROWS fallback	absent, one graph, no split	same
mutex/wait spin	inside `replay.wait`, which is the device leg	same

That is as far as item 1 goes on this box. Items 2 and 3 both need the 87 GiB Flash-Next file: 2 has no host item left to fix, and 3 is a device-side defect that only exists in that model's graph. One decision unblocks both — stop the owner's 27B NPU job, or accept UD-IQ3_XXS (76.4 GiB vs 75-82 GiB available) — and the run is one command each now that the profiler and the libhrx dispatch bundle are both in place.

PRs, neither merged: #646 (docs/hrx.md) and 1bit-MONSTER/llama.cpp#154 (the profiler, twelve phases, off by default).

docs: Flash-Next measured - the ~1.0 s is CPU-side, not the HRX host …

docs: the ~1.18 s is one GET_ROWS, declined by kMaxGetRowsRowCount (e…

Item 1 is closed, on the file itself, and item 2 has a one-line target. The owner's NPU job exited at 04:41 (114 GiB free), so I loaded the 87.3 GiB UD-IQ4_XS and measured it.

Qwen3.8-Flash-Next UD-IQ4_XS, -ngl 99 -fa 1 -ub 512 -p 512 -dev HRX0
  llama-bench                      1,795.4 ms/forward = 285.2 tok/s   (in-tree row: 286.63)
  HRX graph computes per forward   2 graphs: 2,566 dispatches in 618.6 ms, and a 1-dispatch one in 0.2 ms
  host phases inside HRX           <= 0.13 ms total (lookup, bind, cache key, transient, rebind,
                                   upload, launch, flush, download)
  libhrx dispatch bundle           2,567 dispatches/forward, 1,353.5 ms of device kernels per 2 forwards
                                   = 676.8 ms/forward = 37.7 %
  llama-server -lv 4 placement     6060 nodes, 1 off HRX (GET_ROWS x 1), 3 graph splits

So: 676.8 ms device (37.7 %), 0.13 ms of HRX host path, and ~1,177 ms (65.6 %) in one CPU node. The ~1.0 s is real; it is GET_ROWS fallback, the last item on this issue's own suspect list, and it is a single declined dispatch:

the node reads per_layer_token_embd.weight, which this file carries as IQ3_S [160, 320001536] — a ~20 GB table (token_embd.weight, the ordinary one, is Q8_0 [2560, 248320]);
ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cpp declines it at is_supported_row_count(): kMaxGetRowsRowCount = 262208 (line 23) against 320,001,536 rows;
the CPU then gathers 512 rows out of 20 GB per forward, and the cost is the page faults of touching it: ~35 ms at -p 128 (565.5 ms/forward) vs ~1.18 s at -p 512. Super-linear in tokens, because more tokens means more distinct rows and more cold pages. The graph splits 3 ways around it (HRX → CPU → HRX), so the device and host legs cannot overlap.

Item 2 is therefore: let that gather stay on HRX — raise/relax kMaxGetRowsRowCount for a table this shape, or add a dispatch for a gather from a table this large. Ceiling if it lands: the forward goes from 1,795.4 ms to ~677 ms of device plus whatever the gather costs on HRX, i.e. up to ~2.3x pp512 on this file — which is the ~2.4x roof in the brief, arrived at from the other end.

A correction to the tree while I was there: docs/hrx.md's "1,355 ms of the 1,786 ms pass (76 %)" is a two-forwards device total measured against a one-forward pass. 1,353.5 ms is exactly 2 × 676.8 ms from the bundle I measured. Per forward the device leg is 37.7 %, so the brief's 41.9 % was the closer figure and the published 76 % should come out. Fixed in #646.

Controls (a graph with no off-HRX node has no host leg at all): Qwen3-Coder-30B-A3B Q4_K_M 263.5 ms/forward with 0.02 ms of host phases and 1,161 dispatches; Qwen3.8-27B UD-Q4_K_XL 870.2 ms with 0.01 ms and 1,519. Both stay ~100 % device. rocprofv3 collects nothing on HRX (it exits 0 and writes no traceback files — HRX dlopens TheRock's libhsa-runtime64.so.1 itself and libhrx writes its own AQL packets), so the device leg has to come from HRX_PROFILE_FILE + iree-profile statistics.

PRs, neither merged: #646 (docs: the whole profile, the correction, the GET_ROWS target) and 1bit-MONSTER/llama.cpp#154 (the host-path profiler, twelve phases, off by default).

docs: put the Flash-Next prefill forward on a gfx1151 roofline (engine …

The Flash-Next prefill forward on a gfx1151 roofline (TraceLens at ff705ed; no GPU run needed for the projection).

Method: #1175's TraceLens_model_roofline_projection parses a HF checkpoint, and this model exists here only as GGUF — qwen4exp is a hybrid llama.cpp architecture with no HF counterpart. So I used the same machinery a level down: TraceLens.PerfModel's GEMM / flash_attention formulas over shapes and exact per-tensor bytes read from the GGUF directory, against the peaks TraceLens keeps for gfx1151 after #1172 removed the bundled Radeon_8060S.json (measured spec already on this box: 256 GB/s, 59.39 TFLOP/s fp16, 118.78 TFLOP/s int4). Measured device time per class is from the libhrx dispatch bundle of the same model, so each row is roof against what the device actually did.

class	weights GB	touched GB	FLOPs T	mem roof ms	compute roof ms	measured ms
MoE experts	59.52	59.52	2.416	232.5	20.3	306.1
gated-delta-net layers (36)	2.25	2.25	2.126	8.8	35.8	33.7
token embedding + LM head	1.20	1.20	0.651	4.7	11.0	(dense)
full-attention layers (12)	1.02	1.02	0.671	4.0	11.3	136.3
MoE shared expert + router	0.50	0.50	0.306	2.0	5.2	(dense)
hyper-connections	0.34	0.34	0.644	1.3	10.8	35.1
PLE table	28.80	0.002	0	0.0	0.0	0.5
total	93.67	64.87	6.81	253	115	676.8

touched < weights because at 512 tokens with top-10 of 512 experts the expected number of untouched experts per layer is 512·e⁻¹⁰ = 0.02 — the expert weights are compulsory traffic — while the 28.8 GB PLE table is only gathered, one 90-byte IQ4_NL row per token per layer (2.2 MB). So the 87.3 GiB file does not put a 366 ms floor under a 512-token pass: 253 ms is the floor.

Three consequences:

The forward is memory-bound by 2.2× (253 ms against a 115 ms compute roof). No compute-side kernel rewrite can beat 253 ms on this file, which is worth knowing before anyone chases the expert GEMM's tile width again.
The device leg is at 37 % of its own compulsory-traffic roof — 2.7× the floor. The MoE expert GEMMs are the class closest to roof (306.1 ms against 232.5 ms, 76 %), so the slack is not there: 1,189 elementwise/copy/norm dispatches cost 106.6 ms with no weight traffic at all (~90 µs each, dispatch-bound), 273 f32 small matmuls cost 58.4 ms, and 395 dense projection GEMMs 136.3 ms.
The host leg is the largest single line in the forward: the one CPU GET_ROWS costs ~1,177 ms = 4.6× the memory roof of the entire pass, to move 2.2 MB. It is page faults on a cold 28.8 GB mmap'd table, not arithmetic — which is why it does not warm up between forwards (1,788.6 then 1,802.1 ms), while the 77 device-side get_rows dispatches cost 0.5 ms.

So item 2's ceiling, in roofline terms: give that gather a dispatch and the forward becomes ~620 ms of device, i.e. ~2.9× pp512 (285 → ~830 tok/s), and the file then sits at 37 % of its memory roof. The remaining gap to the floor is the dispatch-bound small ops — item 3's f32 batching is 58.4 ms of it and the elementwise/copy stream 106.6 ms.

Reproduce: ~/lb/i572/roofline_fn.py <gguf-prefix> <iree-profile statistics --format=jsonl> 512. The same table is in #646.

docs: where a Flash-Next prefill forward's 1.8 s goes, and the ~1.0 s…

README: current facts in the first heading and paragraph, badges; doc…

README: current facts, badges and links; docs: no line starts with an…

Bump HRX: llama.cpp 5e6388a5a (host-gap profiler, KV zero padding, De…

Status: item 1 shipped, item 2 merged-but-unproven, item 3 is the same file's graph. Both remaining measurements are now blocked on #669, and I have stopped running the 87.3 GiB file on HRX.

What shipped. The host-path profiler (llama.cpp #154) and the GET_ROWS row bound (#159) are both merged; the docs section in docs/hrx.md (#646) is merged. Fork 1bit/hrx-vulkan-patched is now at 5e6388a5a852281c6929303c1219665b2a6d4c0a, and the next HRX pin bump moves the engine's third_party/llama.cpp from d8d2f5b to it, carrying six commits: fe04824 (profiler), b8861a9 (#155 KV padding), 9e8453a/f61f290/01d7df4 (#156/#157/#158 dsv41) and 5e6388a (#159, this issue's item 2). Note for whoever raises it: tools/check_pins.py gates only the hrx-system pin against config/hrx-long-prompt-validated-pin.config, and that pin does not move in this bump — so this llama.cpp-only bump does not need the long-prompt probe re-run.

Item 2 numbers, partial and honest. #159 alone is not enough, which is what the placement analysis predicted:

the bound was necessary (the kernel's own index.assume capped ids at 262,207, mirrored by kMaxGetRowsRowCount), and it is now 2^29 with the byte-base proof;
but placement is two other policies: llama-model.cpp keeps every input-layer tensor on the CPU, and LLAMA_LAZY_MODE_AUTO mmaps every tensor over 4 GiB (qwen4exp.cpp asks for TENSOR_READ_LAZY on this one), which also short-circuits -ot;
measured on the pin-bump head (engine main + llama.cpp 5e6388a), pp512 ub512, default flags, 3 repeats: 281.9 tok/s against 285.2 tok/s on the cand-next build — no measurable change, as predicted;
with -lzm off -ot 'per_layer_token_embd\.weight=HRX0' the table does move: CPU_Mapped 644.14 MiB | HRX0 88688.02 MiB (61,222.07 + 27,465.95). That configuration is exactly what HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 forbids on this box: HRX pins the host weight mmap while holding 88.7 GB of device buffers, MemAvailable went to ~0 with no error from the runner, and the owner killed the run. I am not running that file on HRX again until HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669 lands.

Item 3 is parked for the same reason, and I checked it rather than assuming. From the dispatch bundles already on disk: on Qwen3.8-27B UD-Q4_K_XL there is no ggml_mul_mat_f32_f32_wmma row at all — the f32 matmuls are fused (common.mul_mat.f32_f32_wmma (fused) in its matcher census), with only hrx_dp_f32_to_f16 (246 dispatches, 10.9 ms over 2 forwards) as the f32 traffic. On Qwen3-Coder-30B-A3B the router projection is already batched (qwen3_moe_router_projection_f32_four_row_wave32, 16 dispatches/forward). So Flash-Next's 273 standalone f32 dispatches / 58.4 ms are its own hc_*_inject (N=4) and ssm_alpha/beta shapes, and there is no runnable file on this box that exercises that pattern — a fix could be written but not validated, which the lane rules do not allow.

So: item 2 needs #669 (or the owner freeing the box for one long run with the table on the CPU and the gather dispatched — not viable at 88.7 GB), and item 3 needs item 2's file. I would rather leave both recorded than publish numbers from another thrash event.

The server-path row exists at last — and it disagrees with every number in this issue

Five windows and three diagnostics after the harness row was approved, there is a row. It is the server path
(bench.py against a live /v1/chat/completions), 5 reps, ctx 16384, and it is labelled llamasrv-fork177
rather than onebit for a reason that is item 3 below.

binary   /home/bcloud/wt/fnpre/build/hrx/llama/bin/llama-server  sha256 add45105…  llama.cpp a4f47ab8d
argv     -m …UD-IQ4_XS-00001-of-00003.gguf --host 127.0.0.1 --port 19110 --metrics --device HRX0
         -ngl 99 --jinja -c 16384 --cache-reuse 256          (verbatim from `1bit serve --print-backend`,
                                                               argv[0] swapped for the fork build, no -fa)
host     avail=99GiB budget=96GiB gtt_used=7GiB psi=0, coder-llm stopped, 16 threads

cell	prompt tokens	median	min–max	spread	TTFT median
pp512	580	300.83 tok/s	291.3–408.1	38.8%	1.93 s
pp4096	3856	85.05 tok/s	76.9–135.4	68.8%	45.30 s
ttft2k	1971	tg 24.31 tok/s	22.6–24.3	6.9%	4.54 s
tg128@d0	99	25.88 tok/s	25.3–26.1	3.1%	1.31 s
tg128@8k	7530	10.00 tok/s	6.4–11.1	47.2%	67.17 s
correctness	—	sane 3/3	859/646/219 chars	—	—

Power over the same run: pkg_W mean 82.3 / median 86.4 / max 108.9, peak GTT 99.2 GiB, Tctl max 80.1 °C.
Rows: ~/lb/supreme/raw/llamasrv-fork177/qwen38-flash-next{,-tg}/{bench.json,power.csv,correctness.json} with
provenance.txt (binary sha256, llama.cpp commit, argv) beside them.

1. tg is measurable for this model — on this build, and my earlier "gen path aborts" was the pin, not the model

tg128@d0 25.88 tok/s at 3.1% spread plus a sane correctness gate, on the exact path that has failed with
materialize host weight value 3 failed / hsa_amd_memory_pool_allocate HSA_STATUS_ERROR_OUT_OF_RESOURCES in
every previous attempt. The HSA RESOURCE_EXHAUSTED is real (it is in r23-A-getrows-thr.err), but it belongs to
the shipped pin's host-weight residency, not to Flash-Next. My tg64 framing in this issue was measuring a build.

2. The server path does not show the gains this issue reports — do not mix the two paths on a scoreboard

Same file, same box, same day:

path	pp512	pp4096	pp16384
llama-bench, patched (`13eb1f32d`/later, `-fa 1`)	587.9	546.5	314.9
llama-server, this row (no `-fa`)	300.8	85.1	not run

The 4k cell is 6.4× apart, and the one obvious unmeasured difference is flash attention: every llama-bench arm
ran -fa 1, the product argv carries no -fa at all. One arm of one window settles it (-fa 1 on this same
server); until then, the 2.07× / 6.5× / 3.2× rows are a llama-bench result and this table is what an HTTP
client actually sees. If -fa closes it, that is a product-flag finding, not a kernel finding.

Also worth noting for the prefill work: the warmup graph places 0 of 6060 nodes off HRX (1 graph split) on
this build, so GGML_HRX_STRICT=1 has nothing to refuse at bs=1. The bs=512 prefill graph still reports
splits = 123, which is where any CPU-side work lives.

3. The shipped pin cannot load this model at all, and the front-end's readiness budget is shorter than the load

Two independent facts, both measured tonight with the same argv, weights and box, 5 minutes apart:

inst  /home/bcloud/.local/share/1bit-hrx-a5271d9/…/llama-server  (8cdb5426, engine a5271d9, llama.cpp d8bd0ff)
      t=19.46 s  load_tensors:         HRX0 model buffer size = 61222.07 MiB
      t=19.46 s  load_tensors:   CPU_Mapped model buffer size = 27465.95 MiB
      … 280 s of nothing. sched_reserve is never reached.

lane  /home/bcloud/wt/fnpre/build/hrx/llama/bin/llama-server  (add45105, llama.cpp a4f47ab8d)
      t=2:10     sched_reserve: HRX0 compute buffer size = 140.62 MiB, graph nodes = 6768, splits = 123/97
      t=2:10     warming up the model → placement: 6060 nodes, 0 off HRX
      t=11.5 min READY /health 200

So 1bit serve --device hrx on an 87.3 GiB model fails twice over: the pinned build stalls during load, and even
with a build that loads, 687 s to ready exceeds the hardcoded 600 s in app/serve.cpp:2245 (wait_ready(std::chrono::seconds(600)), also at :1985, no flag). A user on a Lemonade box sees
1bit serve: backend 0 did not become ready after 10 minutes and nothing else — no op name, no phase, no hint,
and the backend's own -lv output is not surfaced on that path.

This is the OWNER item already recorded (the pin trails the fork by 17 commits and #177 is one of them), plus one
new decision that is not mine: the readiness budget needs to be a flag or derived from model size. I am not opening
a competing pin PR, and the row above is deliberately not credited to onebit:hrx.

Next, in order, one window each: -fa 1 (item 2), then the -ub/-b sweep item 2 of my brief has never run,
then pp16384 on its own pass at a larger ctx (a 16384-token prompt cannot fit -c 16384, and this pass already
ran the die to 80 °C).

The 5.2× is two default flags, not a kernel: `-fa` off and `--cache-reuse 256`

Three server-path windows, same binary (add45105, llama.cpp a4f47ab8d), same weights, same box, same 13 argv words
from 1bit serve --print-backend, changing only the two flags named in the title. 5 reps each, bench.py
unique-prompt cells, ctx 16384, /health 200 confirmed, correctness gate sane 3/3 in every run.

config	load → ready	pp512 (583 tok)	pp4096 (3857 tok)	tg128@d0	tg128@8k	pp4096 spread
product defaults (`-fa` unset → `auto`, `--cache-reuse 256`)	687 s	300.83	85.05	25.88	10.00	68.8%
`+ -fa on`	976 s	390.52	153.71	26.10	8.75	9.1%
`+ -fa on --cache-reuse 0`	439 s	419.43	444.89	25.82	12.99	15.7%
llama-bench, patched, `-fa 1` (no cache-reuse exists there)	—	587.9	546.5	—	—	—
llama-bench, unpatched, `-fa 1`	—	285.6	84.1	—	—	—

Four things fall out of that table.

1. On the server path, the two defaults erase the patch's entire benefit. Unpatched llama-bench measures 84.1 at
pp4096; the server on the patched build measures 85.05 with the product's own flags. The 6.5× that #177 buys is
invisible through 1bit serve until -fa is on and --cache-reuse is off. Anyone who concludes from the
benchmark rows that users are getting a 6.5× faster prefill is reading a configuration the product does not ship.

2. --cache-reuse 256 is the dominant cost, and it is not small. Turning it off, with FA still on:
pp4096 153.71 → 444.89 (2.89×), tg128@8k 8.75 → 12.99 (1.48×, and the spread collapses 37% → 1.08%), and
load 976 s → 439 s (2.2× faster). The mechanism is consistent with the model, not with the box: Flash-Next
carries a GDN recurrent state, --cache-reuse works by shifting KV and taking state checkpoints every 256 tokens,
and this harness sends a unique prompt per rep (unique_prompts: true) — so the checkpoint bookkeeping is paid on
every request and never earned. I have not traced it inside the scheduler; what is measured is the cost, and the
prediction is that a workload with repeated prefixes pays far less.

3. -fa is a prefill lever, and only that. pp512 300.83 → 390.52, pp4096 85.05 → 153.71, and the wild spreads
collapse (68.8% → 9.1%) — but decode is unchanged at depth 0 (26.10 vs 25.88) and slightly worse at 8k under
cache-reuse. Product-side it is a default question, not a missing flag: app/serve.cpp:1004 already forwards
-fa <value> to the child and :2724 accepts it. Unset resolves to flash_attn = auto (that line is in the
launch log of the defaults run; enabled in the other two), and auto measurably ≠ enabled for this model.

4. With those two flags the load fits under the front-end's readiness cap. 439 s against the hardcoded 600 s in
app/serve.cpp:2245 — whereas the defaults took 687 s and the box-side failure I reported earlier is what that cap
turns into. So the product could serve this model on a pin that carries #177 today, if serve passed -fa on and
--cache-reuse 0 for it.

Residual after both flags: 444.89 vs llama-bench's 546.5 at the same flags = 19%, which is what an HTTP client
sending unique prompts through the jinja chat template should cost. Nothing in that 19% needs a kernel.

Best-config rows (5 reps, power sampled throughout, provenance beside them):
~/lb/supreme/raw/llamasrv-fork177-fa1-nocr/qwen38-flash-next{,-tg}/ — pp512 419.43, pp4096 444.89,
ttft2k 24.17 tok/s @ 3.75 s TTFT, tg128@d0 25.82, tg128@8k 12.99, correctness sane 3/3 (859/646/219 chars),
pkg_W mean 91.9 / median 92.0 / max 104.9, peak GTT 99.3 GiB, Tctl max 67.5 °C.

Not my call, but the decision these three windows hand to the owner is a small one: the pin bump (17 commits, #177
among them) plus two flags in the argv 1bit serve builds for MoE/hybrid models. Neither is a kernel change, and
together they are 5.2× prefill and 1.3× decode at 8k on this model. Next from this lane: the -ub/-b sweep with
these flags on (brief item2, still never run), then pp16384 on its own pass at a larger ctx.

Retraction, 90 minutes after posting the ladder: `--cache-reuse` was already off in every run, so it cannot be the 2.9×

My previous comment said --cache-reuse 256 costs 2.89× at pp4096. The model's own log says otherwise, and I should have read it
before attributing anything to that flag:

llamasrv-fork177/…/launch.log:244        10.54.167.430 W srv load_model: cache_reuse is not supported by this context, it will be disabled
llamasrv-fork177-fa1/…/launch.log:243    15.43.972.023 W srv load_model: cache_reuse is not supported by this context, it will be disabled
llamasrv-fork177-fa1-nocr/…/launch.log   (no such line — the flag was 0, so the check never ran)

server-context.cpp:1324 disables it at load whenever llama_memory_can_shift() is false, which it is for this model;
server-context.cpp:3370-3375 would then ignore n_cache_reuse per slot anyway. So in both runs that had
--cache-reuse 256 the cache-reuse path never executed. Setting it to 0 changed nothing but the log line.

The resolved context parameters are identical in the two runs I compared:

n_ctx 16384   n_ctx_seq 16384   n_batch 2048   n_ubatch 512   flash_attn enabled      (both runs)

So the i9 → i10 step (pp4096 153.71 → 444.89) has no flag cause, and the "two default flags" headline of my previous
comment is wrong in its second half. What survives is the first half, which was a controlled comparison — i8 → i9 changed
only -fa (auto → enabled) and moved pp512 300.83 → 390.52 and pp4096 85.05 → 153.71 — and the reference to
app/serve.cpp:1004 forwarding -fa.

What the timing data actually points at: whether the host-weight cache survives the load

power.csv from the same three runs, sampled through the bench:

run	pp4096	median RSS	max RSS	GTT median	gate `avail` at launch
defaults (`-fa` auto)	85.05	1,161 MiB	6,865 MiB	101,087 MiB	99 GiB
`-fa on`	153.71	1,072 MiB	13,866 MiB	101,058 MiB	100 GiB
`-fa on --cache-reuse 0`	444.89	9,697 MiB	12,474 MiB	101,014 MiB	106 GiB

The fast run is the one where the process kept ~9.7 GiB of RSS; the two slow runs were trimmed to ~1.1 GiB, i.e. the
host-side weight materialization (#669's residency cache) does not survive. This is the same sawtooth the load probe
showed on 2026-11-05 00:19 (RSS 24 GiB → 4 GiB against a pinned 97 GiB) and it is a better explanation of every
symptom in this issue than any flag: the unpatched llama-bench pp4096 spread of 34.9–61.7 s across three reps, the
68.8% spread in my own pp4096 cell, and the 976 s vs 439 s load times are all "did the reclaim win".

What that means for the number, stated conservatively: the server-path pp4096 cell is bimodal on memory
headroom — ~85-155 tok/s when the host-weight cache is trimmed away on a nearly-full box, ~445 when it survives —
and the boundary sits right where this 87.25 GiB model's ~97 GiB GTT allocation meets this machine's 122 GiB. It is
not a flag, not a kernel, and not something -ub will move.

Next three measurements, in this order

Repeat the slow configuration (-fa on, no --cache-reuse change) on a box held to the same ≥106 GiB headroom
the fast run had. If it comes back at ~445, the flag story is fully dead and headroom is the variable.
Repeat the fast configuration with the box deliberately tighter, to find where it flips — that boundary is the
number users actually live on, and it is the one worth putting on a board ("this model needs ~100 GiB free").
Only then the -ub/-b sweep, which is currently queued and which I will re-read with RSS in hand, because an
arm measured in trimmed mode is not comparable to one measured in resident mode.

I would rather retract this myself 90 minutes late than have someone build a default-flag recommendation on it. The
cells themselves are real and reproducible artifacts — they are in ~/lb/supreme/raw/llamasrv-fork177*/ with
power.csv, provenance.txt and correctness — it is my explanation of what separated them that was wrong.

First controlled lever since the retraction: `-ub 2048` is 2.3× at pp4096 (264 → 615 tok/s), and `-ub 4096` does not fit

Why this one is controlled and the ladder was not: one llama-bench process, one model load, every -ub value measured against the same resident weight mapping. The 5.3× run-to-run spread I retracted on (comment above, 6653354021) is a between-run variable, so it cannot survive inside a single process — a 2.3× here is a 2.3× whatever regime the box was in.

Ticket fnpre flashnext ub matrix: gate opened 06:19 (avail=105GiB budget=98GiB gtt_used=5GiB) after 48 min in the queue, 1201 s, -p 4096,16384 × -ub 512,1024,2048,4096, -b 4096 held fixed, -n 0, -fa 1, -r 2, -dev HRX0 -ngl 99, peak Tctl 80 C:

prompt	`-ub 512`	`-ub 1024`	`-ub 2048`	`-ub 4096`
4,096	264.28 ±10.64	516.70 ±2.40	614.59 ±6.18	68.04 ±3.10 †
16,384	210.17 ±25.94	329.79 ±4.48	366.07 ±4.45	crash †

† -ub 4096 does not fit: HSA_STATUS_ERROR_OUT_OF_RESOURCES on host memory, peak GTT 121,585 MiB against the engine's own 104 GiB ceiling (#680). Both reps of the p4096 cell landed at 65-70 t/s and the p16384 cell then died in prompt warmup, so that cell is the failure, not a slow result. On a 96.5 GiB pool 2048 is the top of this lever for this model, and the refusal arrives as a decode error mid-warmup rather than at the guardrail.

What it settles: item 2 of the brief, in direction and roughly in size — 2.3× at 4k, 1.7× at 16k, saturating between 1024 and 2048. What it does not: close the gap. 615 tok/s at 4k is still far from gufo's 1,679 (measured at pp16384, where my best is 366).

What it does not prove, so nobody over-reads it: this is the llama-bench path, not the server the product ships (that path is 6.4× slower on this model, per the ladder above); n=2 per cell; and I do not know which residency regime this window was in — only that all seven cells shared it. The absolute numbers are not board cells. The ratios are the finding.

One correction to my own reasoning, written down before the run: I predicted -ub 4096 would be ~8× at 4k, on the argument that prefill streams the weights once per ubatch (8 passes ≈ 700 GB → 1 pass ≈ 87 GB). The 4×-fewer-passes step returned 2.3×, and the next 2× step returned 1.19×. So prefill cost is not proportional to full-model passes — it saturates. Per-ubatch weight traffic is a real term, not the dominant one at these sizes, and the 8× prediction is refuted by its own test.

Next window: one server arm at -b 2048 -ub 2048, everything else verbatim from the 444.89 row, to see whether the harness gain transfers to the product path.

arm B (the 2.1x/6.5x/3.2x placement lever) cannot reach the product path on this box: it materialises a 26.9 GiB copy of the PLE table

Both attempts below are llama-server on this lane's fork build, product argv, -fa on --cache-reuse 0, gate passed first poll (avail 82 and 107 GiB, gtt_used 5 GiB, coder-llm stopped), and both died in load:

window	config	failure
11:17	`-b 2048 -ub 2048`, `-ot per_layer_token_embd\.weight=HRX0` + `LLAMA_INPUT_LAYER_ON_DEV=1`	`failed to allocate HRX0_HOST buffer of size 29,144,784,384` — 27.14 GiB at t=2:57
11:24	`-b 2048 -ub 512`, same placement	`failed to allocate HRX0_HOST buffer of size 28,886,447,616` — 26.90 GiB at t=2:02

26.90 GiB at -ub 512 versus 27.14 GiB at -ub 2048: the allocation is tensor-sized, not ubatch-sized. It is the 26.8 GiB PLE table duplicated, so lowering the batch size does not help — I expected it would, ran the second window on that assumption, and the log refuted it.

The arithmetic is then simply over budget: 26.8 GiB (PLE) + 59.8 GiB (rest of the weights) + 26.9 GiB (the copy) = 113.5 GiB of system RAM on a 125,136 MiB box. Neither env knob applies: GGML_HRX_GTT_RESERVE_MB polices the GTT pool and the guardrail did not refuse (it printed pool budget 98.3 GiB and let the launch proceed — the failure is a host-typed allocation underneath it), and GGML_HRX_HOST_WEIGHT_LIMIT_MB is a residency cache bound that is off by default (host-memory.cpp:1454-1471), whose own comment says a bound under the working set thrashes.

In llama-bench the identical placement runs fine — r23-A-getrows-thr: 587.9 / 546.5 / 314.9 tok/s at pp512/4096/16384, peak GTT 99,049 MiB, versus 285.6 / 84.1 / 98.6 for default placement at the same dispatch count — because that session had the box to itself. Through llama-server it needs avail >= ~114 GiB, i.e. 93% of the machine.

So the lever that is worth 2.1x / 6.5x / 3.2x in the harness is available to the product only in a few-minute window on an otherwise empty box. Two ways out, both engine-side and neither a tuning flag: (a) do not duplicate the PLE table on the device — it is read by one GET_ROWS per forward; (b) chunk the staging by ubatch so the allocation stops being tensor-sized. (a) is the one that also shrinks the 113.5 GiB, since the duplication is the whole problem.

Next from this lane: one arm at -ub 512 gated at NEED=114 — the condition above, measured rather than assumed — to see whether the transfer lands at all when the box is empty enough to allow it. If it never gets there, that is the product statement: the placement win needs a machine this box can only be for a few minutes at a time.

`-ub 2048` transfers to the product path: 1.63x at pp4096 (167.13 → 272.61), no change at pp512 (1.00x)

Two llama-server loads in one hold, same box state, -b 2048 held fixed so only -ub moves, -fa on --cache-reuse 0, ctx 16384, 5 reps, product argv with argv[0] on this lane's fork build. Gate passed on first poll (avail=103GiB budget=98GiB gtt_used=5GiB), coder-llm stopped for the window, both arms released clean.

cell	`-ub 512`	`-ub 2048`	ratio
pp512 (583)	373.86 (357-396)	374.87 (328-420)	1.00x
pp4096 (3855)	167.13 (149-201)	272.61 (207-392)	1.63x
load time	513 s	1117 s	—
dispatches / graphs	408,158 / 3,418	299,900 / 2,308	—

Why this one is worth trusting where the earlier ladder was not: the two arms share one window, one box state and (measurably, see below) one residency regime, so the 1.63x is a within-window comparison and not two samples of the 5.2x lottery. The -ub 512 arm is byte-for-byte the config that produced 444.89 — and it produced 167.13 today, i.e. the same configuration moved 2.7x between runs while nothing in the argv changed. That is the retraction's point, now with a companion measurement instead of a hypothesis.

Companion measurement, from mincore() on the GGUF rather than RSS (new in this lane, residency.py, 0.73 s per 87.2 GiB pass, known-answer probed): the mmap'd part of this model is the 26.8 GiB PLE table, and it sits at ~0-5% resident for most of the bench in both arms (ub512 median 4.2%, peak 46.9%; ub2048 median 0.8%, peak 47.4% during load then a collapse to ~0%). So both arms ran their prefill with the host GET_ROWS leg paying major faults, which is exactly the mechanism the brief predicted would cap the ub gain — and the gain still came out at 1.63x rather than the harness's 2.3x, in the direction that story predicts.

Two honest caveats:

pp4096 spread in the -ub 2048 arm is 67.9% (206.57 → 256.07 → 364.32 → 391.61 → 272.61 — the first four improve monotonically, then a drop). The 1.63x is a median over 5 reps of a rising series, so the shape is "warm-up within the cell", and a 5-rep median does not fully remove it. The ub512 arm shows no such trend (201 → 196 → 149 → 167 → 150).
pp512 is unchanged at 1.00x, so this lever is a long-prompt lever only. Anyone quoting "ub 2048 is 1.6x faster" without the prompt length is quoting the wrong thing, which is the same mistake as quoting a prefill number without the regime.

For the board: this belongs as a configuration finding, not a row — -b 2048 -ub 2048 on the product path, +63% at 4k for a 2.2x longer load. It does not close the gap to gufo (272.61 at pp4096, and the harness best is 614.59), and per the previous comment the placement lever that would add the rest needs 113.5 GiB of RAM and cannot load through the server on this box.

Best product-path pp4096 to date: 568.75 tok/s — and the same argv gave 272.61 twenty-five minutes earlier

Two runs, identical argv (-fa on --cache-reuse 0 -b 2048 -ub 2048, ctx 16384, product placement, mmap, 5 reps, same gate, coder-llm stopped both times):

window	load	pp512	pp4096	pp4096 spread	residency max
11:59	1117 s	374.87	272.61 (207-392)	67.9%	47.4%
12:24	503 s	289.88	568.75 (500-618)	20.7%	43.6%

2.09x on pp4096 with the same binary, the same flags and the same weights, and 0.77x on pp512 in the other direction. 568.75 is the best prefill this lane has measured through llama-server, it is within 8% of the harness's 614.59 at the same ub, and it is 6.69x the product-default row (85.05) — though that last ratio mixes in -fa and the regime, so I would not quote it as a factor.

The uncomfortable part, which is the real finding: nothing I measured separates the two runs. Load time does not (503 s and 513 s sit either side of 2.1x differences in opposite directions); power and Tctl do not (Tctl 59-66 C in both, pkg_W p95 ~105-112 W in both); and mincore() residency does not (0-4% during both benches, peaks 47.4% vs 43.6% during the respective loads). So this is not "the slow one was reclaimed and the fast one was not" in any way I can currently observe — it is a state I can detect only by measuring prefill twice.

Two consequences I am adopting rather than arguing about:

Sequential arms inside one hold are not safe either. My own -ub 512 vs -ub 2048 comparison (167.13 vs 272.61, 1.63x) ran 20 minutes apart inside a single hold, on the assumption that the regime was stable within a window. Given the swing above, that assumption needs interleaving to hold: next A/B of this model should run ABBA (512, 2048, 2048, 512) so a drift within the window cannot masquerade as an arm effect. The 1.63x direction is almost certainly right — the harness matrix, which was a single process with one mapping, independently showed 2.3x — but the magnitude deserves the ABBA repeat.
Report best-of-window and worst-of-window for this model, never a bare median. 568.75 and 272.61 are the same configuration; a table that shows one number for that config is hiding a 2.1x.

Also: the pinned arm (--load-mode mmap+mlock, 87.2 GiB) was skipped by its own guard at avail=96 GiB < 100, so the pinning question is still open — re-queued at PINNEED=92, which that window's headroom would have passed.

The 90-94% of wall that nothing was timing is `prepare_command_program` — and on this model it is one graph, 413.9 s

Window 16:06-16:28 (~/lb/supreme/raw/llamasrv-q31-prep/ub2048-lmmmap-armA/), one arm, -b/-ub 2048, ctx 16384,
-fa on --cache-reuse 0, q31 quant (unsloth/Qwen3.8-Flash-Next-GGUF@d9620119, UD-IQ4_XS, 4 shards, first load),
pp512/pp4096 x10, GGML_HRX_HOST_GAP_VERBOSE=1 with the prep.* phases from fork llama.cpp#211.

phase	sum	n	worst single graph
wall	659.35 s	4,933	413.92 s
`graph.execute`	655.27	4,933
`replay.total`	181.74	4,933
`replay.wait` (device)	180.04	4,933
`prep.prepare`	473.52	1,644	413.89 s
unaccounted	0.10 s

execute = prep.prepare + replay.total, to 0.1 s over 4,933 graphs. The shape is not what I predicted in this
issue's 13:5x note. 413.89 of those 473.52 s is a single graph — the first one after load, 453 dispatches, whose
replay.wait was 21.8 ms. The device was idle for 414 s while the host built that command program;
command-program-executor.cpp:1118 (kernel_executables->get_or_compile) is the work in it. The other 1,643
preparations total 59.63 s (median 5.7 ms, 2 over 1 s).

So there are two different costs behind one counter:

A per-process kernel-compilation cost of ~414 s on the first graph, inside the 999 s load→ready. First-request
latency, not steady state — but it is 41% of the load time and it is currently unavoidable per process.
~60 s of re-preparation during the run, whose cause is visible in the same log: trim_graph_programs fired
196 times and evicted 682 programs against budget 8192 MiB while the cache held 8.6-9.2 GiB (64 max). The knob
already exists (GGML_HRX_GRAPH_PROGRAM_CACHE_MB, default 8192, 0 = unbounded), so "make the cache hold this model's
hot shapes" is an env-var test, not a kernel change.

Steady-state prefill is device-bound, which also settles what item 3 is about: the bench graphs (1,865 dispatches
each) are 3.13-3.27 s wall with replay.wait 3.00-3.08 s — 96% device — and a 3,856-token prefill is 6 of them
(ttft 18.8 s ≈ 6 × 3.15 s). prep.* does not tax the steady state beyond that 60 s; the routed-expert GEMMs remain the
owner there.

Retraction, because it is mine: at 13:5x I wrote that "843 s of unaccounted time over 2,308 graphs cannot be one-off
preparation unless there were hundreds of distinct shapes". There are hundreds — 1,644 preparations here, one per new
graph shape. The inference was wrong because I had no count to look at; the count now exists.

And the numbers this issue is actually about, restated. Same box, same argv, same binary, old quant vs q31:

window	model	argv	pp512	pp4096	pp4096 spread	graphs	wall
11:38	old	`-ub 512`	373.86	167.13	31.2%	3,418	587.8 s
11:59	old	`-ub 2048`	374.87	272.61	67.9%	2,308	901.3 s
12:32	old	`-ub 2048`	289.88	568.75	20.7%	2,308	481.7 s
16:23	q31	`-ub 2048`	183.76	205.05	19.8%	4,933	659.3 s

q31 is the slowest at pp512, mid at pp4096, and the only stable one (19.8% vs 20-91% spreads). That makes 568.75 the
top of a lottery distribution rather than a level, and it means the honest target from gufo's 1,679 is being comparable
to a distribution, not to a best row. Two candidate causes for q31 being lower, neither measured: the UD 3.1 requant is
slower on the routed-expert path, or the effective ubatch is smaller on q31 (6 graphs for 3,856 tokens ⇒ ~640 tokens per
ubatch rather than 2,048) because the model plus a 3.6 GiB RSS leaves less room for compute buffers — old windows' RSS
was 1.3 GiB. Thermal/power rule nothing out or in: pkg 92.7 W median, Tctl 69.2 °C, GPU 55 °C, equal to the 12:32 window.

Third strike on residency as an explanation: the engram table shard was 0.0% resident for the whole bench (81 samples)
and TOTAL peaked at 24.3%, yet the run reached 205 tok/s.

Next: one arm, -ub 512, everything else verbatim — whether -ub 2048 is still worth anything on the quant we now have,
and a second q31 measurement to say whether ~205 is a level. Then the device profile at the best ub.

Item 3 (dispatch profile at the best ub) is answered — and it needs no backend change. The stack can time individual dispatches; I had recorded otherwise on 2026-11-05 17:4x because I looked in ggml-hrx and not in the runtime it sits on. Recorded here so nobody re-derives it:

HRX_PROFILE_FILE=<bundle.ireeprof> HRX_PROFILE_MODE=dispatch llama-server ...
iree-profile summary|statistics|dispatch|export --format=ireeperf-jsonl <bundle.ireeprof>

the profiler is libhrx's own IREE HAL profiling, wired at device creation — hrx-system/libhrx/src/libhrx/runtime.c:589 (HRX_PROFILE_FILE), :646 (HRX_PROFILE_MODE = queue|dispatch|executable|all), :1124 (hrx_device_profile_begin inside hrx_gpu_initialize), so ggml-hrx needs no cooperation. MODE=dispatch = EXECUTABLE_METADATA | DEVICE_QUEUE_EVENTS | DISPATCH_EVENTS = device-timestamped per-dispatch records with kernel identity;
the reader is hrx-system: bazel build //tools:iree-profile, binary at bazel-bin/runtime/src/iree/tools/iree-profile/iree-profile.

Measurement. Product path (llama-server, this fork's build), q31 = UNSLOTH UD 3.1 UD-IQ4_XS (4 shards), -ub 2048 -b 2048, ctx 16384, -fa on --cache-reuse 0, mmap, one pp512 request (the pp4096 request could not be profiled — see the last section). Bundle: 65,484 valid dispatches, 669 executables, 18.03 s of device dispatch time, 38.95 s of in-graph wall. Raw: ~/lb/supreme/raw/llamasrv-q31-prof/ub2048-lmmmap-armA/ (bundle 164 MB, prof-{summary,statistics,dispatch}.txt, prof-by-kernel.json).

Whole bundle, straight out of iree-profile statistics (dispatch_function_total=18031.669 ms):

kernel	dispatches	device time	share	avg
`ggml_mul_mat_id_swiglu_f32_f32_wmma` (routed-expert gate/up, fused swiglu)	1,300	8,004.8 ms	44.4%	6.16 ms
`hrx_dp_mm_q6_k_m128_w8_f2_sb`	376	2,065.3 ms	11.5%	5.49 ms
`ggml_mul_mat_id_f32_f32_wmma` (routed-expert down)	272	1,257.5 ms	7.0%	4.62 ms
`ggml_mul_mat_f32_f32_wmma` (dense f32 projections)	1,851	974.6 ms	5.4%	0.53 ms
`ggml_flash_attention_f32_f16_wmma`	348	653.1 ms	3.6%	1.88 ms
everything else (59 kernels)	61,337	5,076.2 ms	28.1%	0.08 ms

Restricted to the 21 prefill launches only (12.51 s of that 18.03 s; a launch is profiled as submission_id), the shape is the same and sharper:

kernel	share of prefill device time
`ggml_mul_mat_id_swiglu_f32_f32_wmma`	45.8%
`hrx_dp_mm_q6_k_m128_w8_f2_sb`	13.4%
`ggml_mul_mat_id_f32_f32_wmma`	7.9%
`ggml_mul_mat_f32_f32_wmma`	5.9%
`hrx_dp_mm_q8_0_{m64,m128}`	6.7%
`ggml_flash_attention_f32_f16_wmma`	2.7%

Per launch the fused routed-expert swiglu kernel is 36.7-60.1% (median 46.7%) of device time — never below a third in any of the 21. So ~60-66% of prefill device time is the routed-expert MoE GEMMs, the shapes item 4 named, and the gate/up side dominates: the expert-down matmul is 7.9% where the gate/up is 45.8%. Attention is 2.7% at this context length (9% in the longest phase measured).

Retraction — an earlier version of this comment claimed "the GPU is idle 53.7% of in-graph time" with a stall table; that was my bug, not the backend's. I grouped dispatch events by command_buffer_id and treated that as one graph launch. It is not: a command buffer is the prepared program, and the executor replays one program several times per graph (host-gap counts 1,865 dispatches for a "graph" that the profiler sees as 2 submissions of ~900). Grouping by submission_id instead: 299 launches, 18.03 s busy vs 18.26 s wall → in-graph idle 1.2%. The GPU is essentially never idle inside a launch. The earlier stall numbers and the hip_top_k_f32/copy_strided "host round-trip" reading are withdrawn; that hypothesis has no evidence behind it and this bundle cannot support it.

One thing I could not measure, stated so it is not mistaken for a null result. pp4096 aborted with RESOURCE_EXHAUSTED; dispatch profiling event ring exhausted: requested 169 events, available 52, ready 65484, capacity 65536 — the ring is a compile-time constant (IREE_HAL_AMDGPU_HOST_QUEUE_PROFILE_DISPATCH_EVENT_CAPACITY, runtime/src/iree/hal/drivers/amdgpu/host_queue_profile_event.c) with no env knob, and the model load + warmup alone already spends ~46k of the 65,536 slots, so a 16k-token cell cannot be profiled in the same session as a load. That is a hrx-system PR away (raise it, or drain at graph boundaries); the numbers above are the pp512 + load mix and I have not claimed them for pp4096.

Caveats that matter for reading the shares. (1) The profiler costs throughput: the same pp512 cell is 99.32 tok/s profiled vs 144.33 unprofiled in the window before it (−31%), so shares are shape-correct but levels are not. (2) Same name ≠ same kernel: ggml_mul_mat_id_swiglu_f32_f32_wmma is at least four distinct executables with per-dispatch averages from 1.3 ms to 58 ms — different quantized weight types and token geometries — so an attribution by kernel name is not enough to write a replacement against; the executable id and workgroup shape have to come with it. (3) iree-profile export --format=ireeperf-jsonl on this bundle emits 130,968 dispatch_event rows for 65,484 events (a second, all-null chunk follows the real one) — anyone re-deriving numbers from the JSONL must filter on non-null submission_id, which is also how I found my own grouping error.

Tracked in ~/briefs/TFLASHNEXT-PREFILL-STATE.md; the harness around it is ~/lb/fnpre/i21-queue.sh + i15-inner.sh (HRXPROF=1).

Re-measured with the new _sb-off default (engine #1248, gpu-kernels #48). The suspected lever is not this issue's gap:

On the issue's own payload binary (Nov 2, pre-fork#177, old default), same model (Qwen3.8-Flash-Next-UD-IQ4_XS), -dev HRX0 -ngl 99 -fa 1 -p 512,4096 -n 0, ABBA:

arm	pp512	pp4096
payload, old default (`_sb` on)	224.53 / 222.72	172.73 / 200.48
payload, `M128_SB=0` (the new default)	227.19 / 224.89 (~+1%)	193.06 / 204.76 (A-side spread 16%, no clean separation)
fork-tip build, new default	414.84	346.59
fork-tip build, `M128_SB=1`	418.70	375.93

The fork-build numbers are not attributable from this ticket: an interleaved same-binary check caught the issue's regime lottery — payload 223.74, fork 418.99, fork 236.35 pp512 back-to-back in one hold (same binary moved 1.77x between consecutive arms).

Attribution

On the issue's own binary the tile-default lever is ~1 % at pp512, which fits its dispatch profile (comment 27, product path, ub 2048): routed-expert MoE GEMMs are 45.8 % of prefill device time and no m128_sb default touches them; expert-down 7.9 %; f32 dense 5.9 %; q8_0 dense 6.7 %; attention 2.7 %. The q6_k dense _sb tile the flip removes is 13.4 %, so even 3.3x on that slice caps the total near 9 %. The 287 tok/s gap is therefore not the spill: it is the expert GEMM path plus the host GET_ROWS/residency regime this issue's later comments describe. Fork #177 (threaded GET_ROWS) is already in the engine via merged #1233 — no pin action there.

On the reference: gufo's 1,679 tok/s is pp16384 with WMMA quant GEMMs / dequant-to-BF16 hipBLASLt at 4096/8192 batches — a different shape and kernel class from our pp4096. Public targets are context, not proof.

Disposition: parked on roadmap work. The engine side of this issue is exhausted by #177/#1248; closing the remaining gap needs a new expert-GEMM/prefill-batching kernel program (the 45.8 % slice), which is a roadmap decision, and any future perf row here should be measured interleaved, same-binary, reported as best/worst (the regime lottery above). Unpark: expert-GEMM kernel work or prefill batching at the MoE shapes.

Scoped PR opened: engine #1269 (docs), and what remains on this issue

Rule-3 treatment for a roadmap issue: a scoped first PR plus this list, rather than a false Fixes.

#1269 (open, 2cead0a5) corrects the prefill record in docs/hrx.md where it had gone stale, including a
sentence that said "a bigger ubatch buys nothing on this model anyway" — the measurement says -ub 2048 is 2.03x
-ub 512 at pp4096 (205.05 vs 101.05 tok/s, one library, two windows). It also records the flag ladder that actually
moves this file through llama-server: product defaults 85.05 → -fa on 153.71 → -fa on --cache-reuse 0
444.89 tok/s at pp4096, with the load dropping 687 s → 439 s (under the front-end's 600 s budget), and the current
pp4096 device attribution (routed gate/up 39.6 %, q6_k m128 11.2 %, attention 11.2 %, from the 2026-11-05T23:15
dispatch bundle).

Against the issue's four asks:

dispatch profile of pp512/pp4096 with owners and CPU splits — done (2026-11-05T23:16 comment for the method;
the pp4096 table is in docs/hrx: the Flash-Next prefill ladder, and a ubatch claim the measurements contradict (#572) #1269). The method needed no backend change: HRX_PROFILE_MODE=dispatch + iree-profile.
expert-GEMM tile width fitted to 512x640x2560 — not the lever, and now measured as such. On identical geometry
the model's IQ3_XXS expert blocks cost 2.4x the IQ4_XS blocks (79.94 vs 33.34 ms per dispatch at 10x639x1), a
weight-format effect, not a tile effect: the same kernel compiled twice differs by 4x in vector ALU and by nothing
else that matters. The kernel-side consequence is being handled outside this repo.
the ub cap — lifted and now measured: 2.03x (-ub 2048 vs 512). -ub 4096 does not fit this 87 GiB file.
an MTP head on HRX for the decode side — not started; no MTP head exists in this tree for this model.

What still stands between us and the 1,679 row: the gap itself (the best product-path pp4096 I can defend is
568.75 tok/s, with a documented 272.61 for identical argv — this file's variance is real), the MTP head, and kernel
work that belongs in the private/fork repos rather than in engine text.

docs/hrx: the Flash-Next prefill ladder, and a ubatch claim the measu…

Gap re-measured: pp4096 211.41 tok/s — and the reason a single number is not defensible

Fresh window tonight (knob on, -fa on --cache-reuse 0 -b/-ub 2048, product placement, q31, ctx 16384, 3 reps):
pp4096 211.41 tok/s (197.37–242.00, 21 % spread), pp512 201.30 (180.32–207.94). PPL 8×512 with the knob on:
1.7198 ± 0.05749 — the third independent window giving exactly that value.

Measured against the 1,679 target, the same arm on the same day:

window	pp4096	gap
02:12	318.52	5.27x
03:26	182.10	9.22x
22:49	211.41	7.94x
best product number ever recorded here, pre-change (11-05)	568.75	2.95x

So the gap is 3.0x-7.9x, and the spread is our own: 568.75 vs 272.61 for a byte-identical argv 25 minutes apart,
and 182-318 across today's three windows of the same configuration. That variance (2-3x) is now larger than any kernel
lever I have left, so it is the next thing I am measuring rather than another 5 % edit.

Covariates ruled out from tonight's logs, so nobody re-derives them: JIT cache warmth (2 compiles in the slow window vs
637 in the fast one — and the warm one was slower, matching the 2026-11-05 18:33 note), load time (1053 s vs 539 s),
residency sampling (44.5 % peak in the slow window vs 60-69 % in faster ones — no monotone relation), host-gap shape
(identical: 1,852 graphs / 204,450 dispatches). Remaining candidates: page-cache/mmap state and co-tenant GTT
pressure. A window is queued that tests the first directly (same config, one arm re-reads the model shards into page
cache immediately before benching).

Status of the issue's four asks: (1) dispatch attribution — done, and now in docs/hrx.md via the merged #1269;
(2) expert-GEMM tile width — measured out as not the lever (the cost inside the routed-expert family was a weight-format
effect, not a tile effect); (3) the ub cap — lifted, and -ub 2048 vs -ub 512 is 2.03x at pp4096, also in #1269 (which
also corrects the stale sentence claiming a bigger ubatch buys nothing); (4) MTP head — not started, nothing in this
tree for this model.

One kernel-side change from this work is merged on the fork and off by default
(GGML_HRX_IQ3S_TABLE_DECODE, fork #216): in-window it moves pp4096 by 1.18x and 1.29x across two independent holds, at
identical PPL. It is not in the product yet because the engine pin predates that merge — that pin move is an owner call,
and I have flagged it.

docs/hrx: document GGML_HRX_IQ3S_TABLE_DECODE, which is in the pin bu…

Ask (4), the MTP head: scoped against the tree, invocation found, harness staged (no measurement yet)

The one ask on this issue that had not been started is now scoped, and the first thing worth recording is that it needs no upstream work — the support and the weights are both already here:

LLAMA_CONTEXT_TYPE_MTP exists (src/llama-context.cpp, src/llama-model.cpp, common/speculative.cpp,
tools/server/server-context.cpp), i.e. the pinned fork already has the MTP context type.
The type name is draft-mtp (common/speculative.cpp:37), selected with --spec-type draft-mtp, and the sidecar
goes through --spec-draft-model (LLAMA_ARG_SPEC_DRAFT_MODEL).
The head is on disk and is not small: ~/models/flash-next/MTP/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf, 1.91 GB.
Both flags parse on the newly pinned binary (092eb9dbc); checked without a model load.

Harness staged: a third arm in the window script (arm M) that is arm A's product placement plus exactly those two
flags — same binary, same argv, one variable — measuring the decode cells tg128@d0 and tg128@8k (the harness's
existing grammar) against arm A in the same hold. The number to beat is this issue's own row: decode 24.0 tok/s on
HRX for this file, against 33 for Vulkan+llama.cpp with MTP.

Not measured, and why: every arm here needs an 87.2 GiB load, and the box is currently at 44 GiB available with
/proc/pressure/io full avg10 ~81 % — the same condition that made last night's load windows unusable (and which
#1339/#231 has since fixed on the load path, so the next window should behave differently). Queued behind a warm
cache and a quiet box, not another blind hold.

What I will not do: quote an MTP number from a window that is thrashing, or claim the head works because the flags
parse. The acceptance for ask (4) is a paired tg row plus the drafter's acceptance rate, in one hold.

Navigation Menu

Repository navigation

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572

14 remaining items

The server-path row exists at last — and it disagrees with every number in this issue

1. tg is measurable for this model — on this build, and my earlier "gen path aborts" was the pin, not the model

2. The server path does not show the gains this issue reports — do not mix the two paths on a scoreboard

3. The shipped pin cannot load this model at all, and the front-end's readiness budget is shorter than the load

The 5.2× is two default flags, not a kernel: `-fa` off and `--cache-reuse 256`

Retraction, 90 minutes after posting the ladder: `--cache-reuse` was already off in every run, so it cannot be the 2.9×

What the timing data actually points at: whether the host-weight cache survives the load

Next three measurements, in this order

First controlled lever since the retraction: `-ub 2048` is 2.3× at pp4096 (264 → 615 tok/s), and `-ub 4096` does not fit

arm B (the 2.1x/6.5x/3.2x placement lever) cannot reach the product path on this box: it materialises a 26.9 GiB copy of the PLE table

`-ub 2048` transfers to the product path: 1.63x at pp4096 (167.13 → 272.61), no change at pp512 (1.00x)

Best product-path pp4096 to date: 568.75 tok/s — and the same argv gave 272.61 twenty-five minutes earlier

The 90-94% of wall that nothing was timing is `prepare_command_program` — and on this model it is one graph, 413.9 s

Attribution

Scoped PR opened: engine #1269 (docs), and what remains on this issue

Gap re-measured: pp4096 211.41 tok/s — and the reason a single number is not defensible

Ask (4), the MTP head: scoped against the tree, invocation found, harness staged (no measurement yet)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572

Description

Activity

bong-water-water-bong commented on Oct 28, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

bong-water-water-bong commented on Oct 30, 2026

14 remaining items

bong-water-water-bong commented on Nov 5, 2026

The server-path row exists at last — and it disagrees with every number in this issue

1. tg is measurable for this model — on this build, and my earlier "gen path aborts" was the pin, not the model

2. The server path does not show the gains this issue reports — do not mix the two paths on a scoreboard

3. The shipped pin cannot load this model at all, and the front-end's readiness budget is shorter than the load

bong-water-water-bong commented on Nov 5, 2026

The 5.2× is two default flags, not a kernel: -fa off and --cache-reuse 256

bong-water-water-bong commented on Nov 5, 2026

Retraction, 90 minutes after posting the ladder: --cache-reuse was already off in every run, so it cannot be the 2.9×

What the timing data actually points at: whether the host-weight cache survives the load

Next three measurements, in this order

bong-water-water-bong commented on Nov 5, 2026

First controlled lever since the retraction: -ub 2048 is 2.3× at pp4096 (264 → 615 tok/s), and -ub 4096 does not fit

bong-water-water-bong commented on Nov 5, 2026

arm B (the 2.1x/6.5x/3.2x placement lever) cannot reach the product path on this box: it materialises a 26.9 GiB copy of the PLE table

bong-water-water-bong commented on Nov 5, 2026

-ub 2048 transfers to the product path: 1.63x at pp4096 (167.13 → 272.61), no change at pp512 (1.00x)

bong-water-water-bong commented on Nov 5, 2026

Best product-path pp4096 to date: 568.75 tok/s — and the same argv gave 272.61 twenty-five minutes earlier

bong-water-water-bong commented on Nov 5, 2026

The 90-94% of wall that nothing was timing is prepare_command_program — and on this model it is one graph, 413.9 s

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 6, 2026

Attribution

bong-water-water-bong commented on Nov 6, 2026

Scoped PR opened: engine #1269 (docs), and what remains on this issue

bong-water-water-bong commented on Nov 7, 2026

Gap re-measured: pp4096 211.41 tok/s — and the reason a single number is not defensible

bong-water-water-bong commented on Nov 7, 2026

Ask (4), the MTP head: scoped against the tree, invocation found, harness staged (no measurement yet)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The 5.2× is two default flags, not a kernel: `-fa` off and `--cache-reuse 256`

Retraction, 90 minutes after posting the ladder: `--cache-reuse` was already off in every run, so it cannot be the 2.9×

First controlled lever since the retraction: `-ub 2048` is 2.3× at pp4096 (264 → 615 tok/s), and `-ub 4096` does not fit

`-ub 2048` transfers to the product path: 1.63x at pp4096 (167.13 → 272.61), no change at pp512 (1.00x)

The 90-94% of wall that nothing was timing is `prepare_command_program` — and on this model it is one graph, 413.9 s