hrx: the fused-swiglu prefill tile ignores GGML_HRX_HIP_DENSE_PREFILL_M128_SB, and its plain tile is already 2.86x faster (26 % of pp4096 FLOPs)

What

swiglu_tile() in addons/hip-hrx/dispatch-hip-dense-prefill.cpp (gpu-kernels) returns Tile{128,8,1,true} for every
fused FFN gate/up dispatch at ≥256 tokens and never consults m128_sb(). So the _sb ("B lane-half pairs") tile is
forced for the largest GEMM in a transformer block, and the knob that turns the spilled _sb tiles off —
GGML_HRX_HIP_DENSE_PREFILL_M128_SB, the behaviour gpu-kernels#48 makes the default — has no effect on it at all.

Under the clang 24.0.0git toolchain this box builds with, every _sb instantiation carries 320-576 B/lane of scratch
(next to 0 under clang 23; measured by the release lane in engine#782), and it shows up directly:

kernel, Qwen3.8-27B UD-Q4_K_XL `blk.24.ffn_gate`+`up` (Q5_K, K=5120, N=17408, T=4096)	ms	TFLOP/s
`hrx_dp_swiglu_q5_k_m128_w8_f1_sb` — what the engine runs today	201.19	7.44
`hrx_dp_swiglu_q5_k_m128_w8_f1` — already exported by the same build	70.34	20.76

2.86x, A-side spread 2.1 %, bad=0, max_err=4.44e-04 identical on both tiles (so the same arithmetic, not a
cheaper one). Measured with addons/hip-hrx/tools/dense-prefill-bench, A B B A per shape, against its own CPU reference.

Why it matters

In this model the fused gate/up path is 26.1 % of M=4096 prefill FLOPs (58.4 of 223.8 TFLOP over every .weight
tensor; 65 layers, of which 40 have gate and up of the same type — which is what the fused kernel requires — and 25 run as
two plain mm dispatches instead). So this is worth roughly what #48 is worth, for one dispatch line, and it is the reason
a swiglu-heavy model still shows the full spill with #48 merged.

Proposed change

Make the call site consult the same predicate as mm_tile():

Tile swiglu_tile(int64_t tokens, ggml_type type) {
    const bool sb = m128_sb(type);            // honours GGML_HRX_HIP_DENSE_PREFILL_M128_SB
    return tokens >= 256 ? Tile{ 128, 8, 1, sb } : Tile{ 64, 8, 1, sb };
}

Two things to check before writing it: the plain m128_w8_f1 swiglu tile must exist for every format the fused
matcher admits (verified here only for q5_k; the shipped libggml-hrx string table also carries
hrx_dp_swiglu_q4_k_m128_w8_f1 and hrx_dp_swiglu_q6_k_*, but that is a name list, not a proof of a working kernel), and
the m64_w8_f1 (decode-side, <256 tokens) branch stays as is unless it is measured.

Acceptance bar

dense-prefill-bench on the model's own gate/up tensors: plain tile bad=0 and faster than _sb, per format.
Identity: the same logprob/rounds harness used for the P4 gate (7 prompts × 197 tokens × 101 values, bit-identical), so
"fast but wrong" cannot pass.

Not measured here, do not promise it

mm_tile()'s K >= 8192 branch is the other forced-sb site ({64,8,2,true}: ffn_down in this GGUF's IQ4_XS
layers, 5.2 % of prefill FLOPs). Its non-sb alternative has not been benched — that needs m64_w8_f2 checked for
existence and speed on that shape first.

Evidence dir ~/supreme-npu/tile-kernel-20261104T230313/, harness ~/supreme-npu/tile-kernel-ab.sh,
summary ~/supreme-npu/tile-kernel-ab-summary.py. Served-cell context (the +30.5 % at pp4096 from #48's default flip):
engine#782 comment 6647562573.

Correction to the "not measured" section: the `K >= 8192` branch cannot be un-spilled by a dispatch line — the plain tile does not exist

I wrote above that mm_tile()'s K >= 8192 branch (returning {64,8,2,true}) needs "m64_w8_f2 checked for existence
and speed". It does not exist, for any format, so that framing was wrong:

$ for n in hrx_dp_mm_{iq4_xs,q5_k,q6_k,q4_k}_m64_w8_f2 hrx_dp_mm_iq4_xs_m64_w8_f2_sb; do
    printf '%-36s %s\n' "$n" "$(strings -a $PKG/hrx/llama/bin/libggml-hrx.so.0.18.0 | grep -cx "$n")"; done
hrx_dp_mm_iq4_xs_m64_w8_f2           0
hrx_dp_mm_q5_k_m64_w8_f2             0
hrx_dp_mm_q6_k_m64_w8_f2             0
hrx_dp_mm_q4_k_m64_w8_f2             0
hrx_dp_mm_iq4_xs_m64_w8_f2_sb        3

and the source says why — kernels/hrx_dense_prefill.hip, DP_INST (line 861) instantiates exactly one 64-row w8_f2
tile, and it is _sb:

DP_MM(hrx_dp_mm_##fname##_m64_w8_f2_sb, FMT, 64, 8, 2, true)     \

So the two call sites are not the same kind of problem, and my earlier "both need a kernel change" (then "one is a
dispatch line") lands here:

call site	share of M=4096 FLOPs	what closing it needs
fused FFN gate/up (`swiglu_tile`)	26.1 %	dispatch line — the plain `m128_w8_f1` is already built for every format; PR #51
`mm_tile()`'s `K >= 8192` branch (IQ4_XS `ffn_down`)	5.2 %	kernel work — a plain 64-row `w8_f2` instancing has to be built and measured, or the branch has to move to a tile that exists (`m64_w4_f2` is plain but takes 4 waves / 2 fragments; `m128_w8_f2` exists plain but `m128_fits()` deliberately rejects IQ4_XS at `K >= 8192` because it measured slower there)

I am not proposing a change for that branch: it is arithmetic nobody has measured on a shape the dispatcher already
decided to avoid, and the honest next step is a bench of the three candidates on an IQ4_XS K=17408 tensor, which is a
device claim rather than a code change. Leaving it open here rather than guessing.

Measured: the `K >= 8192` branch sends IQ4_XS to the worst of the three existing tiles — 2.99x off the best, and `m128_fits()`'s exclusion is only right at small batches

Follow-up to the correction above, where I found there is no plain m64_w8_f2 to switch to. That is true, but it is not
the end of the question: two other plain tiles exist for IQ4_XS, and one of them is 3x faster than what the dispatcher
forces. dense-prefill-bench on blk.0.ffn_down (IQ4_XS, K=17408, N=5120), order A B C C B A, all runs against the
bench's own CPU reference (bad=0, spreads 0.2-3.1 %):

tile	T=1024	T=4096
`hrx_dp_mm_iq4_xs_m128_w8_f2` (plain)	5.90 ms / 30.94 TF-s	25.06 ms / 29.13 TF-s
`hrx_dp_mm_iq4_xs_m64_w4_f2` (plain)	6.90 ms / 26.47	33.82 ms / 21.59
`hrx_dp_mm_iq4_xs_m64_w8_f2_sb` — what `mm_tile()` forces at K ≥ 8192	17.77 ms / 10.27	74.83 ms / 9.76

So the forced tile is 3.01x slower at T=1024 and 2.99x slower at T=4096, and the best tile is the one
m128_fits() explicitly refuses for this type and K:

// dispatch-hip-dense-prefill.cpp, m128_fits()
return m128_mode() == 1 && N >= 1024 && workgroups >= 16 && !(type == GGML_TYPE_IQ4_XS && K >= 8192);

The exclusion is not wrong, it is unbounded

Its own comment records why it exists — "IQ4_XS at K = 17408 (vs the m64_w8_f2_sb tile, 1.50 vs 1.39 ms at 256 tokens,
3.01 vs 2.79 ms at 512)" — i.e. it was tuned where m128 lost by 6-8 % at 256-512 tokens. At 1024 and 4096 it wins by
3x. So the rule needs a token condition, not a type exclusion: the shape where m128 loses is small-batch, and this is
the same shape-dependence that makes the prefill cell and the decode cell different animals.

What I would do about it, and what it is worth

A one-condition change (&& !(type == GGML_TYPE_IQ4_XS && K >= 8192 && tokens < <threshold>)) with the threshold measured
somewhere between 512 and 1024, which is a dispatch line with an existing kernel — the same shape as gpu-kernels#51, not
new kernel work as I wrote above. It matters in this model at 5.2 % of M=4096 prefill FLOPs, and it matters more in the
IQ4_XS-heavy containers, where ffn_down is the second-largest GEMM in every layer.

I have not made the change and am not claiming a served-cell gain from it: the threshold is unmeasured on both sides of
1024 (255/256 and 2048 are the interesting points), and this lane is mid-way through a different port tonight. Numbers,
harness and the exact line are here so whoever takes it does not repeat the two hours I just spent deciding whether the
tile existed at all.

Evidence: ~/supreme-npu/tile-cand-20261105T010153/ (runs.log + summary.txt), harness
~/supreme-npu/tile-candidates-ab.sh (A B C C B A, CPU-reference checked, --check-discovery host-only).

gpu-kernels: the fused-swiglu prefill tile follows m128_sb(), with pe…

Resolved. The kernel fix landed as 1bit-MONSTER/gpu-kernels#59 (merged, pin 94bc42e55c60) and the engine pin moved to it in #1246 (merged e2005858).

swiglu_tile(type, tokens) now takes const bool sb = m128_sb(type) for >= 256 tokens exactly as mm_tile() does, so GGML_HRX_HIP_DENSE_PREFILL_M128_SB finally has an effect on the fused FFN gate/up dispatch; register_kernels() registers the plain 128-row swiglu tile; < 256 stays on m64_w8_f1_sb (no plain m64 build exists).

Device evidence on strixhalo (gfx1151, dense-prefill-bench, A B B A, T=4096, real GGUFs, new code objects): q4_k 2.84x, q5_k 2.71x, q6_k 2.16x, iq4_xs 2.61x, q8_0 2.77x, iq4_nl 2.46x, q3_k 2.12x, iq3_s 2.31x — bad=0 and identical max_err in every row, with f32+f16 whole-output hashes identical (bit-identical on every T×N value). The add-on's own tools/check.sh gfx1151 shows 409 kernel ids matching the manifest, 45 objects ok, and probes PASS at both knob settings (only the pre-existing hyperconn/lightning-indexer fingerprint failures, engine#52).

Scope, stated so it cannot be overclaimed: the shipped default M128_SB=1 still selects _sb for q4_k/q5_k/q6_k — the 2.1–2.8x FFN win needs the knob off (or gpu-kernels#48's default flip, which is a separate decision). What this issue asked for is that the knob has an effect on this tile, and it now does. Logs: strixhalo ~/wt/tile1064-bench-20261106T024951. Closing.

Navigation Menu

Repository navigation

hrx: the fused-swiglu prefill tile ignores GGML_HRX_HIP_DENSE_PREFILL_M128_SB, and its plain tile is already 2.86x faster (26 % of pp4096 FLOPs) #1064

What

Why it matters

Proposed change

Acceptance bar

Not measured here, do not promise it

Correction to the "not measured" section: the `K >= 8192` branch cannot be un-spilled by a dispatch line — the plain tile does not exist

Measured: the `K >= 8192` branch sends IQ4_XS to the worst of the three existing tiles — 2.99x off the best, and `m128_fits()`'s exclusion is only right at small batches

The exclusion is not wrong, it is unbounded

What I would do about it, and what it is worth

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: the fused-swiglu prefill tile ignores GGML_HRX_HIP_DENSE_PREFILL_M128_SB, and its plain tile is already 2.86x faster (26 % of pp4096 FLOPs) #1064

Description

What

Why it matters

Proposed change

Acceptance bar

Not measured here, do not promise it

Activity

bong-water-water-bong commented on Nov 5, 2026

Correction to the "not measured" section: the K >= 8192 branch cannot be un-spilled by a dispatch line — the plain tile does not exist

bong-water-water-bong commented on Nov 5, 2026

Measured: the K >= 8192 branch sends IQ4_XS to the worst of the three existing tiles — 2.99x off the best, and m128_fits()'s exclusion is only right at small batches

The exclusion is not wrong, it is unbounded

What I would do about it, and what it is worth

bong-water-water-bong commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Correction to the "not measured" section: the `K >= 8192` branch cannot be un-spilled by a dispatch line — the plain tile does not exist

Measured: the `K >= 8192` branch sends IQ4_XS to the worst of the three existing tiles — 2.99x off the best, and `m128_fits()`'s exclusion is only right at small batches