hrx: a separate MTP draft head (-md) is rejected for qwen4exp on our route, while a cut-vocab head gives +55% decode on the fork

Summary

Our HRX route cannot attach a separate MTP draft head (-md) to a Qwen3.8-Flash-Next (qwen4exp) model.
Both entry points fail during draft setup:

llama-server -md <head.gguf> --spec-type draft-mtp
  -> qwen4exp requires ctx_other to be set                       (while measuring the draft)
  -> device 'HRX0' does not have support for op RESHAPE needed for sampler 'top-k'

1bit serve ... --mtp   -> fails the same way

The stock UD-IQ3_XXS GGUF carries no in-file MTP head, so on this file the house route can only be
measured without drafting. The cost of that is large and measured elsewhere in the same hour: on a gfx1151
llama.cpp fork holding the same GGUF, a cut-vocab draft head at n=2 gives +55 % decode.

What drafting is worth, measured on the fork (same GGUF, same hour)

Protocol: 512 greedy tokens (ignore_eos), median of 3, short 216-token / long 18,782-token prompt,
-c 32768 --parallel 1, -fa on -ctk q8_0 -ctv q4_0, one gpu-hold claim per arm.

arm	short tg	long tg (18.8k)	draft acceptance	GTT peak
fork, no MTP	28.95	31.15	—	50.4 GiB
fork + its cut-vocab FR65k head, n=2	44.86	47.87	309/403 = 77 %	52.2 GiB
fork + our `…Q4_K_M-dv65k` head, n=2 (round 1)	39.28	n/a	117/236 = 50 %	55.8 GiB

Two facts from that table, both actionable:

+55 % decode from drafting at n=2 (31.15 → 47.87 on the long prompt), and the peak GTT is lower with
the draft (52.2 vs 50.4 GiB is within the box's noise; it did not grow).
The head matters more than the mechanism: a cut-vocab head accepts 77 % against our general head's
50 %, and the cut head is ~4 t/s/52 % better in the two rows measured (39.28 → 44.86 short).

Why our route cannot use this today — three separate blockers

No draft support for qwen4exp on the HRX route. -md dies on ctx_other and then on a missing
RESHAPE for the top-k sampler on HRX0; 1bit serve --mtp fails identically. This is the build gap
this issue is about.
The two engines cannot share one head file. The fork's FR65k head uses a quant type the house binary
does not have (type 43 → block size (0)), and the fork rejects our …Q4_K_M-dv65k.gguf with
check_tensor_dims: tensor 'output.weight' not found. So a head must be built per engine, not shared —
worth knowing before anyone plans a "one head everywhere" step.
No in-file head in this file, so there is nothing to fall back on: the 40–49 tok/s "Flash-Next MTP n3"
rows in our notes came from a file that carried an in-file head, and are not reproducible from
UD-IQ3_XXS.

Smallest useful change

Make the separate-head path work for qwen4exp on HRX at n=2 (the fork's configuration), with the errors
above as the acceptance tests:

ctx_other set for the draft when a separate head is attached (the fork's -md path does it; the house
binary measures the draft without it).
A RESHAPE (or an equivalent) that the top-k sampler needs, on HRX0 — today the op is declined and the
server refuses to come up rather than falling back.
Re-measure with the same protocol and report decode short/long + acceptance; the target to beat is
47.87 tok/s at 18.8k and 44.86 at short context, and the bar to clear on acceptance is 77 %.

A cut-vocab head builder is not part of this issue: it is a separate, larger piece (it changes the vocab
of the draft model and needs its own verification). This issue is only "the route can attach a head at all".

Reproduce

llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
    --device HRX0 -ngl 99 --jinja -c 32768 -np 1 --cache-reuse 256 --no-strict-device \
    -md <draft-head.gguf> --spec-type draft-mtp
# -> qwen4exp requires ctx_other to be set
# -> device 'HRX0' does not have support for op RESHAPE needed for sampler 'top-k'

Arm logs from the fork side (bench.json, per-arm server logs, acceptance counts) are under ~/lb/aval/*/
on strixhalo; the head used there is ~/models/qwen38/avalon-mtp (1.38 GiB, third-party, fetched for the
comparison and deletable).

HRX: Qwen3.8-Flash-Next prefill is 287 tok/s on UD-IQ4_XS vs 1,679 for gufo on the same box (and 500-2,400 across published Strix Halo engines) #572 — Flash-Next prefill on the HRX route (different file and mechanism).
hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679 — the long-context decode gap on this same file (9.70 tok/s at 18.8k vs 31.15 on the fork).
hrx: gfxhub page fault + MODE2 GPU reset when GTT nears the 120 GiB ceiling (Flash-Next UD-IQ3_XXS, 18.8k ctx, 106.5 GiB GTT) #680 — the two [gfxhub] page fault + MODE2 GPU reset bursts from the same arm.
The long-context decode gap on the same file (9.70 tok/s at 18.8k vs 31.15 on the fork) is filed
as hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679; drafting would narrow it, but it is not the cause.

State at tonight's pin (origin/main 23d6972d9, fork d8bd0ff6052a) — this issue has had no comment since it was filed
on 10-31 and no merged PR touches it. Re-checked from source rather than from memory:

Blocker 1 is unchanged, and it is total

The engine has no concept of ctx_other at all: git grep -c ctx_other origin/main → 0 hits in the whole tree, so
nothing measures the draft with it set. The failure the issue quotes is thrown by the fork at
src/llama-context.cpp:147,156:

throw std::runtime_error(model.arch_name() + " requires ctx_other to be set (this warning is normal during memory fitting)");

and app/serve.cpp:1101 still inserts exactly the argv the issue exercised:

if (!o.mtp.empty()) argv.insert(argv.end(), {"--spec-type", "draft-mtp", "-md", o.mtp, "-ngld", "99"});

So -md for qwen4exp on HRX is dead at the same place tonight as on 10-31. Still reproduces.

Blocker 2 may have changed, and only one arm can say

The second error (device 'HRX0' does not have support for op RESHAPE needed for sampler 'top-k') came from
src/llama-sampler.cpp:618's per-node support probe of the sampler's own candidate graph. That is exactly the surface that
moved after this was filed: the fork pin went 4a58da9b0 → 1d32c2260 → ba4fdf01bb → d8bd0ff605, and #161
(199165047, 11-01) added TOP_K on HRX together with a shape-scoped claim —
ggml-hrx.cpp:1286-1290 now routes GGML_OP_TOP_K through hip_top_k_tensor_supported() instead of falling through to the
generic list. So error 2 should be re-tested before anyone writes a RESHAPE for HRX0: if it is gone, this row is blocker 1
alone, which is one ctx_other in the draft's measurement path and nothing else.

What did not change and is worth stating, because it is the reason this row is not a one-liner: the merged MTP work is the
NPU P6 path (#811 → e6af10086, #820 → 134639841, #838), a different engine from llama.cpp's --spec-type draft-mtp. Nothing there reaches this.

The bar cannot be measured on this box right now, by design or by deletion

The fork-side numbers the issue sets as its target (44.86 / 47.87 tok/s, 77 % acceptance) were measured with
~/models/qwen38/avalon-mtp (1.38 GiB, third-party, "deletable" per its own text). Tonight:

ls ~/models/qwen38/          -> DFlash2/  UD-Q4_K_XL/  wikitext-2-raw/     (avalon-mtp is gone)
find ~ -name '*dv65k*' -o -name '*FR65k*'   -> nothing

So the acceptance bar needs either the third-party head re-fetched or a house cut-vocab head — the latter is explicitly out
of scope of this issue ("a cut-vocab head builder is not part of this issue"), which makes re-fetching the only path to the
bar as written. That is an OWNER decision, not a lane one: it is a third-party artifact on a box that is 76 % full.

The model side is fine and present — ~/models/flash-next/UD-IQ3_XXS (76.33 GiB across three shards) — but a run needs
MemAvailable ≥ ~92 GiB; it is 43 GiB tonight (other lanes), so this row cannot be worked overnight. Whoever takes it:
one gpu-hold claim, the >20 GB MemoryMax rule, the #715 no-SVM trio, and one of the two reset-guardrail builds (#680).

Close condition (unchanged)

Both errors gone in one serve, then a re-measure with the issue's own protocol (512 greedy tokens, median of 3, short 216 /
long 18,782, -c 32768 --parallel 1 -fa on -ctk q8_0 -ctv q4_0) at ≥ 44.86 short / ≥ 47.87 long and ≥ 77 % acceptance.
If the intent is the decode gap rather than the drafting mechanism, #679's arm d1 is the cheaper question on the same file
and this row is downstream of it.

Proposed: keep open (nothing merged fixes it), owner pi-attn (with #679), and the next action is the arm above —
not code. Commenting rather than closing per tonight's rule.

Partial fix landed on a fork branch; the acceptance table is blocked on hardware availability (memory), not on code.

Reproduced at the current pin

With the house draft head ~/models/flash-next/MTP/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf (fork format; the issue's third-party avalon-mtp head is no longer on the box), the pre-fix server dies with exactly the issue's two lines: qwen4exp requires ctx_other to be set from the memory-fitting probe, and the -md path never gets past it.

Both minimal fixes are written (fork branch `1bit/i681-mtp-draft @ f1487819e`, build green)

src/llama-context.cpp: for EAGLE3/DFLASH/QWEN4EXP draft heads with no tok_embd/output and no ctx_other, warn instead of throw during the memory-fitting probe (which has no target); the real draft context sets ctx_other in common_speculative_init (common/speculative.cpp:2472), so the semantics for real contexts are unchanged.
src/llama-sampler.cpp: the backend support probe now skips RESHAPE/VIEW/PERMUTE/TRANSPOSE — metadata-only views the scheduler resolves, which HRX0 was rejecting the top-k sampler candidate graph over.

Verified: the patched server loads; the head-only probe now logs the warning and reaches the fork's explicit draft guard ("load it as a draft of its target model (-md)") instead of dying in ctx_other; a -md run against a deliberately wrong small target gets through draft-context creation and fails only on the expected draft and target disagree on the shape of token_embd.weight — i.e. past blocker 1. A normal 0.6B CPU server still comes up (no regression). The sampler-view fix is compile-verified only.

Why no PR and no table yet

The issue's protocol needs the real target (Qwen3.8-Flash-Next UD-IQ3_XXS, 76.3 GiB) plus the draft head — about 92 GiB — and MemAvailable is 47 GiB tonight because other lanes hold GTT; the box's GPU token was held for the entire attempt. So the n=2 decode/acceptance table (512 greedy tokens, short 216 / long 18,782, acceptance, GTT peak) is unrun, and whether RESHAPE still appears in the sampler probe on the real graph is unproven (if it does, the next step is HRX graph-exec view handling, not the probe skip).

Disposition: parked on hardware availability (a quiet box with ~100 GiB free). Unpark: one gpu-hold claim with -md <house head> --spec-type draft-mtp -ngld 99 on the real target; if it drafts, the branch becomes a PR with the measured table (the lane is watching for that window). A smaller target quant, if one exists on the box, would shorten the wait but changes the issue's row — it would be labelled as such.

third_party: bump llama.cpp to 710c7c23 -- a separate MTP draft head …

Resolved: a separate MTP draft head now loads and drafts on the HRX route, with the gain measured. The fork fixes are merged (1bit-MONSTER/llama.cpp#219, tip 710c7c23) and the engine pin bump #1252 carries them.

The two blockers, and the two fixes

qwen4exp requires ctx_other to be set was thrown by the memory-fitting probe, which has no target context. src/llama-context.cpp now warns instead of throwing for EAGLE3/DFLASH/QWEN4EXP draft heads with no tok_embd/output; the real draft context still sets ctx_other in common_speculative_init, so semantics are unchanged.
The top-k sampler's candidate graph needs RESHAPE, which HRX0 refused. src/llama-sampler.cpp's backend support probe now skips RESHAPE/VIEW/PERMUTE/TRANSPOSE — metadata-only views the scheduler resolves, not kernels.

Measured (strixhalo, `UD-IQ3_XXS`, HRX0 `-ngl 99 -c 32768 -np 1 -fa on --cache-reuse 256`, house `mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf`, `--spec-type draft-mtp --spec-draft-n-max 2`, greedy 512 tokens, 1 rep)

arm	short 211 tok	medium 1,030 tok	acceptance	GTT peak
no head	27.27 t/s	27.84 t/s	-	82.0 / 82.5 GiB
+ head, n=2	41.76 t/s	41.25 t/s	80.8 % / 78.4 %	85.0 / 85.8 GiB
gain	+53.1 %	+48.2 %

This matches the issue's +55 % decode and clears its 77 % acceptance bar; the draft costs ~3 GiB GTT. Labelled deviations: the arms are not clock-matched (no-head pp_dpm_sclk samples were lost with the expired ticket), -ctv f16 instead of q4_0, the medium row is 1,030 tokens rather than 18,782, and 1 rep not median-of-3.

What the long row exposed

The 18,754-token prompt dies in the CPU fallback of GGML_OP_SET_ROWS (ggml-cpu/ops.cpp:5136, out-of-range i1) — reproduced without -md and on the unpatched fork tip, so it is pre-existing and not the draft path. Filed separately as #1251, along with the observation that HRX rejects the q4_0 SET_ROWS node outright. The long-context and q4_0 rows of this issue can be re-run unchanged once #1251 is fixed (or in a later window).

Also recorded in the issue

the two engines cannot share one head file: the fork's FR65k head uses a quant type the house binary lacks, and the fork rejects our dv65k head on output.weight dims — a head must be built per engine;
HRX0 also lacks TOP_K for backend-side sampling (warning, falls back) — noted, not blocking.

Closing on the drafting fix + the measured gain; #1251 tracks the long-context gap this measurement uncovered.

Navigation Menu

Repository navigation

hrx: a separate MTP draft head (-md) is rejected for qwen4exp on our route, while a cut-vocab head gives +55% decode on the fork #681

Summary

What drafting is worth, measured on the fork (same GGUF, same hour)

Why our route cannot use this today — three separate blockers

Smallest useful change

Reproduce

Related

Blocker 1 is unchanged, and it is total

Blocker 2 may have changed, and only one arm can say

The bar cannot be measured on this box right now, by design or by deletion

Close condition (unchanged)

Reproduced at the current pin

Both minimal fixes are written (fork branch `1bit/i681-mtp-draft @ f1487819e`, build green)

Why no PR and no table yet

The two blockers, and the two fixes

Measured (strixhalo, `UD-IQ3_XXS`, HRX0 `-ngl 99 -c 32768 -np 1 -fa on --cache-reuse 256`, house `mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf`, `--spec-type draft-mtp --spec-draft-n-max 2`, greedy 512 tokens, 1 rep)

What the long row exposed

Also recorded in the issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: a separate MTP draft head (-md) is rejected for qwen4exp on our route, while a cut-vocab head gives +55% decode on the fork #681

Description

Summary

What drafting is worth, measured on the fork (same GGUF, same hour)

Why our route cannot use this today — three separate blockers

Smallest useful change

Reproduce

Related

Activity

bong-water-water-bong commented on Nov 3, 2026

Blocker 1 is unchanged, and it is total

Blocker 2 may have changed, and only one arm can say

The bar cannot be measured on this box right now, by design or by deletion

Close condition (unchanged)

bong-water-water-bong commented on Nov 6, 2026

Reproduced at the current pin

Both minimal fixes are written (fork branch 1bit/i681-mtp-draft @ f1487819e, build green)

Why no PR and no table yet

bong-water-water-bong commented on Nov 6, 2026

The two blockers, and the two fixes

Measured (strixhalo, UD-IQ3_XXS, HRX0 -ngl 99 -c 32768 -np 1 -fa on --cache-reuse 256, house mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf, --spec-type draft-mtp --spec-draft-n-max 2, greedy 512 tokens, 1 rep)

What the long row exposed

Also recorded in the issue

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Both minimal fixes are written (fork branch `1bit/i681-mtp-draft @ f1487819e`, build green)

Measured (strixhalo, `UD-IQ3_XXS`, HRX0 `-ngl 99 -c 32768 -np 1 -fa on --cache-reuse 256`, house `mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf`, `--spec-type draft-mtp --spec-draft-n-max 2`, greedy 512 tokens, 1 rep)