Skip to content

hrx: attach a separate MTP draft head, and let the sampler support probe skip metadata views (engine #681) - #219

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
1bit/i681-mtp-draft
Nov 6, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
1bit/i681-mtp-draft

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

engine ggml-org#681: a separate MTP draft head is rejected for qwen4exp on the HRX route. Two minimal fixes, plus what I could measure on the box today.

Fixes

  1. src/llama-context.cpp: a qwen4exp/eagle3/dflash draft head has no tok_embd/output and needs its target context. The memory-fitting probe has no target, so the constructor threw "requires ctx_other to be set" before the real draft context (which sets ctx_other in common_speculative_init) was ever built. Warn instead of throw; real draft contexts still get ctx_other.
  2. src/llama-sampler.cpp: the backend support probe rejected the top-k sampler candidate graph on HRX0 because it contains RESHAPE. Reshape/view/permute/transpose are metadata-only views resolved by the scheduler, so skip them in the probe instead of failing the sampler graph.

What this measured (strixhalo, fork tip + these commits)

Target: Qwen3.8-Flash-Next-UD-IQ3_XXS (76.3 GiB), -dev HRX0 -ngl 99 -c 32768 --parallel 1 -fa on --cache-reuse 256, house draft head mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf, --spec-type draft-mtp --spec-draft-n-max 2, greedy 512 tokens, 1 rep per cell (not the issue's median of 3).

arm short (211 tok) medium (1030 tok) draft acceptance GTT peak
no head 27.27 t/s 27.84 t/s -- 82.0 / 82.5 GiB
+ house head, n=2 41.76 t/s 41.25 t/s 315/390 = 80.8 % (short), 312/398 = 78.4 % (medium) 85.0 / 85.8 GiB
gain +53.1 % +48.2 %

Head-run clocks sampled per request: short 655-2857 MHz, medium 1064-2803 MHz (pp_dpm_sclk). The no-head arm's clock samples were not persisted before its ticket expired, so the rows are not clock-matched; the +48-53 % is far above the run-to-run spread seen in these cells.

Why this is not the issue's exact row

  • -ctv q4_0 (the issue's protocol) is rejected by HRX during prefill: unsupported HRX node ... SET_ROWS ... q4_0 cache. The table above uses -ctk f16 -ctv f16.
  • The 18,782-token prompt crashes the server in the CPU fallback: ggml-cpu/ops.cpp:5136: GGML_ASSERT(i1 >= 0 && i1 < ne1) failed inside ggml_compute_forward_set_rows_impl (GGML_OP_SET_ROWS). It reproduces without -md and on the unpatched fork tip 59d7f82 at the same prompt, so it is pre-existing and unrelated to this change (and unrelated to -md). It needs the SET_ROWS/CPU-fallback gap fixed before the long-context row can be measured.
  • HRX also lacks TOP_K for the backend sampler (warning only, falls back).

Unproven

  • Median-of-3, the issue's 18,782-token long row, and the -ctv q4_0 row: all blocked as above.
  • Absolute numbers are on this box with a 76 GiB target and should be re-measured with the exact protocol once the SET_ROWS gap is fixed.

Assisted-by: DeepSeek Harness

…ph skip metadata views (engine ggml-org#681)

A qwen4exp MTP draft head has no tok_embd/output and needs its target context; the memory-fitting probe has none, so the context constructor threw 'requires ctx_other to be set' before the real draft context (which sets ctx_other in common_speculative_init) was ever built. Warn instead of throw: fitting measures, the real draft context still gets ctx_other.

The sampler backend support probe rejected the top-k candidate graph on HRX0 because it contains RESHAPE, a metadata-only view the scheduler resolves; skip view ops in the probe.

Assisted-by: DeepSeek Harness
…e fitting probe (engine ggml-org#681)

Assisted-by: DeepSeek Harness
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant