Skip to content

third_party: bump llama.cpp to 710c7c23 — a separate MTP draft head loads and drafts on HRX (+53% decode) (closes #681) - #1252

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/bump-llama-710c7c23
Nov 6, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/bump-llama-710c7c23

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Bumps third_party/llama.cpp c2c078b5 -> 710c7c23, the three commits of fork PR 1bit-MONSTER/llama.cpp#219 — a separate MTP draft head for qwen4exp now loads and drafts on HRX (engine#681):

  1. src/llama-context.cpp: for EAGLE3/DFLASH/QWEN4EXP draft heads with no tok_embd/output and no ctx_other, warn instead of throw in the memory-fitting probe (the real draft context sets it in common_speculative_init; semantics unchanged);
  2. src/llama-sampler.cpp: the backend support probe skips RESHAPE/VIEW/PERMUTE/TRANSPOSE metadata views, so HRX0 accepts the top-k sampler's candidate graph;
  3. drop the unused fit-path hunk.

Measured (strixhalo, Qwen3.8-Flash-Next UD-IQ3_XXS, HRX0 -ngl 99 -c 32768 -np 1 -fa on --cache-reuse 256, house mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf, --spec-type draft-mtp --spec-draft-n-max 2, greedy 512 tokens, 1 rep)

arm short 211 tok medium 1,030 tok acceptance GTT peak
no head 27.27 t/s 27.84 t/s - 82.0 / 82.5 GiB
+ head, n=2 41.76 t/s 41.25 t/s 80.8 % / 78.4 % 85.0 / 85.8 GiB
gain +53.1 % +48.2 %

The gain matches the issue's +55 % and the acceptance clears its 77 % bar; the draft adds ~3 GiB GTT.

Labelled deviations (not hidden)

  • not clock-matched: the no-head arms' pp_dpm_sclk samples were lost with the expired ticket (head arms: 655-2857 / 1064-2803 MHz), so the rows are labelled rather than claimed as controlled;
  • -ctv f16, not the issue's -ctv q4_0 (HRX rejects the q4_0 SET_ROWS node — see the new engine#1251);
  • the medium row is 1,030 tokens, not the issue's 18,782: the long prompt dies in a pre-existing CPU-fallback SET_ROWS assert (ggml-cpu/ops.cpp:5136), reproduced without -md and on the unpatched fork tip -> filed as hrx: a >2k-token prompt dies in the CPU fallback of GGML_OP_SET_ROWS (out-of-range i1 assert), with or without -md #1251;
  • 1 rep, not median-of-3.

Unproven: the long-context row, the q4_0 row, median-of-3. After this bump they can be re-run unchanged once #1251 (or a quiet window) allows.

Merge gate

Required context is build (the engine build does not compile HRX); the device table above is recorded on the fork PR and on engine#681. registry/architectures.json regenerated (sources: 710c7c23, 298/298) and tools/check_pins.py reports c2c078b58 -> 710c7c23c is ahead. Upstream pin move with a measured behaviour change; no RFC-relevant surface.

@bong-water-water-bong
bong-water-water-bong merged commit 8232f32 into main Nov 6, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/bump-llama-710c7c23 branch November 6, 2026 11:39
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

681 - Partially compliant

Compliant requirements:

  • Make the separate-head path work for qwen4exp on HRX at n=2 (the fork's configuration)
  • -md no longer dies on ctx_other and does not fail on missing RESHAPE for the top-k sampler on HRX0
  • Re-measured and reported decode gains of +53.1% and +48.2% at short and long context respectively, with acceptance of 80.8% and 78.4%, exceeding the 77% bar

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Memory Fitting Issue

The change in src/llama-context.cpp warns instead of throwing when ctx_other is not set for EAGLE3/DFLASH/QWEN4EXP draft heads. While this allows the draft to load, it may mask an underlying issue where the memory fitting logic is not correctly accounting for the draft context, potentially leading to runtime errors or suboptimal memory usage if the draft context is truly missing or misconfigured.

Subproject commit 710c7c23ca92fe9424f08cb851536c03dfdc5538

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant