ggml: refactor selective expert copying to user code by am17an · Pull Request #29943 · ggml-org/llama.cpp

am17an · 2026-10-04T10:20:06Z

Overview

Refactor the selective expert copying added in #15346 to be called from user code. No functional change is expected from this change. The idea is that this will help in #29887 to be complety handed in user-code. After this is merged, the only change there should be in user code.

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

ggerganov · 2026-10-04T15:19:59Z

+    const int64_t n_expert    = src->ne[2];
+    const size_t  expert_size = src->nb[2];
+
+    if (ids != st.ids || (int64_t) st.used.size() != n_expert) {


In which situations, the ids == st.ids?

I see that this case is also handled on master, but it is not clear to me why.

it's because we transfer gate, up and down and they can all re-use the same ids tensors

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* ggml: refactor selective expert copying to user code * tests: enroll two models into selective expert copy test * tests: use deepseek2 as test model * improve comment in ggml-backend.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * cont: fix whitespace * cont : better comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> (cherry picked from commit 6753a03)

…with upstream PR ggml-org#29943 / ggml-org#30042

…ts ahead and with O_DIRECT again The ggml-org#29943 port moved the selective expert upload into sched_copy_experts but left the lookahead source and the staged direct reads behind in the scheduler, where nothing called them. Prefill then copied every non-pool expert from the cold mapping.

Brings LiquidAI d1-3B / d1-omni-600M decision models (ggml-org#30110, ggml-org#30114), CLEF vision, pplx-decider, LFM2.5 encoder, split router child stdout/stderr (ggml-org#29895), MoE expert-copy callback (ggml-org#29943) and general upstream fixes. 27 conflicted files (69 hunks) resolved keeping both sides; fork-line audit on every conflicted file and on the 57 auto-merged co-touched files: no fork lines lost. server-node.cpp ported to the split subproc API. Server keeps the fork's llama_batch decode path and routes decision/joint-head batches through llama_process (common_batch). Fork's non-fatal spec/checkpoint restore kept. Notes: ~/.cache/llama-sync-20261008/ (reports, review, rulings). Backup: backup/pre-upstream-sync-2026-10-08 @ 49e0f85. Not yet compiled. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…-10-10) Conflicts: - ggml-backend.cpp: upstream moved input copies to ggml_backend_sched_copy_input with host weights last (ggml-org#29943); the GGML_SCHED_BATCH_INPUTS fast path now runs in the first loop before falling back to it. - ggml-cuda/fattn-common.cuh: launch_fattn takes upstream async_kv_preload, then our parallel_blocks_fixed (GQA-6 tile launcher updated). - ggml-cuda/fattn-mma-f16.cuh: upstream swizzle refactor (ggml-org#29612); its helpers take ncols2 like the rest of the fork's config getters. - ggml-cuda/gated_delta_net.cu: keep the fork's kernel and launch geometry (its columns_per_block template argument is not upstream's cols_per_warp, ggml-org#30087). - ggml-cuda/mmq.cu: upstream per-tile src1 padding (ggml-org#29953); ggml_cuda_mul_mat_q_src1_nbytes and the MMQ input cache use the worst-case padding so producers and projections with other tiles can share the buffer. - ggml-cuda/mmq-vec-dot.cuh: fork vec_dot kernels pass GGML_PREC_Q8 to the config helpers (ggml-org#30168). - ggml-cuda/mmvf.cu/.cuh: keep both includes and the HIP declarations, upstream warp_size argument. - ggml-cuda/rope.cu: upstream grid-stride rms_norm+mul+rope (ggml-org#28175) with the fork's M-RoPE theta (nchannels instead of gridDim.y) and rope_store_cast. - ggml-cuda/top-k.cu: upstream shape-based selection (ggml-org#28713), RDNA4 tournament kept for few long rows. - src/llama-batch.*: mixed token/embd batches (ggml-org#29622) copy embd rows per entry; other batches keep pointing at the ext rows. - src/llama-context.cpp: expert copy callback also on the second MTP scheduler. - src/llama-graph.cpp: build_rs keeps the GET_ROWS of the ubatch states (GDN fusions, deferred state) and gathers the extra states from the tail like upstream's custom getter path (ggml-org#29856).

…in (the old sched loop did; the move to user code in the ggml-org#29943 merge dropped it): the cache sees the experts of offloaded batches

…->GPU and GPU splits that read no host memory (the fork's refinement of ggml-org#26040, lost in the ggml-org#29943 merge: every CPU split after a GPU split waited for the GPU, 6.6 ms per GLM decode graph, OLMoE -16% on the RTX 4060 laptop)

…ts lent by the expert pool; compute alias on by default Prefetch overlap (removed by the ggml-org#29943 merge): while split i computes, the full expert tensor of split i+1 is uploaded into a staging slot on a second stream of the same device, so a streamed prompt pass costs max(upload, compute) per layer instead of their sum. --prefetch-experts-slots / --moe pf-slots: -1 = auto (3, new default), 0 = off. The slots are ranges of the expert pool that the compute alias lends for the duration of the graph; when the pool cannot lend them the graph runs without prefetch, it never allocates a buffer behind the cache's back. Compute alias default on (--moe alias=0 keeps the scheduler's own compute buffers): the micro-batch shrinks (half, down to 512) until the worst-case graph, its CUDA scratch and the staging slots fit in the lendable blocks; at 512 the staging is skipped, and if even the graph does not fit the alias drops out. The shrink path has not been exercised on a small card yet. Measured, same session, one x16 card at 370 W, --load-mode pin, cut-down GLM (12 layers) 12k prompt, prompt tokens/s: prefetch off 2466 -> on 3034 (+23%). OLMoE, ~3k-token prompt streamed from RAM: off 8052 -> on 11691 (+45%), upstream stock b11552 10270. Full GLM 3.0-bit 12k prompt, alias on vs off at micro-batch 4096: decode 17.8 -> 19.0 tokens/s, cache hit rate 70.2 -> 74.6%; micro-batch 8192 now runs (458.8 prompt tokens/s, out of memory before).

Brings in 221 upstream commits, including ggml-org#14891 (activation-based imatrix statistics for GGUF imatrices). Resolutions: - qwen4exp: take upstream's MTP (ggml-org#29761) over this fork's halo-box#9 head. The two write different GGUFs (upstream keeps the MTP block's own hyper-connection mixer as nextn.hc_head_* and runs the draft with QSA), and current MTP files use upstream's layout. Also drop halo-box#9's attention-only MTP memory for qwen4exp in llama-model.cpp and its extra NEXTN entries in constants.py. - speculative: keep --spec-draft-adaptive's n_draft_eff bound on top of upstream's probabilistic draft sampling (result_q); begin() resets both the acceptance EMA and the sampler. - ggml-backend: take upstream's pass-6 input collection (ggml-org#29634) and ggml_backend_sched_copy_input() (ggml-org#29943), keeping halo-box#23's copy-0 original with input rotation, async-read pinning, and the sanitizer hook. - vulkan: keep halo-box#12's UMA read-back gate (host-cached or <= 64 KiB direct); upstream's AMD-only fix is a subset of it. - server: keep spec-prefill's prompt_src() next to upstream's use_spec_rejection(); use add_embd() with M-RoPE positions. - speculative-prefill: migrate to common_batch / llama_process, as upstream ggml-org#29601 removed common_batch_add/clear. - README: keep this fork's quick start; common.h, test-alloc: keep both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

276 upstream commits. Conflicts and how they were resolved: - K2-Horizon: upstream added its own port of the same ifm-ai code (ggml-org#29535), with the chat parser, the unicode splitter and tests. Upstream's model, converter and names are taken; GGUF names are identical, so existing K2-Horizon GGUFs still load. The fork keeps only the K2-Horizon-36B tokenizer hash, which upstream dropped. LLAMA_VOCAB_PRE_TYPE_K2_HORIZON moves from 60 to 61 to follow upstream (it is not stored in GGUFs). - MoE: upstream moved the selective expert copy out of the scheduler into a llama_context copy callback (ggml-org#29943) and added a GPU cache for host experts (ggml-org#29887). The fork's partial mul_mat_id (id < 0 rows, used by LLAMA_MOE_HOT) now skips negative ids in that callback. build_lora_mm_id takes both the fork's partial/ids_s and upstream's MoE cache slots; build_moe_ffn_exps passes the slots on the untiered path only, so the MoE cache and LLAMA_MOE_HOT tiering do not stack on one layer. - mmvq: upstream's shared-expert fusion and the fork's negative-id early return combined; the shared-expert block never takes the early return. - server: upstream's rejection sampling and checkpoint replay paths gained the fork's on_sample hook, so speculative logprobs cover them too. - qwen4exp / build_attn: turbo Q pre-rotation kept on top of upstream's reworked QSA mask and the new training path. - top-k.cu: upstream's CCCL >= 3.4.3 gate; the fork's hipcub arm was dead since HIP stopped defining GGML_CUDA_USE_CUB. - ggml-cuda init_tensor: upstream's async memset on owned buffers, the fork's host-side padding clear on externally owned ones. - tests/test-turbo-kv.cpp, tests/test-moe-tier.cpp: moved to common_batch and llama_process, as upstream removed common_batch_add. Tested on gfx1100: ctest 76/78 (the two failures are environmental: the ggml-vocabs clone has git-lfs pointers, and system python has no jinja2; test-jinja-py passes with it). test-llama-archs 506/507; the qwen2moe "Meta" failure is a seed-dependent flake that the pre-merge build shows too. Server speculative/logprobs pytests 18/18. Needle at 12k tokens with turbo4 KV: K2-Horizon-36B, Qwen3.8-Flash-Next, Qwen3.8-27B (plus MTP).

* ggml: refactor selective expert copying to user code * tests: enroll two models into selective expert copy test * tests: use deepseek2 as test model * improve comment in ggml-backend.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * cont: fix whitespace * cont : better comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

am17an requested a review from ggerganov as a code owner October 4, 2026 10:20

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Oct 4, 2026

ggerganov self-assigned this Oct 4, 2026

ggerganov reviewed Oct 4, 2026

View reviewed changes

github-actions Bot added the testing Everything test related label Oct 4, 2026

am17an force-pushed the aman/sched-copy-callback branch from 7441797 to 72cb4da Compare October 4, 2026 15:46

ggerganov reviewed Oct 4, 2026

View reviewed changes

Comment thread ggml/include/ggml-backend.h Resolved

ggerganov approved these changes Oct 4, 2026

View reviewed changes

am17an and others added 5 commits October 6, 2026 09:50

ggml: refactor selective expert copying to user code

9b1d1f6

tests: enroll two models into selective expert copy test

0a06090

tests: use deepseek2 as test model

f05e0fe

improve comment in ggml-backend.h

33579f6

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

cont: fix whitespace

ca2934f

ggerganov force-pushed the aman/sched-copy-callback branch from 528e0a3 to ca2934f Compare October 6, 2026 06:50

ggerganov reviewed Oct 6, 2026

View reviewed changes

Comment thread ggml/include/ggml-backend.h Resolved

Comment thread ggml/include/ggml-backend.h Resolved

cont : better comments

d4153ac

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

ggerganov merged commit 6753a03 into master Oct 6, 2026
26 of 27 checks passed

ggerganov deleted the aman/sched-copy-callback branch October 6, 2026 07:59

antomicblitz mentioned this pull request Oct 6, 2026

llama : add a GPU cache for MoE experts kept in host memory #29887

Merged

dzannotti mentioned this pull request Oct 7, 2026

Merge upstream llama.cpp (2026-10-07) halo-box/llama.cpp#34

Merged

sodre90 added a commit to sodre90/llama.cpp that referenced this pull request Oct 8, 2026

llama : reconcile QSA k-pool layout and backend scheduler copy stats …

0c40091

…with upstream PR ggml-org#29943 / ggml-org#30042

ggerganov mentioned this pull request Oct 15, 2026

ggml : bump version to 0.27.0 ggml-org/ggml#1659

Merged

BrewTestBot mentioned this pull request Oct 15, 2026

ggml 0.27.0 Homebrew/homebrew-core#318733

Merged

1 task done

Navigation Menu

Repository navigation

ggml: refactor selective expert copying to user code - #29943

ggml: refactor selective expert copying to user code#29943
ggerganov merged 6 commits into
masterfrom
aman/sched-copy-callback

am17an commented Oct 4, 2026

Uh oh!

Uh oh!

Uh oh!

ggerganov Oct 4, 2026

Uh oh!

ggerganov Oct 4, 2026

Uh oh!

am17an Oct 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

am17an commented Oct 4, 2026

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Uh oh!

ggerganov Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

am17an Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants