Repository navigation
Conversation
This appear to solve ggml-org#153 where error of "ggml_new_tensor_impl: not enough space in the context's memory pool" is thrown in interactive mode. At least the out of memory error come from `ctx0` used here. Although I am not familiar with the code base enough to tell if this is indeed the cause.
Collaborator
|
This definitely increases the memory in the right place. |
Member
|
This does not happen out of interactive mode, so I don't think that this is right. This would mean 1.5 GB more memory usage for 2048 context size so it is significant. |
Collaborator
|
I looked at it more and I dont think this is the way to solve this. We should instead "determine the required inference memory per token" https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#L891 edit: I just saw that it is set, but only after the first run. https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#L749 , which means the reallocation logic has an error. @slaren the 7B model does not run into this issue. |
Collaborator
|
Edit again: after testing the 30B model again, I realized that the bug only happens in an Interactive session. |
|
I've been hitting this with the 65B model in oneshot mode with a 2048-token context. So it's not just interactive sessions that are affected. This is fairly easily reproducible with the following: (Note: smaller batch sizes take longer before aborting, but still suffer the same issue.) I suspect that the actual required buffer size is a linear function of the context length, but with a nonzero constant term. Maybe add debug code to print out the actual high watermark at the end? |
Collaborator
|
I forgot to mention that I ran 30B with ctx2048 :) |
Member
|
Increasing batch size also makes llama.cpp run out of memory, so any solution that only considers the context size and not the batch size is likely wrong. |
Ah. That explains why I was seeing larger batch sizes OOM faster. |
Member
|
Thinking more about this, does it really matter what is the initial value of The "proper" solution to this would be to analyze the code painstakingly to be able to accurately predict how much memory is needed in the context, but that is not going to be easy and it is going to break any time that changes are made. Does it really matter anyway? Any memory reserved for the context won't be committed to physical memory until the pages are touched anyway. Alternatively we could catch the out of memory errors from ggml and realloc the buffer as needed. |
Collaborator
|
@slaren earlier i wrote:
|
|
Throwing some ideas about actual reasons behind the bug. I think it's the classic integer division gotcha: if batch size 'N' > 1 then there will be loss of fraction. Just after these lines I added: And tracked calls after the first one that sets the value: As expected, actual memory usage differs from predicted and sometimes bigger. That's not good. |
|
Looking further, it also slowly creeps up as prompt being read(batch size = 4) In a hindsight that makes sense. Attention mechanism performs more operations when context grows. Today's code makes wrong assumption by expecting that memory usage per token will remain static. It clearly grows by 65536 in each iteration. I think total memory usage at full context can be inferred from measurements taken by multiple |
|
I haven't looked; how does LLaMA handle prompts that are smaller than context size? E.g. a 1024-token prompt with 2048-token context size. Does it just truncate to an effective context size of 1024? Or does it pad somehow? |
Author
To add another observation, the amount of memory increase per iteration seems to scale quadratically with batch size. For the 65B 4bit model. A batch size of 4 gives: Where batch size of 8 gives: Where That could explain the previous observation that larger batch size OOMs faster. |
|
Right, batch processing at least must construct This is also a good avenue for optimization. 'Scientific' code like this is only good for GPUs where compute units share instruction decoder and cannot diverge, e.g. if 9 cores do 1 iteration in a loop and 1 core does 10 iteration then they all must do 10 iterations because they execute same instructions in a lockstep. Assuming same model on CPU is a huge waste of time and memory as we can see. |
|
ask chatGPT for a solution lol |
This was referenced Mar 22, 2023
Collaborator
|
please try #438 and see if it fixes it. |
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
…y 128 (ggml-org#213) * Fix gemm/gemv for legacy quants when row size is not divisible by 128 * Fix typo --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 25, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 28, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 29, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 30, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
Nathanw1014
pushed a commit
to Nathanw1014/llama.cpp
that referenced
this pull request
Oct 31, 2026
hip : Qwen3.8 Flash Next prefill on gfx1151 (GDN, mmb, HC, QSA, multi-sequence)
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 3, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 3, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 4, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 4, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
bong-water-water-bong
referenced
this pull request
in 1bit-MONSTER/llama.cpp
Nov 5, 2026
… published GGUFs have an authoring site (engine#1098) (#206) * convert: Zamba2-VL (Zamba2 language model + Qwen2.5-VL tower), so the published GGUFs have an authoring site The three published `1bit-MONSTER/Zamba2-VL-*` GGUFs carry the template `9df322391abffe14587e6f3cfaea42420eaffec364835debfcb13c0e4c6eeab6` (1045 B), and the code that wrote it exists only in the Vulkan-era checkout (`llama.cpp-vulkan/conversion/zamba2vl.py`, now removed with that build, RFC #213) — so a pin bump could not re-derive those files (engine ggml-org#1098). This brings the module to the pinned tree, unchanged and registered: - `conversion/zamba2vl.py`: `Zamba2VLTextModel` (the `language_model.` half, `MODEL_ARCH.ZAMBA2`) and `Zamba2VLVisionModel` (the `vision_tower.` half as a Qwen2.5-VL `MmprojModel`), i.e. the model + mmproj pair the repos ship. `_CHAT_TEMPLATE` is Zyphra's own multimodal format — `<s>`-delimited turns, `<|vision_start|><image><|vision_end|>` — with the media markers as llama.cpp's `<__media_<id>__>` bridge, which is what makes the file usable by mtmd. - `conversion/__init__.py`: `"Zamba2_VLForConditionalGeneration": "zamba2vl"` in both `TEXT_MODEL_MAP` and `MMPROJ_MODEL_MAP`, so `load_all_models()` imports it like every other family. Verified against the pinned tree (no weights needed): both classes register for `Zamba2_VLForConditionalGeneration` (`ModelType.TEXT` -> `Zamba2VLTextModel`, arch `MODEL_ARCH.ZAMBA2` = 67; `ModelType.MMPROJ` -> `Zamba2VLVisionModel`, arch 1), and the module's `_CHAT_TEMPLATE` hashes to `9df322391abffe14…`, 1045 B — byte-identical to the template in all three published files. Pre-existing, untouched: `load_all_models()` at this pin already warns for `cambrian`, `deepseek41`, `flint`, `qwen4exp` and `dots3` (`Missing property 'model_arch'`, `no attribute 'example'`, a missing module); this change adds no warning. Engine side picks this up with the next llama.cpp pin bump; nothing in the engine changes here. * convert: keep Zamba2-VL's top-level architecture for the text pass, or the VL class is never built Registering `Zamba2VLTextModel` is not enough: `get_model_architecture()` prefers `text_config.architectures[0]` for the text pass, and Zamba2-VL's `text_config` says `Zamba2ForCausalLM`. A conversion therefore built the **plain** `Zamba2Model`, so the ported `set_vocab` never ran and the output carried Zyphra's `chat_template.jinja` (74475f26d1…, 796 B) and an unpadded 32004-token vocab instead of the published 1045-byte template and 32064 tokens — a silent wrong artifact, which is exactly what engine#1098 is about. `conversion/base.py`'s TEXT exception list already exists for this shape of model (`StepVLForConditionalGeneration`, `Sarashina2VisionForCausalLM`, `Exaone4_5_…`, `Step3p7_…`); it now names `Zamba2_VLForConditionalGeneration` too, with the reason in a comment. Measured on `Zyphra/Zamba2-VL-1.2B`, converted end to end through this branch and compared with the published `1bit-MONSTER/Zamba2-VL-1.2B-GGUF/Zamba2-VL-1.2B-Q8_0.gguf`: | | published | before this commit | after | |---|---|---|---| | `tokenizer.chat_template` | `9df322391a…` (1045 B) | `74475f26d1…` (796 B) | **`9df322391a…` (1045 B)** | | tokens | 32064 | 32004 | **32064** | | architecture | zamba2 | zamba2 | zamba2 | | tensors | 398 | 378 | 378 | The 20-tensor difference is the pin's own #203 ("store Zamba2's shared transformer block once per block"): the published files predate it and carry `blk.11.*` / `blk.17.*` a second time. It is not a port difference, and a re-quantise from this tree would not reproduce those 20 tensors. --------- Co-authored-by: agent <agent@local>
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm. TODO ggml-org#269 (r221): the decode remap reaches a TENSOR SPLIT and a covering host pool. The Meta backend now forwards `moe_cache_remap` to each simple device with that device's simple tensors, exactly as it already forwards `moe_cache_update`/`take_over`/`gather` -- one `check_only` pass answering for all devices, then one committing pass, so the scheduler's all-or-nothing skip-copy decision is unchanged. That is not a formality: the remap is per (table, device) by construction (each device holds its own arena, slot count, map and lag buffer), and the eager path it replaces synchronizes the WHOLE meta backend once per routed op, which is why the win grows with the device count -- 1 GPU 57.6 -> 63.3 t/s (+9.9 %), 2 GPUs `-sm tensor` 64.0 -> 74.3 (+16.1 %), 3 GPUs 59.4 -> 76.3 (+28.4 %), all 35B-A3B Q8_0 `-ncmoe 40` `MOE_EXPERT_CACHE_MIB=8000`, byte-identical greedy text (300 and 1000 tokens), and +5.0..6.0 % on `llama-batched-bench` at `-npl 1/4/8`. Under `-sm tensor` the remap was previously inert by construction (the iface member was NULL). The second half is the COLD SOURCE. A `--host-experts pool` table has a pageable master (`host_dev == nullptr`), so it had no cold encoding at all and took the eager path even when its pool COVERED the tensor -- which is the default `-lme auto` layout on a mapped load ("a pinned per-EXPERT pool of the whole host expert set (a MIRROR)"). Such a table now maps a cold expert to `slots + pool slot` (the pool slot, a host-side fact at map-build time) instead of `slots + e`; the consumer already resolved that id (`moe_cache_get_cold`'s pool branch). Measured on that default config, 1 GPU: 57.6 -> 63.3 t/s (+9.9 %), and +5.3 % under `--spec-type draft-mtp` with bit-identical acceptance (0.75655 = 693/916, mean len 3.26) and text. A sub-complement pool (any `-lme N` below the set) still declines every offer (0 of 3840 served, exactly as before), because there a device-side map would have a hole it cannot express; so does a covering pool with `MOE_HOST_POOL_COLD=0`. The freshness invariant the maintainer conditioned ggml-org#269 on is now structural rather than incidental: `dr_pool_ok` is latched once, at sizing (a graph capture needs a per-shape-constant take-over), and a covering pool's expert -> slot map is WRITE-ONCE after sizing -- the demand path never evicts a resident, and the one covering-pool re-rank is a no-op (`pool_rerank_from_heat_locked` returns before any bookkeeping). `devremap_prep_locked` re-counts the LIVE slots on every map rebuild and declines rather than emit an id it cannot resolve. Gates: prefill-logit KLD 0.000000 / same-top-p 100.000 % (data movement only), MTP acceptance bit-identical, `test-backend-ops -o MUL_MAT_ID` 931/931, `scripts/gate-host-pool.sh` 18/18, `check-gpu-faults.sh` clean. Kill switch unchanged (`GGML_MOE_DEVREMAP=0`).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 6, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm. TODO ggml-org#269 (r221): the decode remap reaches a TENSOR SPLIT and a covering host pool. The Meta backend now forwards `moe_cache_remap` to each simple device with that device's simple tensors, exactly as it already forwards `moe_cache_update`/`take_over`/`gather` -- one `check_only` pass answering for all devices, then one committing pass, so the scheduler's all-or-nothing skip-copy decision is unchanged. That is not a formality: the remap is per (table, device) by construction (each device holds its own arena, slot count, map and lag buffer), and the eager path it replaces synchronizes the WHOLE meta backend once per routed op, which is why the win grows with the device count -- 1 GPU 57.6 -> 63.3 t/s (+9.9 %), 2 GPUs `-sm tensor` 64.0 -> 74.3 (+16.1 %), 3 GPUs 59.4 -> 76.3 (+28.4 %), all 35B-A3B Q8_0 `-ncmoe 40` `MOE_EXPERT_CACHE_MIB=8000`, byte-identical greedy text (300 and 1000 tokens), and +5.0..6.0 % on `llama-batched-bench` at `-npl 1/4/8`. Under `-sm tensor` the remap was previously inert by construction (the iface member was NULL). The second half is the COLD SOURCE. A `--host-experts pool` table has a pageable master (`host_dev == nullptr`), so it had no cold encoding at all and took the eager path even when its pool COVERED the tensor -- which is the default `-lme auto` layout on a mapped load ("a pinned per-EXPERT pool of the whole host expert set (a MIRROR)"). Such a table now maps a cold expert to `slots + pool slot` (the pool slot, a host-side fact at map-build time) instead of `slots + e`; the consumer already resolved that id (`moe_cache_get_cold`'s pool branch). Measured on that default config, 1 GPU: 57.6 -> 63.3 t/s (+9.9 %), and +5.3 % under `--spec-type draft-mtp` with bit-identical acceptance (0.75655 = 693/916, mean len 3.26) and text. A sub-complement pool (any `-lme N` below the set) still declines every offer (0 of 3840 served, exactly as before), because there a device-side map would have a hole it cannot express; so does a covering pool with `MOE_HOST_POOL_COLD=0`. The freshness invariant the maintainer conditioned ggml-org#269 on is now structural rather than incidental: `dr_pool_ok` is latched once, at sizing (a graph capture needs a per-shape-constant take-over), and a covering pool's expert -> slot map is WRITE-ONCE after sizing -- the demand path never evicts a resident, and the one covering-pool re-rank is a no-op (`pool_rerank_from_heat_locked` returns before any bookkeeping). `devremap_prep_locked` re-counts the LIVE slots on every map rebuild and declines rather than emit an id it cannot resolve. Gates: prefill-logit KLD 0.000000 / same-top-p 100.000 % (data movement only), MTP acceptance bit-identical, `test-backend-ops -o MUL_MAT_ID` 931/931, `scripts/gate-host-pool.sh` 18/18, `check-gpu-faults.sh` clean. Kill switch unchanged (`GGML_MOE_DEVREMAP=0`).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 7, 2026
…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234. Amendment (TODO ggml-org#129): a routed host-weight op whose routing ids are computed by its own split. `ggml_backend_sched_copy_input` prunes a host expert table's upload to the routed experts, and the ids readback happens when the split's INPUTS are copied -- before that split computes. The generalised consumer search (this block) looks for ANY `MUL_MAT_ID` consumer of the input inside the split, and the split pass only starts a new split for a host weight once the split already has an input to reuse (`split->n_inputs > 0`). With `-nkvo` the attention runs on the host, so the GPU split begins at its output with no input yet: the split then swallows the router chain and the routed op, and the readback returns the previous split's leftover memory -- `GGML_ASSERT(id >= 0 && id < n_expert)` in the readback loop, or, when a stale word lands in `[0, n_expert)`, a silent wrong-expert selection. Two parts: * `ggml_backend_sched_copy_input`: when the ids root (`view_src` chain of `node->src[2]`) is a node of the split being copied, stand every routing-dependent shortcut down (the pruned upload, the device gather and the expert cache's update) and take the general input path's whole-table copy -- the same fallback the existing two-consumer case uses, and what stock upstream does for this shape (its prune callback only fires when the routed op is the split's first node). * `ggml_backend_sched_split_graph`: start the split AT such a routed op, after the router chain, which is the shape every other config already has, so the pruning and the expert cache keep working (`next_moe_layer` keeps the routed ops of one layer in the split that starts here). Measured on 1 x R9700: a 24-layer 128-expert MoE at `-ncmoe 24 -nkvo` aborts before this, runs at 24.6 t/s with the stand-down alone (whole-table upload per op) and 82.7 t/s with both parts (KV on GPU: 176, flat pre/post). Text byte-identical between the `-nkvo` and default arms on bailingmoe3, 35B-A3B (also with draft-mtp), gpt-oss-20b, Flash-Next/qwen4exp and a dense 4B, and identical to the pre-fix default arms. Fold (r202, PR ggml-org#266, briansp2020): the decode band no longer reads the routing back to the host. The eager hook synchronizes the backend once per routed op to read `ids` (the scheduler's `input_backend` sync plus the ids readback's own sync), runs the residency policy and uploads the remap. A new iface member `moe_cache_remap` is offered the split input FIRST (last struct member; NULL in the meta, cpu and rpc backends, so `-sm tensor` keeps the eager path): a kernel maps the device routing through a per-table expert -> slot map (`slots + e` = cold, read in place from the pinned master) into that table's `remap_dev`, and writes the routing into mapped pinned memory; the residency policy consumes that routing at the NEXT call (one call late) and re-uploads the map only after a residency change (`slot_dirty`). One launch serves a layer's tables (gate / up / down) when they were served the same routing tensor on the same stream. Every expert is read from the same bytes it was read from before -- an arena slot, or the same pinned master the cold read already used -- so the output does not change: byte-identical greedy text (35B-A3B `-ncmoe 40` and gpt-oss-20b at 128 and 1000 tokens, including a 2048 MiB arena under eviction pressure, pinned and `--host-experts pool`), identical MTP acceptance (0.76089 / 1387-of-1836 both arms, n=3000), prefill KLD 0.000000 / 100.000 %. Worth **+8.4 % TG** (60.6 -> 65.7 t/s, 1 x R9700, `-ncmoe 40`) and +2.3 % on the MTP arm, and it is inert under `-sm tensor` (98.0 vs 98.1 t/s, identical text). `GGML_MOE_DEVREMAP=0` restores the eager path. Scope, by construction: a pinned master (`table_cold_ok`), one unsplit table, no identity table, no `-sm tensor`, the decode/verify band only (`ne[2] <= band`), and `g_sized` (the eager path primes, sizes and re-primes). The map is rebuilt from `slot_dirty` on EVERY slot mutation (fill, eviction, stand-down, slab eviction, shrink, migration, re-arm), which is the one invariant to keep: a stale map reads the wrong arena slot, and the cold encoding `slots + e` additionally depends on the slot COUNT, so a slot-count change without a rebuild would read past the master. `devremap_free_locked` bumps `ggml_cuda_graph_mem_freed` (the map/lag buffers are kernel arguments of a captured graph). Also `MOE_HOST_POOL_COLD` (default on): `--host-experts pool` has a pageable master, so the in-place cold read stands down and every decode miss was an arena fill even when the expert was already in the pinned pool. Such a miss is now read cold from its pool slot through the pool's device alias (cold id `n_res + pool slot`, `moe_cache_get_cold` returns the pool base), i.e. the same bytes the fill would have copied. Safe because the demand path never evicts a pool resident, a retiring slot (state 2) is excluded, and the eager path it rides synchronizes the backend before the hook; single visible device, single slice. Measured (pool arm, 1 x R9700): fills 21747 -> 15270 and 8445 misses served in place, +4.9 % TG (57.5 -> 60.4 t/s), identical text. `MOE_HOST_POOL_COLD=0` restores the fill-every-miss arm. TODO ggml-org#269 (r221): the decode remap reaches a TENSOR SPLIT and a covering host pool. The Meta backend now forwards `moe_cache_remap` to each simple device with that device's simple tensors, exactly as it already forwards `moe_cache_update`/`take_over`/`gather` -- one `check_only` pass answering for all devices, then one committing pass, so the scheduler's all-or-nothing skip-copy decision is unchanged. That is not a formality: the remap is per (table, device) by construction (each device holds its own arena, slot count, map and lag buffer), and the eager path it replaces synchronizes the WHOLE meta backend once per routed op, which is why the win grows with the device count -- 1 GPU 57.6 -> 63.3 t/s (+9.9 %), 2 GPUs `-sm tensor` 64.0 -> 74.3 (+16.1 %), 3 GPUs 59.4 -> 76.3 (+28.4 %), all 35B-A3B Q8_0 `-ncmoe 40` `MOE_EXPERT_CACHE_MIB=8000`, byte-identical greedy text (300 and 1000 tokens), and +5.0..6.0 % on `llama-batched-bench` at `-npl 1/4/8`. Under `-sm tensor` the remap was previously inert by construction (the iface member was NULL). The second half is the COLD SOURCE. A `--host-experts pool` table has a pageable master (`host_dev == nullptr`), so it had no cold encoding at all and took the eager path even when its pool COVERED the tensor -- which is the default `-lme auto` layout on a mapped load ("a pinned per-EXPERT pool of the whole host expert set (a MIRROR)"). Such a table now maps a cold expert to `slots + pool slot` (the pool slot, a host-side fact at map-build time) instead of `slots + e`; the consumer already resolved that id (`moe_cache_get_cold`'s pool branch). Measured on that default config, 1 GPU: 57.6 -> 63.3 t/s (+9.9 %), and +5.3 % under `--spec-type draft-mtp` with bit-identical acceptance (0.75655 = 693/916, mean len 3.26) and text. A sub-complement pool (any `-lme N` below the set) still declines every offer (0 of 3840 served, exactly as before), because there a device-side map would have a hole it cannot express; so does a covering pool with `MOE_HOST_POOL_COLD=0`. The freshness invariant the maintainer conditioned ggml-org#269 on is now structural rather than incidental: `dr_pool_ok` is latched once, at sizing (a graph capture needs a per-shape-constant take-over), and a covering pool's expert -> slot map is WRITE-ONCE after sizing -- the demand path never evicts a resident, and the one covering-pool re-rank is a no-op (`pool_rerank_from_heat_locked` returns before any bookkeeping). `devremap_prep_locked` re-counts the LIVE slots on every map rebuild and declines rather than emit an id it cannot resolve. Gates: prefill-logit KLD 0.000000 / same-top-p 100.000 % (data movement only), MTP acceptance bit-identical, `test-backend-ops -o MUL_MAT_ID` 931/931, `scripts/gate-host-pool.sh` 18/18, `check-gpu-faults.sh` clean. Kill switch unchanged (`GGML_MOE_DEVREMAP=0`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This appear to solve #153 where error of
ggml_new_tensor_impl: not enough space in the context's memory poolis thrown in interactive mode, if using a larger context size.At least the out of memory error come from
ctx0used here. Although I am not familiar with the code base enough to tell if this is indeed the cause.