Add back top_k by beiller · Pull Request #56 · ggml-org/llama.cpp

beiller · 2023-03-12T18:13:33Z

@ggerganov not sure what you meant by temp is missing it was there. See PR.

gjmulder · 2023-03-14T10:12:00Z

If --top_k is supposed to be working, it doesn't seem to be with the latest code. Also confirmed with 13B, 30B and 65B 4-bit models:

$ git log | head -1
commit 97ab2b257897bfe7e2ae72876a3e50ed41b8c7ce
$ for M in ./models/7B; do for K in 5 10 20 40 80; do echo; echo "$M: --top_k $K"; ./main -m "$M/ggml-model-q4_0.bin" -p "Building a website can be done in 10 simple steps:" -t 16 --top_k $T; echo;done 2> /dev/null; done

./models/7B: --top_k 5
Building a website can be done in 10 simple steps:
Step #2 – Register your domain name. This is the address of your site, like www.yournamehere.com or .net etc… You will need to register this with an ICANN accredited registrar (like GoDaddy) and pay a yearly fee for it ($10-$35).
Step #4 – Choose a web host that meets your needs, like Bluehost which is recommended by WordPress.org or HostGator etc… You will need to sign up with one of these companies in order to have space on the internet where you can upload files and create pages for

./models/7B: --top_k 10
Building a website can be done in 10 simple steps:
Step #2 – Register your domain name. This is the address of your site, like www.yournamehere.com or .net etc… You will need to register this with an ICANN accredited registrar (like GoDaddy) and pay a yearly fee for it ($10-$35).
Step #4 – Choose a web host that meets your needs, like Bluehost which is recommended by WordPress.org or HostGator etc… You will need to sign up with one of these companies in order to have space on the internet where you can upload files and create pages for

./models/7B: --top_k 20
Building a website can be done in 10 simple steps:
Step #2 – Register your domain name. This is the address of your site, like www.yournamehere.com or .net etc… You will need to register this with an ICANN accredited registrar (like GoDaddy) and pay a yearly fee for it ($10-$35).
Step #4 – Choose a web host that meets your needs, like Bluehost which is recommended by WordPress.org or HostGator etc… You will need to sign up with one of these companies in order to have space on the internet where you can upload files and create pages for

./models/7B: --top_k 40
Building a website can be done in 10 simple steps:
Step #2 – Register your domain name. This is the address of your site, like www.yournamehere.com or .net etc… You will need to register this with an ICANN accredited registrar (like GoDaddy) and pay a yearly fee for it ($10-$35).
Step #4 – Choose a web host that meets your needs, like Bluehost which is recommended by WordPress.org or HostGator etc… You will need to sign up with one of these companies in order to have space on the internet where you can upload files and create pages for

./models/7B: --top_k 80
Building a website can be done in 10 simple steps:
Step #2 – Register your domain name. This is the address of your site, like www.yournamehere.com or .net etc… You will need to register this with an ICANN accredited registrar (like GoDaddy) and pay a yearly fee for it ($10-$35).
Step #4 – Choose a web host that meets your needs, like Bluehost which is recommended by WordPress.org or HostGator etc… You will need to sign up with one of these companies in order to have space on the internet where you can upload files and create pages for

* Add power management utilities to NPU device context and update DCVS settings * Update DCVS settings in power_utils to use v3 API and enhance power management * wip * Enhance dequantization functions by adding load_dequant_table support and updating signatures for improved performance * use lut * wip * fix test failure * wip * Refactor load_qual_block_generic to improve block handling and optimize vector operations * Enhance load_dual_block_generic and load_qual_block_generic to accept a mask parameter for improved block handling * Refactor flash_attn_impl to optimize mask l2 prefetch * wip * wip * wip * wip * add log * link against shared libraries instead of static ones * fix swiglu * wip * refactor expf_fix to handle overflow for different data types * enhance is_glu_op_supported to validate shapes for multiple sources * wip * refactor logging macros to use hexagon namespace and improve formatting * fix printf format error * wip * refactor: update static_assert messages for block size validation and add HVX_VectorPred_x3 type alias * rename * feat: enhance fa with mask * wip * wip * refactor: replace instances of Q6_V_vzero() with kZeroV for consistency * wip * wip * wip * fix: improve address alignment check in HVX_Vector handling * refactor: streamline vector dot product implementations for improved readability * refactor: q4k add hvx intrinsic impl * refactor: enhance dequantize_row_q4_K for clarity and performance * refactor: optimize scale mask usage in dequantization functions for improved performance * refactor: optimize dequantize_row_q4_K for intrinsic usage and performance improvements * refactor: move GLU operation implementation into separated file * sync after swiglu * wip * wip * wip * feat: increase prc main thread stack size * fix: replace hardcoded stack size with NPU_THREAD_STACK_SIZE constant * wip * feat: add optimized vector operations for exponential and division with overflow handling * wip * feat: refactor exponential function to handle overflow and underflow with improved logic * wip * wip * feat: add vector loading and scaling functions for improved performance in block processing * wip * feat: optimize block loading by refactoring scale index handling for improved performance * use Q6_Vb_vlut32_VbVbR_nomatch instead * feat: enhance scale loading by adding static assertion and restructuring block handling * wip * feat: refactor vec_dot_product_mixed_impl for improved clarity and performance * wip * feat: simplify vector loading functions and improve alignment handling * wip * feat: enhance scale loading mask with quantization block size validation * wip * feat: implement make_scale_load_mask function and refactor vector handling in vec_ops * feat: enhance load_dual_block_generic to include scale indices for improved vector loading * revert q8 dequant * wip * feat: optimize dequantization functions by removing unnecessary masking and updating lookup methods * wip * wip * add qurt_mutex * Add DMA transfer class and integrate into thread pool * Enhance DMA transfer functionality by adding support for multiple descriptors and initiating transfers in parallel * fix dma crash * fix failed unit tests * wip * use alignas * Improve DMA transfer error handling and update descriptor completion check * Fix VTCM cache size calculation in element-wise operations * Add cache clean operations before DMA transfers in element-wise operations * reduce cache clean operations * Refactor DMA transfer functions to support 1D operations and rename for clarity * Enhance DMA transfer functionality by adding 2D submission support and improving descriptor initialization * Update read buffer method to support forced invalidation and remove unnecessary invalidation calls in element-wise operations * wip * Improve DMA transfer handling in mul_mat_gemv_impl by replacing memcpy with initiate_dma_row_transfer and adding wait_for_dma logic * fix 2d dma * feat: add DMA plane cache * rename * wip * use memcpy for debug * fix cache plane calc * refactor: remove debug logging from mul_mat_impl and optimize cache handling * rename * fix 2d dma type * refactor: enhance DMA transfer handling in mul_mat_gemv_impl and wait functions * refactor: optimize DMA transfer handling in mul_mat_gemv_impl and wait functions * wip * wip * move op impl into sub dir * add log * fix: correct pointer usage in mul_mat_gemv_impl for next plane access * fix: improve DMA transfer error handling in mul_mat_impl and mul_mat_gemv_impl * fix: fix crash by using the entire row bytes * wip * wip * fix: prevent parallelization for scalar src1 in is_mul_mat_supported * fix: add dimension checks for 2D DMA transfers and fallback to 1D if necessary * wip * fix: enable thread barrier for mul multiplication operations * feat: add synchronization checks for tensor operations and update related functions * wip * fix: remove invalidation flag from get_read_buffer calls in element-wise and matrix multiplication operations * Revert "fix: remove invalidation flag from get_read_buffer calls in element-wise and matrix multiplication operations" This reverts commit af3441e. * wip * wip * add comment * wip

* Add back top_k * Update utils.cpp * Update utils.h --------- Co-authored-by: Bill Hamilton <bill.hamilton@shopify.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* BF16 support on Metal * Faster BF16 Metal dot product --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…-q2_0-thresholds tests: add Q2_0 error thresholds to test-quantize-fns

* add vision batch test Signed-off-by: tianchi <tianchi_cai@icloud.com> * Feat/vision batch encode sync (ggml-org#55) * omni: add opt-in toggle for batched vision slice encoding Re-apply the ViT slice batch-encoding optimization on top of current master as a controllable, opt-in switch. Same-size image slices are encoded in a single batched pass instead of one forward pass per slice, giving ~1.6-2.7x speedup on large / high-res images. Because batched cuBLAS GEMM changes numerics slightly (avg diff ~1e-2) and uses more VRAM, it is disabled by default and gated behind: - vision_set_batch_encode / vision_get_batch_encode (vision_ctx.batch_encode) - common_params.vpm_batch_encode, applied in omni_init - --vision-batch-encode CLI flag When the toggle is off, encode_image_with_vision_chunks now encodes every slice serially (previously only the first slice was encoded when the toggle was off with multiple slices). Also add omni_bench_vision / --bench-vision to compare serial vs batched encoding and verify output parity. VRAM is queried via the ggml CUDA backend API (cudaMemGetInfo); the bench log notes these are device-wide single-point samples, not per-process peak. Signed-off-by: wjinxu <1299461899@qq.com> * docs: document --vision-batch-encode and --bench-vision Add a CLI option table entry and a dedicated "Vision Batch Encoding" section explaining what the optimization does, when to enable it, the numerical/VRAM tradeoffs (why it is off by default), and how to benchmark serial vs batched encoding. Signed-off-by: wjinxu <1299461899@qq.com> --------- Signed-off-by: wjinxu <1299461899@qq.com> --------- Signed-off-by: tianchi <tianchi_cai@icloud.com> Signed-off-by: wjinxu <1299461899@qq.com> Co-authored-by: 王金旭 <105263726+wjinxu@users.noreply.github.com>

…ml-org#56) hit_mask was a uint64_t, capping routed rows at 64. At top-6 that is ~10 tokens, while concurrent decode needs 72 rows at N=2 and 144 at N=4 -- so the row mask, not expert-parallel, was the measured ceiling. Replace it with a fixed 3-word ggml_moe_cache_row_mask (192 rows, 24 B) plus zero/set/bit/set_below/andnot/any helpers. No raw `1ull << row` remains. Fixed width rather than a caller-supplied pointer + length: a mismatched length is a short write on a correctness-critical bitmask and fails silently. The UB is converted in the SAME change as the raise, deliberately: `n_rows == MAX_TOPK ? UINT64_MAX : (UINT64_C(1) << n_rows) - 1` is UB for any n_rows in [64, MAX_TOPK). It was unreachable only because rows above 64 were bypassed first; raising the cap is exactly what makes it live, and it drops rows with no assert. mask_set now also asserts in debug rather than silently ignoring an out-of-range row -- that silent-drop shape is what this removes. CPU MOE_CACHE_MAX_TOPK and CUDA moe_cache_node_rows_max both derive from GGML_MOE_CACHE_MAX_TOPK. Two independent copies of 64 is how a raise on one side lands against a stale mask on the other. Tests. The cap moves the SLOT COUNT, not just a production scratch figure: scratch is subtracted from budget_limit before slots are sized, and each component is grown by required*2 + 256, so small-budget scenarios fall under their slab (shape-liveness/complete-pools need ~4.53 MiB > 4; route-override 8.28 > 8). Those get budget headroom, assertions unchanged. cache-admission-policy needed a real fix, not a budget bump: it pinned slots == 64 and filled exactly 64, which stops meaning "fill the pool" once the arithmetic moves -- and the readmit_after flip only happens when free_slots is empty. It now reads the observed slot count and keeps n_expert > n_slots. New cache-row-mask-wide covers rows 65 (the old UB edge), 72, 144 and 191 across all three words; breaking set_below past word 0 turns it red, so it is not a no-op. Suite 44 OK / 0 FAIL / 1 SKIP (43 baseline + the new scenario). Conversion proven behaviour-preserving: the identical diff at MASK_WORDS=1 (cap 64) is 43 OK / 0 FAIL, stable across three runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…raffic Revises the 16 chosen earlier in this branch. 16 was sized on a single -np 4 reading of mean 9.0 tokens bypassed. Two later workloads disagreed: -np 8 gave mean 16.5, and -np 4 with 200-token completions gave mean 24.1, smallest 24. At 16 that traffic still bypassed entirely -- measured on the composed v1.3 binary once the stats line could actually be read: 12,400 nodes / 296,730 tokens turned away in one run, cache hits 0.0-1.7%, full-fusion 0/0. The cache was inert in the workload it exists to serve. 32 covers all three observations and lands exactly where op-offload takes over (>= 32), closing the dead band rather than halving it. 32 x top-6 = 192 rows = the routed-row cap exactly, which is why ggml-org#56 set the cap at 192 rather than padding it. A wider top-k exceeds the cap and is bypassed by the separate n_rows check rather than overflowing. The cost is per-node pool pressure: 192 rows staged vs 48 at the old cap. That is justified only while the traffic is measurably landing above 16, so the header records what would justify re-pricing it. ⚠️ This decision was only possible after the bypass observability fix -- with the earlier instrumentation every field read zero and the cache looked idle rather than starved. Suite 44 OK / 0 FAIL / 1 SKIP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only).

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change).

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change).

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

Add back top_k

7412189

beiller mentioned this pull request Mar 12, 2023

Maybe lower default temp and switch to top_k 40 #42

Closed

ggerganov reviewed Mar 12, 2023

View reviewed changes

Comment thread utils.cpp Resolved

Update utils.cpp

2913d75

ggerganov reviewed Mar 12, 2023

View reviewed changes

Comment thread utils.h Resolved

Update utils.h

90b9b5b

ggerganov merged commit 02f0c6f into ggml-org:master Mar 12, 2023

This was referenced Mar 14, 2023

65B model giving incorect output #69

Closed

Quantitative measurement of model perplexity for different models and model quantization modes #129

Closed

Hades32 pushed a commit to Hades32/llama.cpp that referenced this pull request Mar 21, 2023

Merge pull request ggml-org#56 from anzz1/patch-2

ad0f01b

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

BF16 support on Metal (ggml-org#56)

bd4243b

* BF16 support on Metal * Faster BF16 Metal dot product --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

HEchternacht pushed a commit to HEchternacht/llama.cpp that referenced this pull request Jul 18, 2026

Merge pull request ggml-org#56 from PrismML-Eng/fix/test-quantize-fns…

ceb0790

…-q2_0-thresholds tests: add Q2_0 error thresholds to test-quantize-fns

Copilot AI mentioned this pull request Aug 26, 2026

actions: set explicit workflow token permissions kerrrang9214-tech/llama.cpp#4

Draft

Navigation Menu

Repository navigation

Add back top_k - #56

Add back top_k#56
ggerganov merged 3 commits into
ggml-org:masterfrom
beiller:feature/top_k

beiller commented Mar 12, 2023

Uh oh!

Uh oh!

Uh oh!

gjmulder commented Mar 14, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

beiller commented Mar 12, 2023

Uh oh!

Uh oh!

Uh oh!

gjmulder commented Mar 14, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants