Making weights loading faster by oKatanaaa · Pull Request #216 · ggml-org/llama.cpp

oKatanaaa · 2023-03-16T22:39:40Z

Tried to address slow weights loading. 7B is okay, but 13B is really slow (several minutes), hard to experiment/prototype with larger models.

Replaced std::ifstream with C-style file reading using fopen. Got a considerable boost in loading performance: 3x to 10x faster on my machine (measuring results were kinda inconsistent, but it is definitely a lot faster than before).

I made sure the weights are correctly loaded: fixed the seed, gave the same prompts - model gives the same output, everything is good.

Also increased the buffer size from 1024*1024 to 128*1024*1024 (see line 102) which gave a slight boost as well. Though I am not sure whether it is optimal for edge devices like Raspberry Pi (if it's of any concern).

Kangmo

looks good to me

sw · 2023-03-18T11:34:36Z

Is this faster than using mmap (see #91 / #150)?

oKatanaaa · 2023-03-18T13:31:46Z

Is this faster than using mmap (see #91 / #150)?

Just tested out #150 implementation, it was a lot slower than simple fopen. I believe #150 shoud've been faster on subsequent runs, but I didn't see any perceptible speed up, maybe just a little.

Piezoid · 2023-03-18T14:15:54Z

Is this faster than using mmap (see #91 / #150)?

By implementing #91 correctly, we can prevent creating any redundant copies of the model. This is crucial because if we duplicate it and there's enough space to store just one instance in memory, we run the risk of removing it from the operating system's file system cache.

apaz-cli · 2023-03-18T18:03:59Z

@oKatanaaa switching between std::ifstream and FILE* should make no measurable difference. They are both tunable, do conceptually the exact same thing, and support (almost) exactly the same set of operations. The main bottleneck is the time spent inside of system calls reading from disk, and they make those calls exactly or almost exactly the same way. The first implementations literally called fopen() directly. I forget if they still do.

Your testing methodology is probably at fault. Files that are opened once are likely to be opened again soon after. OSes like to cache the contents of files (and shared memory pages also) so that subsequent reads are faster. On linux you can flush these file and page caches with:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null

You may have noticed that with the vanilla build on master loading the model sometimes skips ahead by quite a few dots in the progress bar, usually at the beginning, but sometimes a few times in the middle. This is why.

That said, I recommend that you don't do this. Justine's version of #150 is much better than mine. To use it, you can just clone the repo and switch to the mmap branch. If you load the model once, subsequent loads happen instantaneously, as long as the kernel hasn't had any reason to evict the model to free up space.

As discussed on #91, it's not a perfect solution, but it's pretty great. Actually properly fixing this problem will require making changes to the ggml file format, and possibly the library. I recommend using Justine's branch in the meantime.

oKatanaaa · 2023-03-18T21:50:53Z

@apaz-cli Thanks for the input. I initially thought the same because performance comparisons for fstream vs fopen I found were not really consistent (and very speculative) and depended on a particular case. Though intuition was telling me fopen should be faster for binaries (as I recall from my experience). So I decided to see if anything changes and implemented fopen model load. To my surprise, the performance gains were substantial, that's why I decided to make a pull request.

Speaking of the mmap implementation, as far as I understand (though I don't understand a thing about mmap really) it simply creates a large binary where model weights are stored contiguiosly (I guess?) and uses it to make faster subsequent loads. The model loading code is not changed at all, only the fancy memory allocation stuff, meaning that my change and Justine's code are perpendicual and can benefit each other.

Getting back to performance comparison of fstream vs fopen (and also mmap), I did some testing and here are results:

Implemetation	First load	Subsequent load	Read speed (mb/s) on subsequent run
master	1m 49s	1m 50s	50
mmap	8m 50s	1m 10s	20-30
fopen	13s	13s	200-300
mmap + fopen	6m 20s	1m 8s	20-30

My machine:

CPU: Ryzen 7 4800H
RAM: 32GB, 3200MHz (not sure if it's dual channel)
SSD: ADATA SX6000PNP 1TB

Testing was done using 13B model. I measured time (both for first and subsequent loads) till the model processed the first token. I also monitored my disk current read/write speed. I made sure to run sync && echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null for each 'first load" and did several subsequent loads picking an average result. Deviations from run to run weren't too large, so the numbers in the table should tell a general picture of things.
I also implemented mmap + fopen by combining my code with the mmap implementation and it did give a slight boost. Though mmap implementation does a lot of writes to memory (page file I guess?) taking up a huge portion of all loading time, so on the first run we'll have to wait still.

An interesting thing is that subsequent runs on mmap still run a lot slower than fopen. I am not sure what's going on as I expected an instantaneous load (but intuition tells me that custom malloc and realloc read required tensors from disk on request as the model is being evaluated during the first run, please correct me if I'm wrong).

As a summary:

fopen is indeed faster than fstream in this case, at least on my machine. It would be really good if other people could confirm performance gains on their machines.
fopen (current PR code) does not conflict with the mmap implementation in any way and both can be merged bringing benefits of both sides.
Current implementation of mmap may not work as expected (still slow subsequent runs), though there might be something up with my machine. Again, it'd be good to have other people verify these results.

jart · 2023-03-18T23:37:19Z

When you say mmap, are you talking about #150? Because that was the first iteration of our mmap() work. The second iteration we did is in the https://github.com/ggerganov/llama.cpp/tree/mmap branch. That code doesn't have a load phase at all in subsequent runs.

It's also important to specify (1) what operating system you're using, and (2) what type of weights you're using. Even with the code in the master branch, the 13B model, with either q4 or f16 weights, your computer should not be taking 1+ minutes on subsequent runs. On my $1000 workstation, subsequent runs with the old method take ~3 seconds to load 13B F16. With the new method, it loads in 0 seconds. The only way I could see a computer taking >1m given your hardware, would be if it were running Windows, or if you had so many tabs open in your browser that there was practically no memory available for caches. For example, my workstation is headless and I use it via SSH, so that things like Chrome don't influence benchmarks.

oKatanaaa · 2023-03-19T07:26:26Z

@jart I was using the mmap branch. I believe I earlier tried #150 and it didn't work for me well.

(1) I am on a Windows 11 machine, but building and running code inside a docker container (kitware/cmake:ci-debian10-x86_64-2023-03-08). Just didn't want to have headaches from win toolchain (as I never worked with cmake on win), so opted for the simplest route.
(2) Q4 weights.

Anyways, I tried building for win and to my surprise fopen and master implementations matched loading speeds (4 seconds, blazing fast). Now it completely agrees with @apaz-cli argument. It seems slow loading times are due to I/O problems inside the container and I am not sure why there is a such dramatic difference.

Sadly I couldn't build mmap for win.

A little bit of googling gave these issues where people were struggling with slow I/O for mounted volumes (though on Mac):

It also seems like the same issue might be present on Linux machines as well: https://forums.docker.com/t/docker-extremely-slow-on-linux-and-windows/129752

Seems like everything works well when building stuff 'natively'. So the benefits of this PR are not clear in native settings, but it might be an option in others (again, only to have other people try out the code and see if it works better for them).

jart · 2023-03-19T08:26:33Z

It's good that you're running Windows, because where we really need your help right now is in making sure our mmap branch is able to build with MSVC. Neither @apaz-cli or myself have access to MSVC at the moment. I wrote some polyfills and pushed them to the branch that I believe will work with WIN32. However the change was written a priori and needs to be cleaned up and debugged by someone who has MSVC. Otherwise we won't be able to merge the improvements into the master branch.

We could still possibly find ways to improve the experience for people who are using virtual file systems like Docker. However I would want to see a better theory that explains why the performance is bad and why it's actionable on our part. We can't help you find those answers. For example, I'm not convinced that using C stdio rather than C++ stdio is itself in any way different. An example of something that would possibly convince me, would be like, "if we use this stride obtained from the statfs() block size then we see optimal i/o behavior" or "if we have a second thread prefaulting pages".

Another thing that would convince me is if you showed me a perf record profiling report showing, "here we see my glibc version of libstdc++ has a bug where it spends all its time chewing up CPU rather than actually doing I/O therefore we should just use C stdio." But until that happens, I can't merge a change like this.

oKatanaaa · 2023-03-19T12:18:13Z

@jart I'd be glad to be of any help with debugging mmap for WIN32. What's needed right now is to clone mmap branch, (try to) compile it using MSVC and see if there are any bugs in there, is that right? And where do I report my findings? #91?

I will also try to do some investigation with regards to loading performance inside docker.

apaz-cli · 2023-03-19T17:50:22Z

@oKatanaaa The branch is already in the repo. Just git pull origin and git checkout mmap.

jart · 2023-03-19T18:37:25Z

You can report in #91. You can also send a pull request that targets the mmap branch.

anzz1

This is great !

However, merge conflicts require solving.

CoderRC · 2023-03-28T01:15:20Z

I have created my own library that implements mmap using mingw32 that makes this project maintainable for windows. It is possible to compile the program using library from https://github.com/CoderRC/libmingw32_extended, make changes like in #564 and the specific make command below:
make LDFLAGS='-D_POSIX_MAPPED_FILES -lmingw32_extended'

anzz1 · 2023-03-29T03:48:37Z

There is no need to use a separate library to replace C++ fstreams with C file streams, as C file streams work perfectly in Windows too. You are conflating two things in the comments now, which are using mmap() , which is non-portable and requires an external library or polyfill, while the PR is about using fopen() and fread() , both of which are portable and do not require any additions.

jart · 2023-03-29T06:40:27Z

@oKatanaaa do you still want to move forward on this PR? I'm still not convinced that c++ stl vs. c stdio is going to make a measurable difference, unless our goal was to convert the c++ code to c. Our mmap() work on the other hand, makes weights load 10x faster for me on Windows now. And 100x faster on Linux.

jart · 2023-03-29T06:48:41Z

@anzz1 I looked at @CoderRC's project and it looks pretty good and it's unfair to say mmap() isn't portable because it's standardized by the IEEE. You'll also be happy to hear we now have our own working WIN32 mmap() polyfill, in our mmap branch: https://github.com/ggerganov/llama.cpp/blob/mmap/mmap.c Me and @oKatanaaa spent the past few weeks building that. As I mentioned in my earlier comment, it's going to help make loading larger models with multi-dimensional tensors (e.g. 13B) about 10x to 100x faster, in terms of wall time. It doesn't require any libraries or dependencies. You're encouraged to take a look at the code and share your feedback!

oKatanaaa · 2023-03-30T07:45:14Z

@oKatanaaa do you still want to move forward on this PR? I'm still not convinced that c++ stl vs. c stdio is going to make a measurable difference, unless our goal was to convert the c++ code to c. Our mmap() work on the other hand, makes weights load 10x faster for me on Windows now. And 100x faster on Linux.

I still want to tinker with the code a bit and find the reasons of slow loading inside docker on win, but I am focusing on other things at the moment, so we can close the PR. I'll try to do some research regarding this problem once I have some time. If I find anything I'll make a new PR with new fixes.

…MD (ggml-org#216) Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…y GB10 (+6% prefill on Ada) (ggml-org#216) * cuda: use the 4-column GDN warp layout on all Ampere+ NVIDIA, not only GB10 (+6% prefill on Ada) The S_v == 128 GATED_DELTA_NET kernel has a 4-columns-per-warp variant that was gated to GGML_CUDA_CC_DGX_SPARK. The recurrence is serial over tokens, so prefill on a hybrid model lives or dies on this kernel's per-step efficiency, and the wider layout is a straight win on consumer Ampere/Ada as well. Host launch geometry and compiled kernel now derive the column count from one constexpr predicate, gdn_cols_per_warp(arch, S_v, KDA): the kernel evaluates it on __CUDA_ARCH__, the host on ggml_cuda_highest_compiled_arch(cc) (the arch the device will actually run, not the raw runtime capability), and only for GGML_CUDA_CC_IS_NVIDIA(cc). Under GGML_USE_HIP / GGML_USE_MUSA the predicate is 1 on both sides, so AMD capability values (which carry an offset above GGML_CUDA_CC_AMPERE) can no longer make the host launch 8 z-blocks of 4 columns against a 1-column kernel. Fixes the geometry mismatch raised in review. RTX 4070 12 GB, Bonsai 2 27B PTQ1_0: pp2048 1220 -> 1297 t/s, decode unchanged. Maintainer paired runs at the previous revision: pp512 +3.7% RTX 3090, +1.3% RTX 4090, +0.6% H100, +0.3% RTX 5090; tg128 within noise everywhere. test-backend-ops CUDA0 vs CPU: GATED_DELTA_NET 39/39 supported cases. * cuda: keep only the GDN cols-per-warp launch invariant in comments. --------- Co-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com>

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com>

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change).

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change).

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. Two halves, both inert with a single drafting sequence: (1) draft-mtp-adaptive raises every drafting sequence's cap to the deepest cap in the SAME draft() call. The cap only ever moves UP, and each sequence keeps its own hard bound (--spec-draft-n-max, itself capped by the MTP layer count, and the caller's per-slot n_draft_max from the remaining context), so no sequence is ever asked for a depth it would not draft when it runs alone. (2) common_speculative_draft() cuts the drafts to the shortest one when the lengths still differ, which covers a different implementation on each slot (an n-gram hit on one, an MTP draft on the other, and an n-gram match has no cap at all) and a slot whose own n_max is smaller (the response is running out of tokens). It tells the implementation that produced the cut draft (draft_truncated()) so its acceptance statistics count only the positions the target actually verified. A slot that drafted NOTHING is left alone: cutting the neighbour's good draft to nothing is the worse trade, and those steps still split. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Measured on Qwen3.8-27B UD-Q4_K_XL, 2 GPU (the x8 pair) -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of the concurrent pair, one high-acceptance prompt (recall) and one low (reasoning): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76 t/s. At 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %), per slot 55/45 -> 110/72 -- the penalty scales with the number of devices the duplicated pass spans, which is why both are gated. On the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU) 116.2 -> 149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts, which converge on one depth without the change) is unchanged: -0.4..-2.1 %. A single drafting sequence is untouched by construction, and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh, 2 and 3 GPUs plus the host-expert arm. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used. Amended (r132): the split-state cache-hit path returns split_state_entry::get() directly instead of copying the state out of a named local, so a compiler that does not elide NRVO out of this multi-return function builds it once in the caller's object. Same values, same assertions. Contributed in PR ggml-org#213 (@briansp2020), where the copy was ~half of this function's self time; on this tree it is measured neutral -- clang 24 and clang 23 at -O3 already build the entry directly in the return slot (identical function size and codegen; the function's sampled self time is 1.6-1.9 % before and 1.6 % after, within run-to-run noise) and 12/12 same-seed configs are byte-identical. Taken as an explicit-intent change, not as a measured speedup. Co-authored-by: Brian <bsp2020@gmail.com> Meta: two halves of issue ggml-org#232 (the `ggml_new_object: not enough space in the context's memory pool (needed 2261360, available 2260992)` abort, reported by @MrDrMcCoy). (1) A simple-tensor container grows instead of aborting. One generation (everything registered between two graph rebuilds) can register more simple tensors than one context holds: a state restore creates one view per KV cache chunk per layer (`llama_io_read_device`) and every graph shape contributes its own. A full context is now followed by another one, and a rebuild resets and releases every context in the list, so the 128-views-per-static-tensor budget is a first-context size rather than a cliff. `GGML_META_COMPUTE_HEADROOM` overrides that size (diagnostic/testing only: a smaller value only makes the container grow sooner and never changes an output). `ggml_new_object`'s overflow warning now also reports the context, its `mem_size`, its object count and the bytes used, so a future report of this class is diagnosable. (2) Views created *between* graph evals go to a scratch container (`ggml_backend_meta_set_scratch_views` / `ggml_backend_meta_reset_scratch_views`, caller-owned lifetime). This is the reported trigger: `llama_kv_cache::park_move` moves a run in chunks no larger than the gap to its destination and creates 2 views per (layer, K/V) per chunk, so a nearly-full unified cache (two ~258k-token prompts in 524288 cells) parks in dozens of tiny chunks -- hundreds of thousands of views in one move -- and they all landed in the per-generation container, which nothing reclaims between graphs. The reporter's diagnosis and patch; the copies are synchronous (a meta buffer has no `.cpy_tensor`, so `ggml_backend_tensor_copy` stages through the host), so no scratch view outlives its reset. Validation (3x R9700 gfx1201, ROCm 10.1.0, Flash-Next GSQ-RCO IQ3_XXS, `-sm tensor`). The trigger reproduces exactly at the default headroom (2 slots, `-c 32768`, two ~16.3k-token prompts, so the park gap is ~100 cells): the r149 build aborts with `needed 2261360, available 2260992` in `llama_kv_cache::seq_park` -> `llama_kv_cache::park_move`; with (1) alone the run completes and the attention-KV container grows 6144 -> 12288 (the indexer's 3072 -> 6144); with (1)+(2) it completes with **zero** growth events, i.e. parking no longer touches the per-generation container at all. Byte-identical output across r149 / (1) / (1)+(2) on that flow; `llama-perplexity` wikitext-2 `PPL = 2.0901 +/- 0.05592` on all three builds; `GGML_META_COMPUTE_HEADROOM=1` aborts the r149 build at `sched_reserve` and completes here (105 growth events) with identical text; the 2-slot 524288-cell flow (two ~240k-token prompts, parking + checkpoint restores) is text-identical and clean; `gate-park-idle-slots.sh` PASS (6 park / 2 unpark moves, park == lazyoff on all four requests). Also hardened while reading the same file: `get_node_aux()` now asserts against its pool (`nodes_aux`, `3 * n_reduce_steps * max_subgraphs` tensors per device). Its ALL_GATHER sibling already asserted; an over-budget graph would have written past the pool instead of reporting (not reachable on any recorded config, no behaviour change). Amended r153 (issue ggml-org#234): the aligned verify-batch depth is bounded by this device's MoE expert-cache verify band. Block 02 aligns the drafts of the sequences that verify in the same step so the target batch stays one ubatch, but on a host-expert config a batch wider than the band leaves the arena and streams the routed experts from the pinned host master on every step (see common_speculative_clamp_ngram_n_max), and alignment turns a wide batch from an occasional step into the steady state. The aligned depth is therefore bounded by `band / n_drafting - 1` rows per sequence, because the whole ubatch has to fit: a two-slot verify on RDNA4 (band 16) aligns to at most 7 each, not 15 each. A sequence already deeper than that keeps its own cap -- the band bound may only WIDEN a shallow draft, never shorten a deep one, which is what keeps every single-sequence run bit-identical -- so it needs no new knob; LLAMA_SPEC_ALIGN=0 (block 02) disables the alignment with it. The value is read once, after the target context exists, from llama_model_moe_cache_band(), so a config where the slab declined (no arena) is unbounded. Measured with the bound in place on 35B-A3B UD-Q4_K_M -ncmoe 40, 2 GPU, --parallel 2 --kv-unified: the diverging pair goes 116.2 -> 149.0 t/s (+28 %) and the split decode steps 58 -> 0, with the single-slot requests byte-identical on/off. Issue ggml-org#234.

@briansp2020

Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234. Amended r185 (PR ggml-org#169 re-assessment, @briansp2020's finding): bound the aligned depth by a VERIFY WIDTH where there is no expert-cache arena. r153's bound was the arena's band (`band / n_drafting - 1`), and r176 made `band` answer only for a config with host-resident experts -- so on a band-0 config (all-in-VRAM, -sm layer, or a model with no routed expert) a multi-slot draft was UNBOUNDED: four slots at --spec-draft-n-max 7 present ~28-32 rows, the batch leaves the decode kernels' small-batch families, and the DEFAULT recommended speculator (`draft-mtp-adaptive`) runs 9.5-31.8 % slower than the same arm with the depth bounded at 16 rows. The bound is now `verify width / n_drafting - 1`, where the width is the arena band when an arena is active (unchanged) and LLAMA_SPEC_ALIGN_WIDTH_BAND (default 16, 0 = off) otherwise. 16 and not 8: at the two widths the fixed-depth arms measure 16 rows AHEAD on 3 of 4 shapes (35B-A3B 3 GPU 432.7 vs 389.8 t/s, 2 GPU 387.2 vs 359.6, dense 27B 3 GPU 240.1 vs 211.2; qwen4exp Flash-Next GSQ-IQ3_XXS 2 GPU + resident PLE is the exception, 204.6 vs 220.8), so PR ggml-org#169's "above ~8 rows each extra draft token costs more than it gains" does not hold -- the *unbounded* width was the defect, not width 8. 4 concurrent 300-token greedy requests, one binary, band on vs off: 35B-A3B 3 GPU 330.8 -> 434.7 t/s (+31.4 %), qwen4exp IQ3_XXS 2 GPU + PLE 175.5 -> 208.9 (+19.0 %), dense 27B 3 GPU 218.5 -> 249.6 (+14.2 %), and a fixed `draft-mtp` n_max 7 arm 310.9 -> 362.8 (+16.7 %). 3 slots also gains (+16.9 %); 1 and 2 slots are INERT (16 / 2 - 1 = 7 = the default n_max), and the 2-slot text stays byte-identical, so it is a pure multi-slot change. A host-expert config takes the arena band and is untouched (same code path, verified in the log). Gate: scripts/gate-spec-width-band.sh.

oKatanaaa added 2 commits March 16, 2023 22:22

refactor: make weights load faster

7252a2b

minor change

fc924e9

gjmulder added enhancement New feature or request performance Speed related topics labels Mar 17, 2023

Kangmo approved these changes Mar 18, 2023

View reviewed changes

merge

b10383f

anzz1 approved these changes Mar 25, 2023

View reviewed changes

anzz1 mentioned this pull request Mar 26, 2023

Ability for ./main to keep the model in memory and pass it more text #23

Closed

anzz1 mentioned this pull request Mar 29, 2023

Add support for memory mapping models #586

Closed

1 of 4 tasks

anzz1 mentioned this pull request Mar 29, 2023

Enable -std= for cmake builds, fix warnings #598

Merged

oKatanaaa closed this Mar 30, 2023

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

Hopefully this really fixes the confusion between AVX512 and FANCY_SI…

17d4387

…MD (ggml-org#216) Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

Conversation

oKatanaaa commented Mar 16, 2023

Uh oh!

Kangmo left a comment

Choose a reason for hiding this comment

Uh oh!

sw commented Mar 18, 2023

Uh oh!

oKatanaaa commented Mar 18, 2023

Uh oh!

Piezoid commented Mar 18, 2023

Uh oh!

apaz-cli commented Mar 18, 2023

Uh oh!

oKatanaaa commented Mar 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jart commented Mar 18, 2023

Uh oh!

oKatanaaa commented Mar 19, 2023

Uh oh!

jart commented Mar 19, 2023

Uh oh!

oKatanaaa commented Mar 19, 2023

Uh oh!

apaz-cli commented Mar 19, 2023

Uh oh!

jart commented Mar 19, 2023

Uh oh!

anzz1 left a comment

Choose a reason for hiding this comment

Uh oh!

CoderRC commented Mar 28, 2023

Uh oh!

anzz1 commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jart commented Mar 29, 2023

Uh oh!

jart commented Mar 29, 2023

Uh oh!

oKatanaaa commented Mar 30, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

oKatanaaa commented Mar 18, 2023 •

edited

Loading

anzz1 commented Mar 29, 2023 •

edited

Loading