fix coloring of last n_batch of prompt by bitRAKE · Pull Request #217 · ggml-org/llama.cpp

bitRAKE · 2023-03-17T00:52:41Z

https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L980-L983
This can fail to colorize the last params.n_batch part of the prompt correctly because embd was just loaded with those tokens and not printed, yet.

j-f1 · 2023-03-17T01:39:04Z

Can you rebase? It looks like there are some unrelated commits in the history.

* Fix ggml-org#217 * Remove stuff commited by mistake --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

Based on ggml-org#217 (ken00H): skip speculative init and its startup warnings when no non-NONE speculative type is configured. Do NOT skip common_context_can_seq_rm(ctx_tgt) — ctx_tgt_seq_rm_type also drives completion checkpointing (FULL / RS models), independent of speculative decoding. Leaving it at the default NO would disable those checkpoints for non-spec runs. TurboQuant: no kernel impact.

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881.

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged.

…b and cache engine) old-06 (generic MoE expert-cache iface, scheduler routed-expert rebalance, Meta delegation, the movable-boundary slab + allocator fail-soft, the host-expert pinned loader / --host-experts, the expert-cache engine, the h2d staging ring, GGML_ENV_STR) + old-09b (slab headroom) + old-11 (graph handling for expert-offload decode) + old-15's slab/cache extensions (narrow2, host pool, stage_gather, the fattn_stage arena accounting, the cache-aware mmvq/mmq reads and the try_fuse stand-down). The qwen4exp/HC and cache-blocks (cpy_batch) try_fuse arms are still stripped (they belong to 10 qwen4exp); the cache-aware arms present here are the slab/engine ones. Gates: build all OK; 35B-A3B -ncmoe 40 cache off == on (identical greedy text). Env-read hygiene (no behaviour change): resolve the environment variables this block adds on hot paths once into function-local statics instead of calling getenv() per alloc/reserve/ubatch -- GGML_SCHED_STAGE, LLAMA_DROP_COMPUTE_BUFFERS, GGML_CUDA_WORK_ALLOC_DEBUG, GGML_CUDA_CACHE_GEOM_DEBUG, GGML_CUDA_POOL_MIN_FREE_MIB, MOE_EXPERT_CACHE_VALIDATE_LAYER. Also make GGML_ENV_STR cache the value (std::string) rather than the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Prefill cache staging (r83): a whole-table staging copy of a host-resident expert table is now filled from the cache arena for the experts already resident and from the pinned host master for the rest, instead of re-uploading every expert over PCIe. The source map is a per-table "prefill residency snapshot": a small device copy of the authoritative host `expert_slot`, rebuilt whenever the cache writes (`g_write_seq`), the arena moves or the slot count changes, never mutated by decode, and not the removed device-remap policy. The destination is the ordinary staged table in expert order, so every consumer is unchanged and the bytes are identical. DEFAULT ON; kill switch `GGML_MOE_CACHE_STAGE=0` restores the plain whole-table upload. A table the slab stands down mid-prefill, an unpinned host master or a geometry mismatch declines and falls back to the plain upload. `moe_cache_report` prints the staged arena/host split. Measured (2x R9700 gfx1201, Qwen3.8-Flash-Next IQ4_NL 176.9B-A3B, `-sm tensor -ncmoe 48`, 32k ctx): prefill 1092 -> 1436 t/s (16k-token prompt, `-ub 4096`), 583 -> 839 (`-ub 2048`), 1181 -> 1585 (31.6k); `-sm layer` 742 -> 1269. Staged H2D falls to 27% of the table (arena 46.6 GiB, 71.9% residency). Greedy text byte-identical to r82; prefill-logit mean KLD 0 / same-top-p 100%. Slab drop reference (r85): the prefill -> decode drop recorded the work region it had just re-reserved as its "settled narrow" reference, but that re-reserve is the `n_rs_batch` shape, narrower than the real MTP decode/verify graph. Every following decode pass therefore looked "materially wider" and dropped again: measured one drop per decode step, 66 tables re-armed and ~3.65 GiB/device of arena evicted each time (decode 17-25 t/s instead of 45). The reference is now max(work region observed before the drop, shape the drop re-reserved). Fold (r86): beta 12, slab-transient-absorption (common half). The draft reserve is derived from the draft GGUF instead of assumed, the draft model is loaded before the target context so the slab measures its per-device footprint, and the common-layer and all-reduce transients are served by the slab. Coherence identical, prefill-logit KLD 0.000000 / same-top-p 100 %, adaptive-MTP 4 axes + phase probe exactly equal on the development box. Fold (r87): force the post-prefill drop on a wide -> narrow transition. r85 recorded `slab_narrow_boundary = max(re-reserved narrow, drop_observed)`, but at a prefill -> decode drop `drop_observed` is the WIDE prefill region, so an equally-wide second prefill compared equal and never dropped: the wide work buffer stayed allocated, the arena was not re-armed, and every request after the first decoded at ~half speed (Flash-Next IQ4_NL field config, 2 GPUs -sm tensor -ncmoe 46 -ub 6144 + MTP n6: 68.5 -> 35.7 t/s; with the fix 68.1 -> 69.4). Fold (r91): never auto-size the MoE expert arena on a shared host/device memory pool. On an integrated GPU (gfx1151/Strix Halo) the device "VRAM" window is the same physical pool as host RAM, so AUTO arena sizing and the movable-boundary slab (free - reserve) both map the machine: measured 121 GiB resident + 13 GiB swap, and bounding the arena alone hands the same overcommit to the slab. `moe_cache_preflight` now declines the cache on `prop.integrated`, keyed before any table registers, so the run is the documented host-expert path and the wrapper disarms the slab; `common/fit.cpp` stops reserving slab headroom + arena for such a device. Opt-in override `MOE_EXPERT_CACHE_UNIFIED=1` requires an explicit `MOE_EXPERT_CACHE_MIB` AND an explicit `GGML_CUDA_SLAB_RESERVE_MIB` (bounded by construction). A discrete GPU is untouched: gfx1201 35B-A3B `-ncmoe 40` loads with identical preflight/SLAB_INIT/arena lines; coherence and the prefill-logit gate are unchanged (guard == `MOE_EXPERT_CACHE_MIB=0`). Record: archive/work/umem-arena-guard/. Fold (r95): rdna3-verify-fusions (PR ggml-org#193, briansp2020): the batched f32 CPY run is admitted on RDNA3_0 as well (GGML_CUDA_CC_IS_RDNA3_0 alongside GGML_CUDA_CC_IS_RDNA4; RDNA3_5 stays out). One launch for up to 16 same-layout copies, element copies in ggml_cpy's flat order: bit-identical. The fusion is inactive where a split's concurrent events are live, and GGML_CUDA_FUSE_CPY_BATCH=0 still turns it off. Fold (r96, PR ggml-org#186, briansp2020): in `ggml_backend_sched_split_graph`, a second `MUL_MAT_ID` consumer of the expert weights the current split already stages starts a split of its own, so the re-registration path stages them again for that consumer's own routing. With both consumers in one split neither can be served by the expert cache (the arena holds one routing's slots) and the scheduler's guarded fast paths decline, so the whole table is copied for every such prompt chunk: qwen4exp's gathered logits tail next to its full-row MTP export tail meant the last layer's 1062 MiB table per chunk (Flash-Next `-sm tensor -ncmoe 10`: -30 % / -47 % TTFT on the follow-up turns, bit-identical text), and on `-sm layer` the second consumer read the arena slots staged for the first consumer's routing, which silently produced non-greedy-pure text with 0.00000 draft acceptance. `GGML_SCHED_MOE_DUP_SPLIT=0` restores the merged split (A/B and bisection). Fold (r96, PR ggml-org#189, briansp2020): the meta backend's `get_tensor_async` shares the synchronous read's split arithmetic instead of asserting `offset == 0` and one segment, so a caller can batch many per-device reads behind one synchronize (a context checkpoint's per-sequence cells). Calls that worked before issue the same copies as before. Fold (r99, TODO ggml-org#56): the scheduler must invoke the copy callback again, so the upstream `llama_moe_cache` (`--moe-cache-mib`) works. This block's rewrite of `ggml_backend_sched_copy_input` kept no `sched->callback_copy` call, and that call is the only thing in the tree that drives `llama_moe_cache::copy()`: the slot map stayed `0xff` (-1) while the graph had already swapped the routed op's `src0` for the cache bank and its ids for `GET_ROWS(slot_map, ids)`, so `MUL_MAT_ID` indexed the bank with -1 (wrong values where the page below the bank is mapped, an HSA page fault where it is not). The call is restored at base's position (after the wait-before-overwrite) and in the branch with no `MUL_MAT_ID` consumer, which is where the slot map lands (its consumer is a `GET_ROWS`); the delivery's own expert branch (device gather, cache take-over, pruning) is byte-for-byte unchanged, so no host expert table can reach the callback. `llama_context::sched_copy_experts` now answers only for the upstream cache: its per-op staging/pruning copy has been dead since this block's scheduler took that over, and would double-stage every host expert table if it ever ran again. Inert unless `--moe-cache-mib` is set (`moe_cache` is then null and the callback returns false). Gates (gfx1201, ROCm 10.1.0): `test-llama-archs -s 42` **375/375** (was 373/375; `deepseek2`'s `MoE cache` row 2.10e-10, the upstream base's own value, was 4.02e-03), also 375/375 with `GGML_CUDA_ASYNC_INPUT=0`; host-expert greedy text byte-identical to r98 (35B-A3B `-ncmoe 40`); prefill-logit KLD 0.000000 / 100.000 %. The upstream cache's large-batch `copy_experts()` arm stays unreached (this block's scheduler owns that pruning); values are identical either way, so it is a missing upstream optimisation on a path the delivery serves with its own cache, not a correctness gap. `--moe-cache-mib` is now correct but slower than `MOE_EXPERT_CACHE_MIB` on this box (decode 50.1 -> 37.8 t/s on 35B-A3B `-ncmoe 40`), so it stays documented as "works, not recommended". Split-state walk scratch (r100, issue ggml-org#184): the per-buffer-context split-state cache pre-warm walks the not-yet-cached ancestors on an explicit stack, and its wstack/wseen containers allocated one heap node per visited tensor (plus the rehash and the frees) on every graph re-allocation -- ~4.7k split states and ~150k tensors per re-allocation in the reporter's 4-stream workload, where 94% of the mallocs under ggml_gallocr_alloc_graph came from that set. Both containers now use a reused per-thread monotonic_buffer_resource (1 MiB, released per walk); only the outermost call walks, so the resource is never re-entered, the walk visits the same tensors in the same order and computes the same states, and an exhausted arena falls back to the upstream allocator. GGML_META_SS_SCRATCH=0 restores the heap containers (A/B only). Second-scheduler slot for DFlash/DFlash2 drafts (issue ggml-org#208): the draft context's second scheduler (LLAMA_DRAFT_SCHED2) was selected with `n_outputs > 0`. That fits MTP (a no-output ingest graph and a draft step with an output) but not a DFlash2 draft block, which has no outputs either: its lattice is read from the nextn embeddings, not from logits (`output = !is_dflash2`). The KV injection and the noise-block draft step therefore shared one scheduler AND one `gf_res_prev` slot, so they overwrote each other's graph every cycle and the draft context rebuilt and re-allocated both of them on every verify step (measured 0 of 362 draft ubatches reused). `sched_alt_for()` now selects the slot from the batch kind for LLM_ARCH_DFLASH -- the injection is the batch that carries the target features (host embd rows, or the F1 device injection whose token ids are target rows), the draft step is the token batch -- and keeps `n_outputs` for every other arch, where the batch kind would merge MTP's ingest and step graphs instead of separating them. `get_gf_res_prev()` takes the same decision, so the scheduler and the graph-reuse slot cannot disagree. Measured (gfx1201, 512 greedy tokens, `prompts/prose-rdna-boosts.txt`): draft graph reuse 0 -> 97.8 % (dense 27B UD-Q4_K_XL 2 GPU tensor) and 97.9 % (35B-A3B Q4_K_M `-ncmoe 40`); decode +1.4 % (27B 2 GPU), +1.8 % (27B 3 GPU), +4.0 % (35B-A3B host experts), +1.3 % (host injection, GGML_LF_DFLASH_DEV=0), +0.5 % (27B `-sm layer`), +0.2 % (27B 1 GPU); a 2-slot unified server with two concurrent requests is identical per slot (text, acceptance 0.66894 / 0.73801) at +1.0 / +1.9 %. Generated text is byte-identical in every arm, the dense 4B coherence hash is unchanged, and the adaptive-MTP four-axis gate (27B, `-n 3000`, reasoning pinned per axis) is identical to the pre-fix build on all four axes (t/s, acceptance and text). `LLAMA_DRAFT_SCHED2=0` reproduces the old behaviour exactly. Per-block build (blocks own their code): the identity-preference layer-drop ranking (r117) called `heat_rank_vec()`, which block 12 introduces with the band-split heat prior (r114), so block 09 alone did not compile (`use of undeclared identifier 'heat_rank_vec'`) since r117. This block now ranks by `t.heat`, which is byte-identical in this tree (there is no decode counter yet), and block 12 switches this site over along with the two it already owns. `gate-per-block-build.sh` selects block 09 again. Verify-width band clamp (TODO ggml-org#59, ngram-mod x host experts): the expert cache serves a routed `MUL_MAT_ID` from its arena only up to its decode/verify band (`moe_cache_max_tok_dev()`: 16 tokens on RDNA4, 8 on RDNA3/NVIDIA). A WIDER batch leaves the arena entirely -- `moe_cache_update_host()` returns false above the band, so the routed experts are streamed from the pinned host master on every step. A speculative verify batch is `n_draft + 1` rows, and while the MTP/draft family is bounded to 15 (i.e. 16 rows) by `LLAMA_SPEC_DRAFT_N_MAX_CLAMP`, the DRAFTLESS n-gram speculators have no width bound at all: `ngram_mod.n_max` defaults to 64 (`n_min` 48), the ngram-map family's `size_m` to 48. On a host-expert config the README's own `draft-mtp-adaptive,ngram-mod` recommendation therefore put every ngram step outside the band, which is the "ngram-mod halves decode with host-resident experts" report (TODO ggml-org#59). Measured (2x R9700 gfx1201, Flash-Next IQ4_NL + MTP head, `-sm tensor -ncmoe 36`, `-c 16384`, `prompts/recall.txt`, `-n 2000 --temp 0 --seed 42`): sweeping `--spec-ngram-mod-n-max` gives 154.8 t/s at 12 (13-row verify), 159.2 t/s at 15 (16 rows, in band), **55.0 t/s at 16 (17 rows -- one extra row, -65 %)**, 58.1-58.3 at 20/24, 48.1 at 32; the arena hit rate drops 0.978 -> 0.898 and the cache's counted `reaches` collapse 1086540 -> 129660 (the wide steps never reach the arena). The README combo measures 73.9 t/s uncapped against **168.3 t/s** at a 15-token draft. Controls: fully device-resident (3 GPU, `-ncmoe 0`) 144.7 (MTP n3) -> 378.7 (ngram default) vs 244.6 capped -- there the wide draft is the win; the CPU expert path (`MOE_EXPERT_CACHE_MIB=0`) 38.5 -> 67.6 (wide) vs 52.2 capped -- upstream's shape and the reason the reporter's config in issue ggml-org#214 was fine there. The fix is a new `moe_cache_band` device query (appended to `ggml_backend_device_i`, zero-initialized for backends that do not set it; CUDA answers `moe_cache_enabled() ? moe_cache_max_tok_dev(device) : 0`), `llama_model_moe_cache_band()` (the smallest band across the model's host-expert devices), and `common_speculative_clamp_ngram_n_max()`: `common_init_from_params` clamps an ENABLED n-gram speculator's draft length to `band - 1` once the model (and therefore the preflight's decision) is known, lowering `n_min` with it (ngram-mod discards a draft shorter than `n_min`, so a naked `n_max` clamp would silently disable it), and refreshes `cparams.n_rs_batch` so the slab's narrow verify floor and the chunked-GDN rollback bound see the capped depth (the uncapped default also cost ~5 GiB of arena: 32938 -> 27814 MiB). `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` restores the uncapped draft. A model with no active arena reports band 0 and is never clamped; a `draft.n_max` above the band is warned about, not clamped (the 15 bound is a correctness one and matches RDNA4's band). Gates: prefilled-text purity with `ROCBLAS_USE_HIPBLASLT=0` (per the delivery's fresh-process rule) -- `plain` / `draft-mtp n3` / `ngram-mod` hand-capped / `ngram-mod` auto-clamped all hash identically (1793 chars, sha 9d914f47aebb) on the host-expert cell, the same hashes the pre-amend build produced; `LLAMA_SPEC_NGRAM_MOD_BAND_CLAMP=0` reproduces 74.1 t/s exactly; the device-resident and CPU-expert controls are unchanged (380.2 and 68.1 t/s, no clamp line). The clamp never touches a non-ngram config (the MTP-only arms log no clamp). Record: `wip/ngram-mod-host-experts/EVIDENCE-2026-10-24.md`. Windows HIP build (issue ggml-org#217): the three `std::min`/`std::max` calls on int64_t in moe-expert-cache.cu (the sizing slack guard, the auto-arena floor and the heat-coverage bound) are replaced with two local ternary helpers. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (int64_t is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged: greedy text is byte-identical on the dense 4B (`-sm tensor`, 3 GPU) and the host-expert arm (35B-A3B `-ncmoe 40`), prefill-logit mean KLD 0.000000 / same-top-p 100.000 %, MTP acceptance 0.73881. Amended r125 (issue ggml-org#216): the draft-reserve log line is now gated on the model actually having host-resident experts (llama_model_moe_host_expert_dev_count() > 0). llama_model_moe_cache_preflight() is a no-op without one, so a dense model used to print "keeping the flat 3584 MiB draft reserve" for a reserve nothing consumed -- and in issue ggml-org#216 that line was read as evidence that the slab or the expert-cache arena was involved in a dense model's VRAM failure. No behaviour change: the reserve, the derived value and the MOE_EXPERT_CACHE_AUX_RESERVE_MIB override are identical wherever they are actually used.