Warn user if a context size greater than 2048 tokens is specified by Ronsor · Pull Request #274 · ggml-org/llama.cpp

Ronsor · 2023-03-18T23:15:48Z

LLaMA doesn't support more than 2048 token context sizes, and going above that produces terrible results. Still allow larger sizes, but print a warning so users don't suspect a bug in the software instead.

LLaMA doesn't support more than 2048 token context sizes, and going above that produces terrible results.

j-f1 · 2023-03-19T00:12:11Z

Thank you!

low_level_api_chat_cpp.py: Fix missing antiprompt output in chat.

…ml-org#274) LLaMA doesn't support more than 2048 token context sizes, and going above that produces terrible results.

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…ml-org#274) LLaMA doesn't support more than 2048 token context sizes, and going above that produces terrible results.

This branch carries the GGML_API change that moves extern into the macro on every path. That makes the definition site expand to extern int turbo3_cpu_wht_group_size = 0; which gcc rejects under -Werror: ggml-turbo-quant.c:28: error: 'turbo3_cpu_wht_group_size' initialized and declared 'extern' [-Werror] cc1: all warnings being treated as errors Declares the symbol with GGML_API, so it still carries dllexport/visibility, then defines it plainly on the next line. Verified: builds clean, and the ODR property from ggml-org#274 is preserved -- the symbol is still exported from libggml-base and libggml-cpu still has zero definitions of it. Compiles warning-free under -Werror -Wall -Wextra for static, shared-ELF and MinGW expansions of GGML_API.

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

@slavap

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004. r207 (2026-11-03): the host-pool-speed campaign is folded in here (it was developed as wip/host-pool-speed/ and applies to this block's pool code). Everything below is `--host-experts pool` only; the default `pinned` mode is byte-for-byte unchanged. * A POOL-backed expert table is served by the device GATHER in the prefill band by default. Its whole-table staging is the slow path at every width (the source is the pageable model mapping plus the pool's host alias, and it degrades as coverage rises): 122B-A10B, -ncmoe 48, 5258-token prompt, -ub 2048, three R9700s -- pool FRAC 95 prefill 1507.7-1631.3 t/s with the gather against pinned's 634-638, and 16.9 with the gather switched off; decode at parity (67.2-67.7). Prefill logits are bit-identical to the staging path (KLD -0.000000 / same-top-p 100.000 %). Mechanism: a new optional iface query moe_cache_pool_backed(backend, weight) (CUDA answers "this table's master is the read-only mapping"; the Meta backend forwards it to its simple devices) plus a BAND gate in sched_input_gatherable (n_tokens > moe_cache_band) -- taking the decode band over with the gather costs 1.2-4.1 t/s, because that band is served from the arena by the slot-remap read. GGML_SCHED_DEVGATHER is three-valued: unset = pool-backed tables only (shipped), 0 = kill switch, 1/force = the historical A/B arm (not the default: it costs a pinned master ~47 % prefill at two devices). The gather resolves the arena first and the pool second, which subsumes MOE_HOST_POOL_COLD (COLD=0 19.57 vs 19.25-19.86 with COLD=1, 15.54 without the gather). * MOE_HOST_POOL_RAM_PCT now SCALES EVERY TABLE instead of shaving the registration-order tail to the one-slot floor (122B at RAM_PCT=70: 26 of 144 tables at one slot, 10.94 t/s -> 17.44 t/s at identical pinned bytes, +73 % with SIZING=1; RAM_PCT=85 19.52 -> 20.97), and MOE_HOST_POOL_SIZING=1 fits its concentration plan TO the ceiling. * MOE_HOST_POOL_SIZING / MOE_HOST_POOL_COV_PCT (opt-in): per-table pool slots from routing concentration; +7.5 % decode at FRAC 80 and +16.4 % at FRAC 60 at identical pinned bytes, for -13/-15 % prefill. * Diagnostics: moe_cache_report now reports the RUN (it used to print an emptied cache in every mode -- the artefact that once made "the pool never activated" look like a bug), a forced MOE_HOST_POOL_DIO logs its mode, bounce_acquire owns the pinned bounce ring (the ring total is finally true: 2209-3108 MiB on the 122B, uncapped on purpose -- a cap costs 8-25 % of prefill at FRAC 80), pool fills are timed by source, and the gather reports its calls/pool-backed/miss-fills. * MOE_HOST_POOL_PREFETCH is REMOVED (parsed, never read -- TODO ggml-org#268). Held back deliberately: MOE_HOST_POOL_FRAC=100 as the default. It is +16 % decode over 80 on one GPU, but the RAM ceiling does not yet reserve the model's own pageable footprint, and on a multi-device split 100 % measured 9 % SLOWER than 95 % (4 GiB more pinned, 42 GiB more disk per run), over-committed a 91 GiB box once, and lost to 80 % on the 1-GPU srv cell. Default stays 80. Gates (records: wip/host-pool-speed/README.md SS18, WORKLOG.md 261103.009-012): greedy text byte-identical across modes on 1/2/3 GPUs (5835 chars dc178c29fe89 / 6079 01607c16b483 / 5917 5421cf18bed5; 257 5531ae33f0f7 on the 64-token oracle); prefill-logit KLD -0.000000 / top-p 100.000 %; MTP four-axis acceptance identical to 5 decimals on all four axes (0.81370 / 0.75408 / 0.89175 / 0.98148) with `none` == `draft-mtp` text; llama-batched-bench -npl 4,8 pool 338.5/835.8 vs pinned 81.8/136.0 pp t/s; check-gpu-faults.sh clean. r219 amendment (issue ggml-org#274, Windows) -- the pool's fill must never require a tier the platform does not have, and a FAILED fill must not be resident: * the pool's DISK TIER (MOE_HOST_POOL_DIO=1) is best-effort. `pool_open_file` is a compile-time stub on Windows, so requesting O_DIRECT there failed EVERY pool fill, which failed every POOL (table_pool_locked() -> null, i.e. "the pool never appeared"), and the bounce fallback asked for the same missing tier and failed too ("no pinned host source ... (pool and bounce both unavailable)"). The mapping is always a valid source, so `pool_fill_slot_locked`, `expert_host_src` and `pool_mirror_bulk_fill_locked` fall back to it, and `pool_dio_effective()` refuses a disk tier the platform lacks with one WARN. The pool, its pin and the in-place cold read are unchanged. * `access_locked()` recorded the slot as RESIDENT before the copy, so a fill that could not be serviced served stale arena bytes (`async fill failed ...: no error` -- cudaGetLastError() is empty for a host-side failure): 20 t/s prefill and a ONE-token answer on the reporter's box. The admission now rolls back, the table is marked `fill_failed` and stands down wholesale (the same policy as `alloc_failed`), so the scheduler serves the op from the host master, and the census reports `FAILED fills: N`. * the tier census (`report_host_expert_placement`) is WARN, not INFO -- at the default verbosity INFO does not exist, which is why the reporter saw every failure and none of the decisions -- and it carries the pool-source count. * diagnostics: MOE_HOST_POOL_DIO_FAULT (a platform with no pool disk tier) and MOE_HOST_POOL_FILL_FAULT (no source at all), the two arms `scripts/gate-host-pool.sh` uses to reach the Windows shape and the rollback from Linux. Gates: gate-host-pool.sh 19/19 (four arms byte-identical; pre-fix 7281 `async fill failed` lines and garbled text); see WORKLOG.md 261105.005. * r220, cross-backend copy ordering (from @slavap, PR ggml-org#268): the general D2D split-input copy used the backend's SHARED per-device `copy_event` -- issue on the source stream, re-record that event, wait the destination stream on it. A queued HIP stream wait resolves against the LATEST record at execution, so a copy still in flight re-records the event and re-binds the consumers already queued on it; independent copies then serialize against each other. `cpy_tensor_async`'s body is now `cpy_tensor_async_impl(..., ev_nowait)` and the new optional iface member `cpy_tensor_async_nowait` takes a CALLER-owned event, recorded on the source stream right behind the copy and waited on by the consumer, so no wait is ever re-bound. The scheduler keeps a ring of those events per source backend (a slot is re-recorded only after its previous record completed) and restarts the rings at every full synchronize. `GGML_SCHED_COPY_EVENT=shared` restores the old path (kill switch). Bit-exact and throughput-neutral on this box (same-seed greedy text identical on `-sm tensor` / `-sm layer` / `-sm layer -ncmoe`; pp8192 3-GPU `-sm layer` 6311 vs 6317 t/s); the win is on a box where two cross-backend copies overlap.

Warn user if a context size greater than 2048 is specified

052027d

LLaMA doesn't support more than 2048 token context sizes, and going above that produces terrible results.

j-f1 approved these changes Mar 19, 2023

View reviewed changes

j-f1 merged commit d7def1a into ggml-org:master Mar 19, 2023

niansa mentioned this pull request Mar 19, 2023

Prevent user from setting a context size that is too big #266

Closed

Piezoid mentioned this pull request Mar 19, 2023

Longer and infinite output #71

Closed

Deadsg pushed a commit to Deadsg/llama.cpp that referenced this pull request Dec 19, 2023

Merge pull request ggml-org#274 from dmahurin/fix-missing-antiprompt

34ad71f

low_level_api_chat_cpp.py: Fix missing antiprompt output in chat.

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

Specify tensor name regex for tensors to be repacked (ggml-org#274)

4158743

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

Navigation Menu

Repository navigation

Warn user if a context size greater than 2048 tokens is specified - #274

Warn user if a context size greater than 2048 tokens is specified#274
j-f1 merged 1 commit into
ggml-org:masterfrom
Ronsor:patch-2

Ronsor commented Mar 18, 2023

Uh oh!

j-f1 commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Ronsor commented Mar 18, 2023

Uh oh!

j-f1 commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants