Docker - Fix publish docker image in GitHub Registry by bernatvadell · Pull Request #235 · ggml-org/llama.cpp

bernatvadell · 2023-03-17T09:57:36Z

In the PR that was resolved (#132), the action defined to publish the packages used the user and token of the author of the commit in master.

In this case, I have been yours, but I do not have permissions for such an action.

That is why I have modified the file, so that the owner's token is included as a secret with the name GHCR_PAT.

It is important to select the read:packages & write:packages permissions.

NOTE: Checks will fail until the secret is included.

j-f1 · 2023-03-17T12:23:56Z

Would github.repository_owner / secrets.GITHUB_TOKEN work? (with the appropriate permissions, namely packages: write)

ggerganov · 2023-03-19T18:46:50Z

Let me know if I need to do something here - not familiar with Docker

…repository_owner

bernatvadell · 2023-03-20T15:38:14Z

Would github.repository_owner / secrets.GITHUB_TOKEN work? (with the appropriate permissions, namely packages: write)

I think this should work, I changed it and the login at least went well. It would be necessary to test if we have the appropriate permissions during the push of the image.

@ggerganov can you authorize this PR? Ty!

j-f1

Seems reasonable to me but we can’t fully test until merging.

bernatvadell · 2023-03-20T17:45:51Z

Seems reasonable to me but we can’t fully test until merging.

it's working! ty

Decrement CUDA version and bump Ubuntu

* fix publish permission * try to fix docker pipeline using as password github_token & username repository_owner

The `-mla` command line option turns into an int from a bool. mla = 0: use standard attention mla = 1: use MLA with transposed cache mla > 1: use MLA without transposed cache Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

* fix publish permission * try to fix docker pipeline using as password github_token & username repository_owner

…(_abi3) - Pass num_q_heads from runtime through compile.py -> op.py -> relay NUM_HEADS - Cache key bumped to _abi3 to force fresh 3B xclbin - f3best_pack.h: extended F3BestHostAbi with num_q parameter - 3B relay now built with NUM_HEADS=24 (was hardcoded 32) - 52/52 contract tests pass, build clean Note: 3B probe still shows rel_l2 ~0.9 on all boundaries; Num_Q hypothesis disproven.

The uncommitted NH=num_q=24 change was a regression: the f3best emitter hardcodes NH=8 in ALL output-layout paths (8 partial FFN streams + attention residual at offset 8*PH_STRIDE). Reading 24 partial heads made the host read attention at 24*3072 (=73728, zeros) while the kernel wrote at 8*3072. Reverted to NH=8. f3best-sblk diagnostic confirmed: with NH=24, s@24=max0.0 vs s@8=max1.71; with NH=8 both coherent. After revert: npu_post material, final rel_l2 ~0.84->0.2-0.5, no crash across all 28 layers. This isolates the TRUE 3B symptom: post_attention rel_l2~0.34-0.85 with huge max_rel (~1e3) despite near-matching L1 (coordinate/permutation pattern H2). 1B (same NH=8) stays perfect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…_BYTES/SCALE_OFF) The f3best host-side differential tests (NIB0/ZERONIB/ZEROSC/ZEROGU/ONESC/ IMPULSE/DOWNSEL) hardcoded 4096 as the per-tile nibble-region size and scale offset. 4096 = bcast_kc*(bcast_n/2) is the 1B nibble region (M=4,E=2048). For 3B (M=4,E=3072) the nibble region is 6144 bytes and scales begin at offset 6144, so 4096 zeroed only the first 4096 of 6144 nibble bytes and zeroed scales starting at 4096 (mid-nibble-region) instead of 6144. Introduced NIBBLE_BYTES=M*E/2, SCALE_OFF=NIBBLE_BYTES, SCALE_BYTES=PACKED- NIBBLE_BYTES (geometry-derived) and replaced every 4096 in the diff-test block. NIB0 now logs its applied geometry for verification. Note: NIB0 with the corrected full zeroing (NIBBLE_BYTES=6144, all 1440 tiles) STILL yields post_attention rel_l2=0.808 != 0, which disproves the 'NIB0=resid' expectation: Wo zeroing alone does not make O-proj output zero for 3B — the attention/O-proj residual path is being investigated separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ck_bcast writes KC*(N/2)) pack_bcast (f3best_pack.h:124) writes scales at byte offset KC*(N/2)=256*16=4096 in EACH PACKED-byte tile, geometry-independent — 4096 nibble bytes + 512 scale bytes per tile, then pad to PACKED (1B:4608, 3B:6912). The previous commit wrongly set NIBBLE_BYTES=M*E/2 (=6144 for 3B), which is the total nibble bytes across all M*E, NOT the per-tile value. That broke ZEROSC/ONESC (zeroed wrong region, FFN stayed non-zero). Reverted NIBBLE_BYTES=4096, SCALE_OFF=4096, SCALE_BYTES=PACKED- 4096 (scales+pad). ZEROSC now zeroes scales correctly (npu_ffn=0). Lesson: verify the packing layout in pack_bcast source before deriving constants. NIB0 APPLIED log retained for verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…any geometry The [qdump-cpuq] block (ggml-xdna.cpp ~6300) hardcoded the 1B geometry: guard E==2048, ck=256, row stride 64*18, b<64, hh2<4 (ag=4), i<32 (hd=64), prescale 0.125 (1/sqrt(64)). For 3B (E=3072, ag=3, hd=128) the block never ran, so the decisive Q-path-vs-attention bisection (suggested by external review CONSULT_3B_F3BEST_SYSTEMATIC_REPLY §6) was unavailable for 3B. Generalized: guard on head_dim>0; ck=attn_group*head_dim; row stride nblk*18 (nblk=E/group_size); b<nblk; hh2<attn_group; i<head_dim/2; prescale 1/sqrt(head_dim). Now prints ag/hd/rows. Verified on 3B: qdump-cpuq rel_err=10.3 (=sqrt(128) prescale-mismatch, NPU Q = raw CPU Q x sqrt(128)), consistent with Q-GEMV/RoPE being correct (Q-path not the bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…de Code - Guide: same-card, same-model table against PrismML's prism branch (basic SYCL ggml-org#235): prompt 207 vs 895 t/s, plain 20.6 vs 42.3, 32K depth 8.1 vs 35.8, chat with MTP 37.6 vs 88.4. - New "As a coding agent (Claude Code)" section: env setup, a CLAUDE.md rules file, what to expect. - bonsai-chat-template.jinja: the model's template, but a mid-conversation system message becomes a user-turn note instead of an error (Claude Code sends those). Used by the guide's run command and run-bonsai.bat, and shipped in the Windows zip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@dwymark

…org#235) * sycl: add PTQ1_0 and PQ2_0 support with MMVQ vector dot kernel - Implement base-3 trit unpacking and dequantization (dequantize_ptq1_0, dequantize_pq2_0) in dequantize.hpp and convert.cpp - Implement vec_dot_ptq1_0_q8_1 and mul_mat_vec_ptq1_0_q8_1_sycl in vecdotq.hpp and mmvq.cpp - Add get_rows and cpy dispatch for PTQ1_0 and PQ2_0 - Enable PTQ1_0 in can_use_mul_mat_vec_q() for fast in-register Level-Zero vector execution - Verified on Intel Arc Pro B50 with oneAPI 2026.1 / Level-Zero (2.20 tok/s on 27B) * sycl: vectorize PTQ1_0 dot product with SIMD trit unpacking and DP4A * sycl: add native PQ2_0 MMVQ vector dot kernel with DP4A * sycl: spread PQ2_0 MMVQ across 16 lanes per block (~2x tg) The MMVQ kernel derives its lane split from qi/vdr. PQ2_0 was passing QI_PQ2_0 (= QK_PQ2_0/32 = 4), which put only 4 lanes on each 128-element block and left every lane an 8-deep serial dp4a chain over 8 bytes of qs. Pass a dedicated MMVQ_PQ2_0_QI of 16 instead, so a block is covered by 16 lanes reading 2 bytes / 8 elements each with 2 dp4a per call, and a sub-group keeps more blocks in flight. Measured on an Arc iGPU with test-backend-ops perf, MUL_MAT m=4096 n=1 k=14336, alternating the two builds to cancel out background load: 827/846/837 us before, 397/395/384 us after (~2.1x). On an idle machine the same case goes 247.6 -> 123.0 us. 8 and 32 lanes per block were both tried and were slower; at 32 the per-call scale setup dominates the single remaining dp4a. Also fold the 2-bit-to-byte-lane unpack shared with the q2_0 path into unpack_2bit_to_byte_lanes(), and switch the epilogue to the q4_0-style sumi*ds.x - zp*ds.y form. The previous PQ2_0 epilogue applied the zero point with byte_sub_4 and then used only ds[0], dropping the ds.y term. test-backend-ops test -o MUL_MAT -p type_a=pq2_0: 38/38 on both SYCL devices, 3/3 backends passed. The unrelated q1_0 failures on SYCL1 are present with and without this change. * sycl: gate mmvq on implemented types instead of aborting can_use_mul_mat_vec_q() claimed every quantized type via ggml_is_quantized(), so any type without a case in mul_mat_vec_q_switch_type() reached the GGML_ABORT in its default arm and killed the process. test-backend-ops -o MUL_MAT could not complete on SYCL for this reason: it died at mmvq.cpp:2710 on tq2_0, which also meant every case generated after that point never ran. Gate on an explicit list of the types the switch actually implements. The list matches the switch exactly, and is also identical to the set ggml_get_to_fp16_sycl() handles, so a type this declines cannot be served by the dequantize fallback either - hence the matching check in supports_op(), which lets the scheduler place those nodes on CPU rather than abort mid-graph. Arc Pro B50, oneAPI 2026.1, Level Zero: before: test -o MUL_MAT aborts, exit 134 after: 1098/1098 tests passed, exit 0 tq2_0 goes from "fatal error: unsupport data type" to "not supported". * test-backend-ops: add Bonsai-shape MUL_MAT cases for PTQ1_0/PQ2_0 The stock generator only emits 16 x 256 cases for these types, all of which take the single-column mmvq path, so the multi-column dispatchers (ncols_dst 2..8) had no coverage at all. Add the real Bonsai row lengths (k = 1024/5120/6144/17408) at odd row counts (m = 67/70, landing a partial row group) with n sweeping 1..8. Arc Pro B50, oneAPI 2026.1, Level Zero, per type: MUL_MAT 9 -> 102/102 passed MUL_MAT_ID 75/75 passed * sycl: decline float -> PTQ1_0/PQ2_0 copies instead of asserting Adding the two types to ggml_sycl_is_quantized_type() made supports_op accept f32/f16/bf16 -> PTQ1_0/PQ2_0 copies, but there is no quantize-rows kernel for either, so ggml_sycl_cpy() took the float -> quantized branch and died on GGML_ASSERT(ggml_sycl_can_quantize_rows_sycl(...)) at cpy.cpp:1298/1302/1308. Neither format can be produced by a row-at-a-time GPU kernel: the packing assumes the Hadamard rotation the offline converter applies, so the right answer is to decline the pair and let the scheduler place it elsewhere, not to add a kernel. Quantized -> same-quantized copies are unaffected and still take the memcpy path. Reported by @dwymark on an Arc B390 (Panther Lake Xe3); reproduced on an Arc Pro B50. test -b SYCL0 -o CPY -p "ptq1_0|pq2_0" before: abort at cpy.cpp:1298 after: 18/18 tests passed, float -> ternary reports "not supported" No regression: MUL_MAT 204/204, MUL_MAT_ID 150/150, GET_ROWS 8/8 for the two types. The unrelated tq2_0 -> tq2_0 abort at cpy.cpp:1455 and the SET_ROWS abort at set_rows.cpp:549 are both present on prism and untouched here.

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record.

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms.

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com>

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004.

qsa : keep the incremental QSA prefix across mrope images (rebase of ggml-org#163)

…lls, DFlash, FA build grouping, server prompt cache) The delivery content not owned by blocks 00-11: the CPU top-k block prefilter (common/sampling.cpp), the x86 quants / repack / arch-fallback campaign wins, the ggml-alloc compute-buffer drop/realloc plumbing, the per-backend stage_gather / moe_cache_* / slab_* NULL iface fills, the DFlash F1 model (src/models/dflash.cpp + common/speculative.cpp), and the FA per-(ncols,head) template-instance build grouping. With this block the Phase-1 tree equals the reference tip af0595360 (release.json.tree) exactly. Also fill the newly appended slab_compute_plain device-iface entry (NULL) in the CPU and RPC device interfaces, matching the other optional slab_* fills: the field was appended by block 09 after these fills were written, which left a -Wmissing-field-initializers warning. Fold (r86): beta 12, slab-transient-absorption (draft ownership). The speculative init accepts the pre-loaded draft model from common_init_result, so the slab has already measured the draft's per-device weights when it sizes itself; the fallback load keeps other callers unchanged. Fold (r96, PR ggml-org#189, briansp2020): `llama_io_write_host` queues its deferred state-save reads as `ggml_backend_tensor_get_async` into one pinned staging buffer kept by the context, synchronizes each backend once, then copies to the caller's buffer. A checkpoint of a recurrent model is ~75 small tensor ranges and under `-sm tensor` each was one synchronous read per device, so a checkpoint paid ~150 device syncs before every prompt chunk of a follow-up turn (Flash-Next + MTP: -55 ms / -59 ms TTFT on the short follow-up turns, same-binary A/B against `LLAMA_STATE_ASYNC=0`). States above `LLAMA_STATE_ASYNC_MAX_MIB` (default 256 MiB) and `LLAMA_STATE_ASYNC=0` keep the synchronous path; so does any state with a tensor no context backend owns. The restore direction is folded in r119 (PR ggml-org#209), below. The staging buffer is retained by the context (16 MiB granular, bounded by the cap) and is one buffer per context, so state saves on one context must not run concurrently. `src/llama-context.cpp` also gains `#include <algorithm>` for `std::find` (the patch as submitted does not compile without it). Fold (r106): DFlash2 under `-sm tensor` (TODO ggml-org#78). A new ggml `GGML_OP_ALL_GATHER` collective lets the Meta backend materialise a whole vocab-split tensor on every device, so the DFlash2 selector's `ggml_top_k` / `ggml_get_rows` over the draft logits -- which come from the target's vocab-split `output.weight` projection, and are therefore split on exactly the axis the selector reduces -- run locally instead of aborting; the r54 clean rejection is removed. The same fold gives a tensor-split drafter its compute region out of the slab's arena when the unmapped VA spare is too small (narrow-2): a 1280 MiB draft compute buffer against the estimator's 512 MiB tier is what exposed it, and any DFlash drafter under `-sm tensor` now also takes the host path for the F1 device layer-input copy (a pre-existing crash on a Meta buffer). Fold (r119, PR ggml-org#209, briansp2020): batch the device writes of a context state restore. The restore side of the r96 fold wrote the ~75 pieces with one synchronous `ggml_backend_tensor_set` per tensor, and under `-sm tensor` each of those bounced through a pinned chunk and synchronized every device. `llama_io_read_host` now takes the context, packs the deferred pieces into the same pinned `state_stage` that the save side keeps, queues each with `ggml_backend_tensor_set_async` on the backend that owns it, and synchronizes each backend once. It falls back to the per-tensor path when there is no context, a tensor's backend cannot be resolved, the state exceeds `LLAMA_STATE_ASYNC_MAX_MIB`, or with `LLAMA_STATE_ASYNC_READ=0` (or `LLAMA_STATE_ASYNC=0`). Measured on r118 (2x R9700 gfx1201, `-sm tensor`, q8_0 KV, 1571-token checkpoint): 35B-A3B restore 49 -> 40 ms and Flash-Next GSQ 89 -> 75 ms, with the DFlash2 draft 68 -> 57 ms; one GPU is ~2-6 ms slower because the extra host copy has no multi-device sync to save. The restored 48 greedy tokens equal a fresh run (35B-A3B plain and DFlash2, Flash-Next), the four-axis MTP text/acceptance and the prefill-logit KLD are unchanged, and `test-save-load-state` / `test-state-restore-fragmented` pass. The stage is one buffer per context, so saves and restores on one context must not run concurrently. Per-block build (blocks own their code): this block also switches the r117 identity-preference layer-drop ranking in `moe_cache_apply_identity_pref_locked` over to `heat_rank_vec()`. Block 09 ranks it by `t.heat` there so that block 09 builds on its own (`heat_rank_vec` is introduced here with the band-split prior); the two rankings are byte-identical in block 09's tree, which has no decode counter. The merged tree is unchanged. Fold (r130, PR ggml-org#210, overdoingism): server prompt-cache residency. With a unified KV cache and `--cache-idle-slots` (default on), every new task saved each idle slot to the host prompt cache and cleared it, so two conversations taking turns on two slots copied their whole state out and back on every switch (Qwen3.8-27B: 330 MiB of recurrent state plus 3.8 GB of KV at 110K tokens, 1.6-2.7 s per switch). An idle slot now keeps its context when a new task starts; the KV-full path (`try_clear_idle_slots`) saves the least recently used idle slot to the prompt cache before clearing it (it used to purge it); a restore that fails for want of cells held by another idle slot moves idle slots out of the way one at a time, with the entry being restored held out of that eviction, and retries; and the slot-selection save holds the entry the task is about to restore out of its own eviction. `server_prompt_cache::find()` is split out of `load()` for that. The buffers of restored/evicted states are kept for the next save (smallest sufficient spare, counted against `--cache-ram` together with the cached states). `LLAMA_SERVER_LAZY_IDLE_SLOTS=0` and `LLAMA_SERVER_CACHE_REUSE=0` restore the eager behaviour. There are no arithmetic changes; a different cache history places a sequence's KV in other cells, which is the already-accepted `--kv-unified` non-determinism (`GREEDY-PURITY.md` section 38), and the patch's teacher-forced KLD is at the long-context noise floor. `find()` also skips an entry whose main state buffer was already consumed: a restore that failed on the draft buffer leaves such an entry behind, and re-matching it compared against zero bytes and reported success. The upstream server test `unit/test_kv_keep_only_active.py` is updated for the new default and covers the kill switch. Fold (r130, PR ggml-org#212, overdoingism): DFlash prompt-ingest tail. Every layer of the Qwen3.8-27B DFlash2 draft attends over a 2048-position sliding window and its KV cache is sized for the window, but `process()` fed it every prompt row: a 110K prompt encoded 110K rows, all but the last ~2048 dropped without ever being read, and each prompt batch waited on the target before the draft could start, which also broke the queue of the target's ubatches. When every draft layer has a window, the DFlash impl now takes the prompt end from `set_prompt_end()` (the block-02 hook) and skips the prompt rows before the last `n_swa` positions; a batch left with no rows returns before the target wait, and the first generation row resets the threshold as in the MTP impl. Exact because DFlash sets `LLAMA_SWA_TYPE_STANDARD` unconditionally (`src/models/dflash.cpp`), so the mask is `p1 - p0 >= n_swa`. New staging query `llama_model_n_layer_swa()` in `src/llama-ext.h` (the per-layer pattern is not reachable through `llama_model_meta_val_str()`, which skips array values). `LLAMA_DFLASH_INGEST_TAIL=0` feeds every row. Measured (3x gfx1201, `-sm tensor -mg 0`, 27B UD-Q4_K_XL plus DFlash2-Q4_K_M, `-fa on -ctk/ctv q8_0`): prefill +8-12 % at 6K/26K/88K tokens (88.6K: 1271 -> 1395 t/s, no-draft 1415 t/s, so the draft's share of the prefill goes 10.2 % -> 1.4 %), and 35B-A3B Q4_K_M plus its DFlash2 draft +14.6 % at 6.1K; greedy text and draft counts byte-identical in every arm (one and two slots, `-ub 512` and `-ub 2048`). Fold (r136, PR ggml-org#225, overdoingism): idle-slot parking. ggml-org#210's lazy idle slots (ggml-org#210 is the r130 fold above) keep an idle slot's KV resident in a unified cache, so a request that runs alone no longer starts in an empty cache: its cells land after the resident ones and its attention range (n_kv) covers them. Flash attention's tiling/split depends on those indices, so near-ties can flip: the reporter's 16-request repeat test agreed on 4/16, and our same-slot schedule on 13/32. A staging API (llama_memory_seq_park / llama_memory_can_park, src/llama-ext.h) now moves a sequence's cells to the top of a unified cache below the previously parked sequences, copying the K/V rows on the device and moving the cell state (llama_kv_cells::mv, including seq_pos and the extension); seq_unpark moves one back to the lowest free run and park_compact keeps the stack together. New cells (find_slot) and get_n_kv() stop below the lowest parked cell, so a running sequence attends over the same range at the same indices as with the idle slots saved and cleared. Parking requires one stream, unshared cells, V not transposed and no sliding window; anything else keeps ggml-org#210 and logs once. The server parks idle slots at task start and unparks the slots that run (LLAMA_SERVER_PARK_IDLE_SLOTS, on unless 0, with the lazy idle slots and --kv-unified); a slot whose cells are not one run of its own (shared through seq_cp, e.g. a cached prefix restore) is saved and cleared, which is the lazy-off output. Delivery fix on top of the PR: with --no-cache-idle-slots such a slot is left resident instead (there is no saved copy to restore, and the lazy slots are meant to stay resident). LLAMA_KV_PARK_DEBUG=1 logs every move and refusal. Measured (gfx1201, 27B UD-Q4_K_XL + DFlash2, 4 slots, 131072, q8_0 KV, 16 prompts two rounds, cache_prompt false): 16/16 rounds and 32/32 equal to LLAMA_SERVER_LAZY_IDLE_SLOTS=0, against 13/16 and 13/32 with parking off; --fa off, a sliding-window model and an unparkable cache are text-identical on/off, hybrid attention/recurrent is 16/16 equal to lazy-off, and 2 GPU -sm tensor works. GREEDY-PURITY.md section 57. Amended (r143): the MoE expert cache is process-global but its tables point at host expert masters the MODEL owns, and nothing told the cache when those went away. `llama_model_free` frees the masters; the deferred arena sizing -- which fills EVERY registered table from its master -- then read a freed pointer: a host memcpy SIGSEGV in `fill_slot_from_host`, from a table belonging to a model that no longer existed. Reproduced on Flash-Next GSQ-IQ3_XXS, 2 GPU `-sm layer -ts 59,41 -ncmoe 24` (llama-bench runs one test per `-ts` value and frees the model between them; the second test's first band pass died filling the first model's tables), and in the same shape with `--host-experts pool`, whose master is the read-only model mapping, so a host-buffer-free hook could not have covered it. `llama_model_free` now calls the new `llama_model_moe_cache_forget()` (device iface entry, appended at the very end of the struct for the positional-initializer reason above), which drops the device's tables -- freeing each one's arena slot range, remap staging, heat mirror, bounce ring and host pool -- re-points every id map so a reused `ggml_tensor*` registers a fresh table instead of inheriting a dead model's geometry, forgets the per-model preflight accounting, and un-sizes the cache so the next model sizes on its own first band pass instead of silently running uncached (a second model previously inherited `g_sized` and never sized). Single-model runs are unchanged: the teardown only adds the drop log lines. Parking and the meta backend's view lifetime (issue ggml-org#232, with block 09): `park_move` creates 2 views per (layer, K/V) per chunk of a moved run, and a nearly-full unified cache parks in dozens of tiny chunks (`chunk == the gap`). Those views are created between graph evals, so they now go to the meta backend's scratch container (`ggml_backend_meta_set_scratch_views`), reset once each chunk's synchronous copies are done, instead of accumulating in the per-generation compute container. Reproduced at the default budget (2 slots, `-c 32768`, two ~16.3k-token prompts): the r149 build aborts in `seq_park`/`park_move` with "not enough space in the context's memory pool (needed 2261360, available 2260992)"; with this, three park/unpark cycles complete with no growth of that container. `gate-park-idle-slots.sh` PASS (park == lazyoff byte for byte on all four requests). Fold (r151, PR ggml-org#233, briansp2020): pipeline the batched state copy. The r96/r119 batch above queues every piece and then moves the host side and the device side one after the other, so a 112.6 MiB recurrent checkpoint pays its memcpy and its device copies serially. The pieces are now cut into up to LLAMA_STATE_PIPE_GROUPS (default 8; 1 = the old unpipelined copy) size-balanced groups: a restore packs a group into the stage and queues its async sets before packing the next (the groups own disjoint stage ranges, so a queued copy is never overwritten), and a save records one event per (backend, group) after that group's copies are queued and copies the group out as soon as its event completes. A backend without events (the CPU backend) falls back to the single llama_state_sync_all path, i.e. the code above, unchanged. Measured (gfx1201, same build dir, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, medians of 5 cycles): Flash-Next GSQ + MTP head 2 GPU `-sm tensor -ncmoe 40` 134.5 -> 129.7 ms (-3.6 %), 35B-A3B Q4_K_M 1 GPU 69.4 -> 67.4 ms (-2.9 %, and below the pre-r119 synchronous path's 71.3 ms); 35B-A3B 2 GPU tensor within noise. The server's own save+load trace moves 81.7 -> 78.4 ms (Flash-Next) and 44.9 -> 39.9 ms (1 GPU). The restored greedy text is byte-identical to the SYNCHRONOUS arm in every cycle on every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), the in-tree `test-llama-context` state cases keep an identical PASS/FAIL set, the coherence text and the prefill-logit KLD are unchanged, and no GPU faults. Fold (r152, PR ggml-org#235, overdoingism): stream the batched state copy on two threads. The r151 pipeline moves a checkpoint's host side one piece (tensor range) at a time with a plain memcpy, and that copy is bound by host memory bandwidth. It is now a non-temporal (SSE2) store over cache-line-aligned runs, split over LLAMA_STATE_COPY_THREADS (default 2; 1 = one thread but still streamed; 0 = the pre-r151 plain memcpy, the A/B arm) threads per group. A streaming store drops the read-for-ownership a plain store pays for every destination line: that is the whole win where the checkpoint does not fit in L3 (the submitter's 3900X + DDR4: 19.4 -> 11.8 ms per 150 MiB) and ~0 where it does (a 9950X3D has 192 MiB of L3, so a 164 MiB checkpoint's destination lines are mostly resident). The threads are where this box's win is: 164 MiB in a 35B-A3B checkpoint's real piece sizes (30 x 80 KiB, 30 x 2.6 MiB, 20 x 4.2 MiB) copies in 6.4 ms plain / 6.4 ms streamed on one thread / 3.0 ms on two, and 3.3-3.7 ms on three or four, so two is the knee on a dual-channel DDR5 controller. The workers stay inside ONE group: r151's copy-out may only read a group's stage range after that group's event, so a worker that crossed into the next group would read a range whose device copy is still in flight (a torn state, not a crash) - the same reason the per-group events exist. States below 16 MiB keep the plain copy (a thread hand-off then costs more than the copy), the streaming store is x86-only (`__SSE2__`/`_M_X64`, MSVC x64 included) so other targets gain only the threads, and every streamed run ends with `_mm_sfence()` before its bytes are handed to the caller (save) or to the device (restore). The restore's pack also resolves each piece's stage offset once instead of re-deriving it in the queue loop (the same addresses; the `off = off_g` reset it replaces is gone). `LLAMA_STATE_TIMING=1` logs the enqueue and copy time of each batched read/write, the diagnostic this was measured with. Measured (gfx1201, ROCm 10.1.0, `scripts/gate-state-pipe.sh`, prompt-cache restore of a 5246-token hybrid state, same build dir): the copy itself is -12 / -21 / -29 % for a 62.8 / 73.2 / 165.3 MiB save on 35B-A3B Q4_K_M 2 GPU `-sm tensor -ncmoe 10` and -21 / -22 / -29 % for a 112.6 / 126.6 / 250.9 MiB save on Flash-Next GSQ-RCO IQ3_XXS + the shared Q8_0 MTP head 2 GPU `-sm tensor -ncmoe 40`, with the restore's pack -26 / -30 %; the server's own save+load trace reads 41.2 ms (35B-A3B 2 GPU) and 72.1 ms (Flash-Next + MTP) against r151's 44.9 (1 GPU) / 78.4 ms. The restored greedy text is byte-identical to LLAMA_STATE_ASYNC=0 in every cycle of every arm (save side isolated by LLAMA_STATE_ASYNC_READ=0), `test-llama-context` keeps an identical PASS/FAIL set (`rf` + `rollback`, a slab/VRAM property of this box, on 1 GPU and 2 GPU tensor, as r151), coherence 4B Q8_0 `1c5d32ac537d` / 35B-A3B 2 GPU `627b2ebd7792` equals the r151 tree, and the prefill-logit KLD is 0.000000 / 100.000 %. Fold (r155, PR ggml-org#236, overdoingism): keep the context checkpoint a task just restored, and reuse the buffers of dropped checkpoints. A prompt that resumes from a context checkpoint (a hybrid/recurrent model reusing a cached prefix) restored the checkpoint and then saved the identical state again at the same position before its first batch ("superseding"), and saved one more near the end of the prompt; each save allocates, zero-fills and faults in a fresh host buffer of the recurrent state's size (150 MiB on Qwen3.8-27B, 174 MiB with the DFlash2 draft). A slot now remembers the n_tokens of the checkpoint its task restored; when a checkpoint lands at that same position, nothing has been decoded since the restore, so the checkpoint already holds the current state: it is moved to the back of the list (the ordering a re-save produces) and kept instead of being written again, and its spec state (data_spec) is the one the restore re-applied. The buffers of erased, superseded and invalidated checkpoints are kept (4 in total, the largest kept) and an exact-size one is taken by the next checkpoint, so a save is only the copy. LLAMA_SERVER_CKPT_REUSE=0 restores the eager behaviour. There are no arithmetic changes and no KV cell moves, so output is unchanged. Delivery hardening on top of the PR: n_tokens_ckpt_restored is also cleared in server_slot::reset(), so the invariant does not rest on every task entering the SLOT_STATE_STARTED branch, and the keep path refuses a checkpoint whose data_tgt is empty (load_tgt is a no-op for such a checkpoint, so it can never stand in for the live state). Measured (gfx1201, ROCm 10.1.0; every arm's greedy text compared byte for byte against LLAMA_SERVER_CKPT_REUSE=0 on the same build). Prompt time of a request that resumes from a checkpoint, -ms off -> on: Qwen3.5-4B (hybrid, 50.3 MiB state) 1 GPU 45.1 -> 31.4, 40.0 -> 28.7, 46.4 -> 32.4; the same at 2 slots --kv-unified with the idle-slot parking 43.4/42.2/44.6/42.2/43.9 -> 29.3/28.7/28.3/27.3/27.7; Qwen3.8-27B UD-Q4_K_XL + DFlash2 n-max 5 (checkpoint 149.6 + 21.3 MiB) 2 GPU -sm layer 197.1 -> 163.2, 186.9 -> 161.9, 144.1 -> 112.0, 187.6 -> 126.7, and with a 128-token decode 198.7 -> 162.3, 193.1 -> 147.2, 142.2 -> 110.7, 189.4 -> 128.3; 3 GPU -sm tensor 407.9 -> 373.6, 291.2 -> 236.2, 182.4 -> 142.0, 251.7 -> 150.6. The same log reads "superseding" 3 times off against "keeping" 3 times on and creates 9 checkpoints off against 6 on. Text is byte-identical in every one of the 36 requests across those 6 pairs plus a slot save/restore endpoint round trip (which the PR lists as uncovered and which is unchanged in both saved file size and text). Resident memory +0.15 GiB (4B) and +0.45 GiB (27B), i.e. the 4 spares. The saves are cheaper on this box than on the submitter's (DDR4, 4 KiB pages), which is why the win is 25-100 ms here against their 195-223 ms TTFT; the mechanism and the direction are the same. Gates: see the r155 record. Fold (r163, issue ggml-org#240, overdoingism): write the DFlash layer inputs into F1's device store from inside the target graph. F1 kept the extracted layer inputs on the device but copied them after the graph ran, so set_outputs() had to mark them graph outputs - and an output cannot be reused by the allocator, so every extracted input stayed live to the end of the graph (5 x n_embd x n_ubatch f32 = 188.8 MiB of compute buffer at -ub 2048 on a 27B; the same config reserves 568.42 MiB without the extraction). set_outputs() now emits a GGML_OP_SET_ROWS per extracted layer into the persistent store, and llm_graph_result::set_layer_inp() (which every arch graph now calls instead of assigning t_layer_inp[il]) closes each layer's window: at the next layer's recording it expands that layer's nodes and then emits the pending copy, so the copy lands right after the input's last consumer and the allocator frees it. The store's row offsets are an I32 [n_tokens] graph input filled per ubatch (a baked view offset would be captured into a HIP graph and replay for the wrong ubatch), the store is created in lf_layer_inp_dev_arm() before the first graph that writes it (one drop+re-reserve, so the run settles on the smaller layout), and the wait on the draft's consumed event moves before that graph. The host layer-input rows are reserved lazily by layer_inp_host_reserve(): F1 never read them and they were 200 MiB of pinned host memory per context. GGML_LF_DFLASH_DEV_GRAPH=0 restores the pre-r163 shape. Measured (27B UD-Q4_K_XL + DFlash2-Q4_K_M): target compute buffer 757.26 -> 625.27 MiB, output buffer 200.95 -> 0.95 MiB, greedy text and draft acceptance identical in every arm (1 GPU, 2 GPU -sm layer, -sm tensor host fallback, 35B-A3B -ncmoe 20, 2-slot server), decode flat, prefill-logit KLD 0.000000 / 100.000 %, MTP Protocol A byte-identical to r162 in 8/8 arms. Backend sampling: a compute-buffer reserve is what recomputes graph_max_nodes(), so the buffers are reserved once per sequence and only a WIDER sampler chain asks for another one -- a chain set again, a narrower one, or a removal cannot grow the graph's node capacity, the scheduler's arrays or the compute-metadata arena. LLAMA_BACKEND_SAMPLER_RESERVE_ONCE=0 restores the per-set reserve. Contributor: overdoingism (PR ggml-org#245). Co-Authored-By: overdoingism <overdoingism@users.noreply.github.com> Fold (r173, PR ggml-org#248, overdoingism): a new prompt-cache state buffer is faulted in with huge pages. `server_prompt_cache::take()` reserves a buffer of 64 MiB or more, advises its 2 MiB-aligned part `MADV_HUGEPAGE` and only then zero-fills it, so a multi-GiB slot state is faulted in 2 MiB pages instead of 4 KiB ones (Linux, THP in `madvise` mode; `LLAMA_SERVER_CACHE_HUGEPAGE=0` restores 4 KiB pages), and a recycled spare buffer keeps the pages it already has. Measured on this box (9950X3D/9950X3D2, `hugepage-bench` 3472 MiB): alloc+zero 470 -> 125 ms, and the server's prompt-cache update for a 100K-token slot saving into a new 3.4 GiB buffer 828.1 -> 497.0 ms (-40 %); the recycled-buffer case is unchanged (292.6 -> 291.0 ms). Nothing is copied differently, so no text or logit can move. The submission's other half -- taking the batched state path above `LLAMA_STATE_ASYNC_MAX_MIB` by cutting a piece at a byte offset -- is deliberately NOT taken: a cut piece is not a window the meta backend accepts, so a 2-device `-sm tensor` 27B at `-c 266240` with a 250K-token slot aborts on the save (`GGML_ASSERT(size % chunk_size_full == 0)`, ggml-backend-meta.cpp:2523) where the release path completes with identical logits; and on this box it is not a win even single-device (324 MiB state: save 51.9 vs 32.0 ms, restore 53.9 vs 23.1 ms; 3.47 GiB at 100K tokens: save 278 vs 304 ms, restore 281 vs 242 ms -- the pieces of a whole-slot state are large enough that the per-tensor path already runs at 8-14 GiB/s). See WORKLOG 261029.004.

fix publish permission

584add5

gjmulder added enhancement New feature or request good first issue Good for newcomers build Compilation issues labels Mar 17, 2023

j-f1 mentioned this pull request Mar 18, 2023

Docker container repository permissions are causing access denied error #263

Closed

try to fix docker pipeline using as password github_token & username …

98b31b7

…repository_owner

j-f1 approved these changes Mar 20, 2023

View reviewed changes

bernatvadell merged commit 0f1b21c into ggml-org:master Mar 20, 2023

Deadsg pushed a commit to Deadsg/llama.cpp that referenced this pull request Dec 19, 2023

Merge pull request ggml-org#235 from Pipboyguy/main

69f9d50

Decrement CUDA version and bump Ubuntu

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

Docker - Fix publish docker image in GitHub Registry (ggml-org#235)

30c583d

* fix publish permission * try to fix docker pipeline using as password github_token & username repository_owner

ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026

Docker - Fix publish docker image in GitHub Registry (ggml-org#235)

97e63ce

* fix publish permission * try to fix docker pipeline using as password github_token & username repository_owner

pwilkin pushed a commit to pwilkin/llama.cpp that referenced this pull request Oct 30, 2026

Merge pull request ggml-org#235 from halo-box/fix/qsa-image-recover

e383595

qsa : keep the incremental QSA prefix across mrope images (rebase of ggml-org#163)

Navigation Menu

Repository navigation

Docker - Fix publish docker image in GitHub Registry - #235

Docker - Fix publish docker image in GitHub Registry#235
bernatvadell merged 2 commits into
ggml-org:masterfrom
bernatvadell:fix/docker-publish

bernatvadell commented Mar 17, 2023 •

edited

Loading

Uh oh!

j-f1 commented Mar 17, 2023 •

edited

Loading

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

bernatvadell commented Mar 20, 2023

Uh oh!

j-f1 left a comment

Uh oh!

bernatvadell commented Mar 20, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bernatvadell commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

j-f1 commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

bernatvadell commented Mar 20, 2023

Uh oh!

j-f1 left a comment

Choose a reason for hiding this comment

Uh oh!

bernatvadell commented Mar 20, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bernatvadell commented Mar 17, 2023 •

edited

Loading

j-f1 commented Mar 17, 2023 •

edited

Loading