sentencepiece bpe compatible tokenizer by eiz · Pull Request #252 · ggml-org/llama.cpp

eiz · 2023-03-18T03:05:18Z

I believe this largely fixes the tokenization issues. The example mentioned in #167 as well as my local tests (e.g. "accurately" should tokenize as [7913, 2486]) are fixed by it. Have not tested extensively though, especially with Unicode.

I saw some discussion around file format updates so just take this as an rfc, I just hacked something in

sorry if my coding style is not to your liking ;)

slaren · 2023-03-18T04:10:12Z

Breaks quantize.cpp currently, needs to update the tokenizer part to add the score.

eiz · 2023-03-18T04:19:49Z

doh, thanks for pointing that out, I've only been using fp16 =) will fix.

j-f1

Neat! I think this is pretty close but the Unicode handling isn’t quite right. In particular I don’t believe the tokenizer should be UTF-8 aware, since LLaMA should be perfectly capable of handling invalid UTF-8 strings. It seems to operate on the byte level so I believe this PR as-is will prevent characters that are not in the token dataset from being tokenized. Unrecognized characters are currently represented using their UTF-8 bytes as separate tokens.

eiz · 2023-03-18T04:47:47Z

The handling of UTF-8 here is exactly the same as SentencePiece does. Multi-byte characters that don't form tokens will be output one byte at a time.

j-f1 · 2023-03-18T04:53:31Z

That’s what happens when I do code review at 1am! Everything looks great now (but it’s still 1am so I am not going to approve it until tomorrow morning when I can take a proper look)

ggerganov · 2023-03-18T17:07:40Z

@eiz
Before you merge this, add temporary notice at the top of the README to regen models (like this: 8a01f56)

Also, let's start bumping the magic when the ggml models change:
https://github.com/ggerganov/llama.cpp/blob/master/convert-pth-to-ggml.py#L96

Or add a version ggml version number. Whichever you prefer.
Check during loading and if it is not correct - print message asking the user to regen the models.

P.S. Need a few more days before I start looking into details, so appreciate all the help from the collaborators so far

slaren · 2023-03-18T17:26:40Z

The tokenization look great, I couldn't find any differences with the original llama tokenizer.

Ronsor · 2023-03-18T18:53:06Z

@ggerganov I would suggest a version number. That allows for better error messages like version unsupported versus something like invalid model file.

eiz · 2023-03-18T19:20:56Z

"why not both?"

changed file magic so existing unversioned files don't misparse (ggml -> ggmf "gg model file")
now a version number in the header

j-f1 · 2023-03-18T20:16:25Z

Suggestion: move the format_version to a shared header file of some sort, and then say (unsupported version 2, expected 1)

ggerganov · 2023-03-19T18:44:56Z

@eiz Apologies for the convert-pth-to-ggml.py refactoring - please resolve the conflicts and merge when you are ready

bakkot · 2023-03-20T01:45:21Z

Good news: I ran this using the scoring logic in #270 and saw an improvement in perplexity for the 7B FP16 on wikitext-2/test from 10.4625 before this PR to 5.8149 after. That's a huge improvement.

* potential out of bounds read * fix quantize * style * Update convert-pth-to-ggml.py Co-authored-by: slaren <2141330+slaren@users.noreply.github.com> * mild cleanup * don't need the space-prefixing here rn since main.cpp already does it * new file magic + version header field * readme notice * missing newlines

…org#252) PTQ1_0 has no float mat-vec shader of its own, so it falls back to the generic mul_mat_vec.comp, which reaches it through dequantize()/dequantize4() and calls ptq1_0_trit() once per weight: an 8-bit load plus a loop of up to four multiplies to skip the earlier trits in the same byte. Every packed byte is re-read and re-decoded five times. Since ggml-org#238 an integer-dot mat-vec (mul_mat_vecq_ptq1_0.comp) covers devices with VK_KHR_shader_integer_dot_product, and ggml_vk_should_use_mmvq() picks it for k >= 2048 on AMD and NVIDIA, so on those GPUs this shader is not on the hot path. It still matters for devices without integer dot support, for k < 2048, and whenever MMVQ is declined or disabled. mul_mat_vec_ptq1_0.comp reads each 28-byte block as seven 32-bit words and peels the five trits off a byte with the base-3 recurrence two bytes at a time in 16-bit lanes (255*3 = 765 stays inside a lane), which is 10 multiplies per 20 weights instead of about 60. Consecutive bytes of a word map to consecutive elements for a fixed trit, so each trit step consumes one vec4 of activations. (trit - 1) is split into sum(trit*y) - sum(y) so the row-independent term is gathered once per work item. A work item is half a block so rows of 5120 still spread across the workgroup, and the loop order (word, row, column) keeps the register footprint independent of NUM_COLS. Measured on an RX 6700M (RDNA2, Windows, Vulkan 1.4.357) with Ternary-Bonsai-2-27B-PTQ1_0, -ngl 99 -fa 1 -ctk q4_0 -ctv q4_0, against an unmodified build of the parent commit with the same toolchain, runs alternated to cancel thermal drift: GGML_VK_DISABLE_MMVQ=1 (this shader on the hot path) tg64 3.50 -> 10.44 t/s (2.98x) default (MMVQ active, this shader off the hot path) tg128 24.6 -> 24.8 t/s (unchanged, within noise) pp512 71.0 -> 70.4 t/s (mul_mm untouched, within noise) test-backend-ops -b Vulkan0: MUL_MAT 28/28 and MUL_MAT_ID 75/75 pass for type_a=ptq1_0; the full suite is 17187/17190 and the 3 GATED_DELTA_NET failures also fail on an unmodified build (36/39 both). Perplexity over a fixed 512-token sample is identical to the unmodified build: 8.6924 +/- 1.43177.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. Its third patch (a pass gathering small GDN projections into the row concat) is NOT taken: it needs a quantized <=256-row weight and skips Q8_0, and in every GGUF we have the GDN ssm_alpha / ssm_beta are F32 (the UD quants) or Q8_0, so it cannot fire -- measured neutral to -0.7 % on the 35B on both arches.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm. The gate/beta fusions' unit oracle covers both weight types at the delivered models' contraction length (r202, issue ggml-org#265 follow-up): tests/test-fusion-gate-beta-multiseq.cpp runs {BF16, 1024}, {Q8_0, 1024} and {Q8_0, 5120} over the same 11 (T,S) shapes = 33 cases, because BF16-at-1024 alone is neither the type nor the shape any delivered GDN model runs (blk.N.ssm_alpha/beta.weight is Q8_0, K=5120) and that blindness is what let the softplus rounding difference ship for ten releases. 33/33 bit-identical and all fused on gfx1100 and gfx1201.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@4096

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 and q4_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - The dense 1-token row concat across a K-quant / Q6_K weight pair (PR ggml-org#260): mul_mat_vec_q_ksplit_cat2 runs the Q6_K body (with that segment's own row stride) for attn_v's row blocks and the launch type's body for the rest, for the K-quant "M" mixes. GGML_CUDA_MMVQ_ROW_CONCAT_MIXED=0 is the kill switch. The ksplit body lives in mmvq-ksplit-body.inc and is #included into both, so the pre-existing kernel's source is unchanged: as a __device__ body it changed the compiled code of the multi-column verify kernels and cost 6-7 % of MTP decode on gfx1201 (TODO ggml-org#128). - The q4_0 KV cache write fold (PR ggml-org#262): with -ctk/-ctv q4_0 the rotated K and V rows are written by the same fused rope + FWHT kernel that already served a q8_0 cache, through ggml_cuda_quantize_q4_0_group (one block from the 32 lanes of a wave, bit-identical to quantize_f32_q4_0_block: the scan's FIRST element of largest |v|, never a NaN, vmax = +0 for an all-zero group). The quantizer is a compile-time choice (the kernels' kv template value 0/1/2 -- a runtime select cost the q8_0 kernel 12.8 -> 15.1 us), wave32 only, and GGML_CUDA_KV_FOLD_Q4_0=0 is the kill switch. Worth +10.7 % tg128 / +10.2 % @4096 on gfx1201, +12.0 % / +11.3 % on gfx1100 and +2.6 % on gfx1151 on a q4_0-KV 30B-A3B, with f16 and q8_0 KV unchanged; bit-identical (the --kv q4_0 --ub 1 prefill-logit A/B reads -0.000000 / 100.000 %, the width probe matches at P=3584 and P>=4096, and greedy text is byte-identical on all three arches). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

eiz force-pushed the mack/sentencepiece-bpe branch 6 times, most recently from c67e5ef to 448f398 Compare March 18, 2023 04:03

eiz force-pushed the mack/sentencepiece-bpe branch from 9711125 to de29a00 Compare March 18, 2023 04:23

slaren reviewed Mar 18, 2023

View reviewed changes

Comment thread convert-pth-to-ggml.py Resolved

j-f1 reviewed Mar 18, 2023

View reviewed changes

eiz force-pushed the mack/sentencepiece-bpe branch from 51d3b4a to 0acb5f5 Compare March 18, 2023 12:54

Ronsor approved these changes Mar 18, 2023

View reviewed changes

j-f1 reviewed Mar 18, 2023

View reviewed changes

thement mentioned this pull request Mar 19, 2023

Commit c9f670a (Implement non-greedy tokenizer that tries to maximize token lengths) breaks llama? #280

Closed

ggerganov approved these changes Mar 19, 2023

View reviewed changes

ggerganov mentioned this pull request Mar 19, 2023

Improve Alpaca integration to match it's trained prompt syntax #302

Closed

bakkot mentioned this pull request Mar 20, 2023

Compute perplexity over prompt #270

Merged

eiz force-pushed the mack/sentencepiece-bpe branch from 5d54356 to 649cee5 Compare March 20, 2023 10:03

eiz merged commit 074bea2 into ggml-org:master Mar 20, 2023

eiz mentioned this pull request Mar 20, 2023

move file magic/version to header, print expected version #319

Merged

Navigation Menu

Repository navigation

sentencepiece bpe compatible tokenizer - #252

sentencepiece bpe compatible tokenizer#252
eiz merged 1 commit into
ggml-org:masterfrom
eiz:mack/sentencepiece-bpe

eiz commented Mar 18, 2023 •

edited

Loading

Uh oh!

slaren commented Mar 18, 2023

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

Uh oh!

j-f1 left a comment

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

j-f1 commented Mar 18, 2023

Uh oh!

ggerganov commented Mar 18, 2023

Uh oh!

slaren commented Mar 18, 2023

Uh oh!

Ronsor commented Mar 18, 2023

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

j-f1 Mar 18, 2023

Uh oh!

ggerganov commented Mar 19, 2023 •

edited

Loading

Uh oh!

bakkot commented Mar 20, 2023 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

eiz commented Mar 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slaren commented Mar 18, 2023

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

Uh oh!

j-f1 left a comment

Choose a reason for hiding this comment

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

j-f1 commented Mar 18, 2023

Uh oh!

ggerganov commented Mar 18, 2023

Uh oh!

slaren commented Mar 18, 2023

Uh oh!

Ronsor commented Mar 18, 2023

Uh oh!

eiz commented Mar 18, 2023

Uh oh!

j-f1 Mar 18, 2023

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Mar 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bakkot commented Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

eiz commented Mar 18, 2023 •

edited

Loading

ggerganov commented Mar 19, 2023 •

edited

Loading

bakkot commented Mar 20, 2023 •

edited

Loading