Added install instructions for latest python3 by Reelix · Pull Request #182 · ggml-org/llama.cpp

Reelix · 2023-03-15T21:32:10Z

Added install instructions for the versions of torch and sentencepiece missing from the pip repo on the latest python3 - Used to get this working on Python 3.11.0

Added install instructions for the versions of `torch` and `sentencepiece` missing from the pip repo on the latest python3 - Used to get this working on Python 3.11.0

Better future-python support instructions.

bitRAKE · 2023-03-15T21:46:19Z

Presently, this is the only way to get sentencepiece working on Windows with Python 3.11 without building from source - which is many more hurdles.

prusnak · 2023-03-18T21:42:11Z

This seems really hack-ish. It seems better to just explain that people are better of using Python 3.10 - see 1e5a6d0

* Slightly faster AVX2 implementation for q4_k_r4 * Even better AVX2 implementation for q4_k_r4 We now arrive at PP-512 = 328 t/s for LLaMA-3.1-8B on a Ryzen-5975WX CPU, up from 291 t/s when I last measured on 3c5f872. With FA and Q8_0 K-cache we get to 339.5 t/s. * Fix llama-bench labels that I broke with ggml-org#181 * Faster AVX2 implementation for q5_k_q4 We arrive at 302 t/s for LLaMA-3.1-8B on a Ryzen-5975WX CPU, up from 273 t/s. * Use AVX2 implementation of q4_k_r4 and q5_k_r4 also on Zen4 After the changes I made to AVX2, it ends up being slightly faster compared to what I had for Zen4. * Minor tweak * Cleanup --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

…ill-common-batch examples : port speculative-prefill to common_batch

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call.

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com>

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change).

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change).

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005).

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11).

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11).

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. Its third patch (a pass gathering small GDN projections into the row concat) is NOT taken: it needs a quantized <=256-row weight and skips Q8_0, and in every GGUF we have the GDN ssm_alpha / ssm_beta are F32 (the UD quants) or Q8_0, so it cannot fire -- measured neutral to -0.7 % on the 35B on both arches.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

@slavap

…rastructure) The infrastructure half of the old block 10. qwen4exp, QSA and the hyper-connection ops now live in their own block (11); this block is what they are built on, and it no longer references them. Contents: - The movable-boundary slab and its arena (ggml-cuda-slab.cu, ggml-cuda-arena.cu, ggml-cuda-vmm.h, ggml-cuda-moe-cache-hooks.cu): one VMM reserve per device split by a movable boundary -- compute below, MoE arena above, no HIP call at runtime. Plus the narrow-band learning, the wide-phase auxiliary region, pool-wide routing, and the slab transient absorption. - The fusion dispatcher (ggml-cuda-try-fuse.cu): the matcher arms moved out of ggml-cuda.cu, with every arm implemented inside the kernel TU that owns it. - The async graph-input ring: batched uploads + lazy done event (PR ggml-org#182), the r90 snapshot fix (the scheduler's set_tensor_async path is used for a CPU split's output, which the next ubatch rewrites), default-on, and the per-origin (per host thread) ordering state of issue ggml-org#201. - Rope/FWHT fusions: M-ROPE and IMROPE in the RMS_NORM + MUL + ROPE (+ FWHT) matcher, BF16 SET_ROWS kept off the 5-op fusion, the q8_0 K/V cache writes folded into the rope + FWHT kernel, and the paired GDN q/k norms. - GDN / attention-gate decode fusions: the BF16 alpha/beta pair with softplus(alpha + dt) * a in one mul_mat_vec_f launch (with the r108 multi-sequence fix and its model-free regression test), the q/k norm pair in the direct conv kernel, the attention gate's row-strided view read in place by the gate -> Q8_1 kernel, the GDN output norm inside the gate kernel, and GGML_CUDA_GDN_TAIL (the sequential tail that writes the rollback snapshots). - MMB and the HC16 BF16 streams: the per-context MMB state (two contexts interleaving must not share the activation cache or the marks), the BF16 activation copies, DOWN16, and the graph-optimize marking passes. - The MoE/MMQ decode fusions: mmvq-moe (the dedicated MoE multi-token kernel), the gate+up pair fold, silu(gate)*up into the mmq quantize, the dual-output matmul, the MoE down fold, and the weighted-down reduction. - The verify-band arch extension: RDNA3_0 joins RDNA4 in the Q8_1 pre-quantize predicate and in the direct GDN conv. - Meta/scheduler/allocator: the split-state refusals for a straddling split, the two-pass graph-optimize protocol (allocs_only / marks_only), graph-memory generations so a graph is recaptured when the pool / FA staging / H2D ring memory it captured is freed -- including the slab work-boundary shrink of issue ggml-org#187 -- the KV prefill pad (LLAMA_KV_N_PAD_PREFILL), and the FA accuracy instrument (LLAMA_TEST_OP_NMSE / LLAMA_TEST_SEED). - bf16 RoPE destinations (TODO ggml-org#83): the missing NORMAL/NEOX bf16 instantiations, so -ctk bf16 -ctv bf16 prefill does not abort in rope.cu. Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16 and the GDN_DBG_SKIP_* diagnostics once into values rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call overwrites, so a cached pointer is unsafe on Windows. Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper. Split note (r109): this commit plus block 11 carries exactly the tree the old block 10 carried -- the net set is unchanged. What moved to block 11 is the qwen4exp model graph, the QSA/indexer ops and their kernels, the hyper-connection ops, the managed lazy reader and the keys-only indexer cache, i.e. everything that exists only because of that architecture. The pieces of the old block 10 that exist for qwen4exp but are implemented inside a general kernel TU (the GDN gate/beta fusion, the HC16 MMB BF16 streams, the qwen4exp weighted-down reduction, the PLE conv fusion) stay here with the TU that owns them; the HC_BLK16/RES16 marking pass and the indexer relu-sum alloc-dep moved to block 11 with the matchers they call. Windows HIP build (issue ggml-org#217): the tables-registration slack guard uses the int64_t min/max helpers block 09 defines instead of `std::min<int64_t>`/`std::max<int64_t>`. With HIP on Windows (clang + the MSVC STL) clang's force-included `cuda_wrappers/algorithm` and the MSVC STL `<utility>` each define an inline `std::min<long long>` (`int64_t` is `__int64` there, and the `enable_if` that makes clang's version win overload resolution is not part of the mangled name), so a translation unit that instantiates both fails to codegen with "definition with same mangled name". Behaviour is unchanged. Amended (r132): the block-10 arms of the Q8_1 cache fix that block 08 introduces. The fused GLU -> Q8_1 site's key walk stops at the first pointer change (the fused write is keyed by the matmul's src1, and must not take the root's entry for an offset view), and both of this block's gated-q8_1 launch arms -- the fused norm+gate kernel and the GLU write -- validate their sources with q8_1_src_range_ok() before launching. From PR ggml-org#218 (@slavap); the rule and the validation are recorded with block 08. Co-authored-by: Slava P. <slavap@users.noreply.github.com> Amended (r171): the pair fusion's MMB stand-down asks the dispatch, not the weight type (TODO ggml-org#115). `mul_mat_q_pair` stood down for any weight type MMB supports, a superset of the shapes MMB actually claims, so on a shape MMB declines (T < mmb_min_t(), n_used > 64, a K that is not a whole super-block count, ...) the pair had already stood down and the mmq/mmvq triple took the gate+up pair instead. It now stands down on the predicate the mmb dispatch uses (both muls claimed), which is a strict SUPERSET of the fused-GLU predicate, so the priority chain is well defined and cannot preempt an MMB claim: MMB's fused GLU -> the mmq/mmvq triple -> this pair (the fallback for shapes none of them takes) -> the two plain muls. The routed SWIGLU -> MMQ matcher's stand-down is the same exact claim (its GLU output can be marked BF16-only for an MMB reader). GGML_PAIR_ROUTED_OFF is the new A/B arm for the routed half; GGML_PAIR_OFF still means both halves. Measured neutral: gfx1151 pair vs triple (LFM2 Q6_K 3379/3399 vs 3348/3407, 35B-A3B Q4_K_M 2304/2312 vs 2296/2313 t/s at pp512, interleaved r=3, dense MMB path untouched) and byte-identical greedy text on pair == triple == unfused == GGML_CUDA_MMB=0 (LFM2 765 chars, 35B-A3B 723 chars); gfx1201 inert (routed = 0 there, so the pair already won before this change). Fold (r174, PR ggml-org#249, briansp2020): the M-RoPE section power is computed once instead of once per section arm, in `rope_multi` and in the M-RoPE arm of the fused `rms_norm_mul_rope_f32`. Every arm multiplied its position index by the same `powf(theta_scale, iw/2)`, and adjacent lanes take different arms (`sector % 3`), so the divergent arms were serialized and `powf` executed once per arm taken. It is hoisted before the branch: same input, same value, so the result is bit-identical (the non-M-RoPE path and the "no arm matched, `theta_base` stays 0" case are untouched). Measured on this box, gfx1201: `rms_norm_mul_rope_f32` 3.20 -> 2.85 us (__half dst) and 3.06 -> 2.70 us (f32 dst) per launch, greedy text byte-identical on 4B Q8_0 / 27B UD-Q4_K_XL / 35B-A3B Q4_K_M and the `-ub 1` prefill-logit A/B reads 0.000000 KLD / 100.000 % same-top-p; wall clock is inside the noise on RDNA4 (+0.0 %) and +0.25 to +0.65 % tg on gfx1100 (7900 XTX: 35B-A3B Q4_K_M 137.6 -> 138.6). The submission's other half (a GDN direct conv for 1-token decode) is NOT taken -- it adds a launch per GDN layer wherever the conv's q/k norm fold is refused and measures -1.1 % on RDNA4 dense Q8_0, +0.0 % on the submitter's own 7900 XTX arm (WORKLOG 261029.005). R175 FOLD (PR ggml-org#250, @briansp2020): the seven RDNA3 decode fusions. They sit here because this is the block that owns the fusion dispatcher, the rope matcher and the MoE multi-token kernel (mmvq-moe), so no earlier block can carry them. All default-on, each with a =0 kill switch: - FA combine publishes its output's Q8_1 (GGML_CUDA_FA_COMBINE_Q8): the attention output projection skips its quantize_q8_1 launch. Only for a contiguous F32 output whose row length is a multiple of MATRIX_ROW_PADDING. - MoE gate/up + GLU publishes the down projection's Q8_1 (GGML_CUDA_MOE_GLU_Q8), via self-resetting (slot, 32-row group) counters; the completing block also zeroes the row's padding blocks, as quantize_q8_1 writes them. - The fused K norm+rope+SET_ROWS kernel also writes the V row (GGML_CUDA_ROPE_SET_ROWS_V), so the V SET_ROWS launch goes away. - MoE GEMV takes 4 rows per block on RDNA3_0 when >= 32 blocks per WGP remain (GGML_CUDA_MOE_RPB_WIDE). - MoE down + weighted reduction in one launch (GGML_CUDA_MWR_FOLD): the last expert lane of a row block writes the reduction in moe_weighted_reduction's order and expression, 8 rows per block. - Q and K norm+rope of a layer run as one launch (GGML_CUDA_ROPE_QK_PAIR). The submitter's qwen3moe graph reorder (make the two chains adjacent) is NOT taken: it edits upstream model code for a CUDA fusion and is worth ~7 pts only on qwen3moe (a retired arch family). Without it the pair fires where the graph is already adjacent. - Dense V/K/Q projections in one launch on RDNA3_0 (GGML_CUDA_MMVQ_ROW_CONCAT), a separate row_cat template instance so the plain ksplit kernels are unchanged. - The V / K / Q projections of a Q-norm attention layer are reordered in front of the Q projection by a structural pass in graph_optimize (GGML_CUDA_QK_ROTATE), so the Q GEMV can join the row concat and the Q / K norm+rope pair fires (the allocator no longer hands V / K the Q chain's freed buffers). Same nodes, one stream, every ubatch size, idempotent. - The dense row concat also runs on RDNA4 (GGML_CUDA_MMVQ_ROW_CONCAT_RDNA4). At one token a row's dot product does not depend on the row count there (one warp per row, no small_m), and its two-rows-per-block launch only needs every segment to have an even row count; RDNA3_0 keeps its small_m / rows-per-block rule. Needs the adjacency above. Measured (release tree, 1 GPU, 2-3 alternating rounds, greedy text byte-identical, prefill-logit KLD 0.000000 / 100.000 %): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 172.3 -> 192.8 (+12.0 %), d4096 +11.7 %; dense 27B +0.5 %; gfx1201 Qwen3-30B-A3B tg128 131.4 -> 145.7 (+10.9 %), Qwen3.6-35B-A3B 102.1 -> 104.6 (+2.5 %), dense 27B +0.4 %. Width purity holds (plain == draft-mtp on 27B dense and 35B-A3B), rule-5 batched bench flat, MTP Protocol A unchanged (layer 51.6, tensor 100.5 -> 100.8). Blocks 11-14 re-cut on top; block 11's fattn-qsa.cu call site takes the new combine_results argument (the file is created by block 11). PR ggml-org#252 measured (1 GPU, alternating rounds, -sm layer -t 8, greedy text byte-identical on 30B-A3B / 35B-A3B / dense 27B / Gemma4-31B / Flash-Next -ncmoe 48 / 2-GPU -sm tensor, prefill-logit KLD 0.000000 / 100.000 % at the default ubatch and -ub 1): gfx1100 Qwen3-30B-A3B Q4_K_M tg128 194.3 -> 205.8 (+5.9 %), d4096 170.0 -> 180.2 (+6.0 %); gfx1201 the same model 152.0 -> 160.9 (+5.9 %), d4096 131.2 -> 137.2 (+4.5 %) -- the reorder alone is +3.2 % there and the RDNA4 concat adds +2.6 %. Qwen3.6-35B-A3B +1.8 % on gfx1201, flat on gfx1100; dense 27B flat on both. - The GDN small-projection gather (GGML_CUDA_MMVQ_CONCAT_REORDER) ships since r180. From the first quantized MUL_MAT of <= 256 rows it moves the later MUL_MATs that share that activation, weight type and K (at most one of them larger) directly after it, up to 3 nodes, so the row concat runs the Qwen3.5 / 3.6 GDN ssm_beta + ssm_alpha + attn_gate projections as one launch. It never moves a MUL_MAT whose output feeds a CONCAT within 8 nodes (ggml_cuda_ssm_conv_in_try_fuse skips the nodes up to its CONCAT) and never one with a Q8_0 weight (the SSM prescan matches nodes by position); both guards are load-bearing. Measured on the submitter's file (lmstudio-community Qwen3.6-35B-A3B-Q4_K_M, sha256 4ac6a06b...a410d, whose ssm_alpha / ssm_beta / attn_gate are Q4_K): gfx1201 tg128 129.4 -> 132.9 (+2.8 %), d4096 126.2 -> 129.7 (+2.8 %); 60 fewer launches per token (30 GDN layers x 2, kernel-census verified). Prefill-logit KLD 0.000000 / 100.000 % at the default ubatch, -ub 1 and -ub 8; text byte-identical on 1 / 2 / 3 devices, in both MTP split modes, and on the DFlash2 draft arm. It cannot fire where the GDN small projections are F32 (the UD quants) or Q8_0, so our own GGUF set still measures it as inert. - The 1-token router fold (GGML_CUDA_ROUTER_TOPK, PR ggml-org#253, r182): when a MoE layer's ffn_gate_inp is a 1-token F32 MUL_MAT that the dispatcher would run as mul_mat_vec_f with its 256-thread block, topk_moe_cuda computes the logits itself (the same thread-to-column map, float2 loads, ggml_cuda_mad and warp + cross-warp butterfly, so every logit keeps its bits) and the block that finishes last (a fence, then one atomic on a self-resetting counter in the last MoE-reduction slot) runs the unchanged selection, so the router launch and the topk launch become one. It does not fire with a bias, a delayed softmax, a Hadamard hint or a padded compute-buffer src0, and it steps aside on RDNA4 above 128 experts, where the one launch measured slower than the two. Measured (llama-bench -r 5, two interleaved rounds, 1 GPU, on vs off): gfx1100 Qwen3-30B-A3B tg128 205.8 -> 212.0 (+3.1 %), d4096 180.2 -> 183.1 (+1.7 %); Qwen3.6-35B-A3B (256 experts) 141.1 -> 142.8 (+1.2 %), d4096 137.5 -> 139.0 (+1.1 %); gfx1201 the 128-expert model +1.7 % (noisy) and the 256-expert model flat through the step-aside (fold on there measures -0.3..-0.5 %). Purity: greedy text byte-identical on/off; the decode-band logits (gate-prefill-logits.sh --ab GGML_CUDA_ROUTER_TOPK=0 --ub 1, i.e. every token a 1-token graph with the fold firing) measure 0.000000 KLD / 100.000 % same-top-p on 30B-A3B (gfx1201) and 35B-A3B (gfx1100); test-backend-ops -o TOPK_MOE 608/608; MTP Protocol A unchanged (51.53 t/s, acceptance 0.62134). It is inert with a speculator (a verify batch is more than one token wide) and on every router with one of the excluded shapes. GGML_CUDA_ROUTER_TOPK=0 is the kill switch; a launch aborted mid-kernel would leave the counter non-zero, the same caveat the MoE down fold's counter carries. Amended (r183): PR ggml-org#256's three GDN decode folds plus its shared-expert gate fold, and the 2D-copy guard that fixes a Flash-Next `-sm tensor` load abort. Nothing here changes an arithmetic order. - The direct GDN conv also writes the recurrent snapshot tails (GGML_CUDA_DISABLE_GDN_CONV_SNAP=1 off), so the CONCAT and the snapshot CPYs launch nothing wherever the direct conv runs. It is taken only when nothing between the CONCAT and the conv writes over state / x or touches a snapshot destination and the destinations are disjoint (a runtime byte-span check), and the batched-copy run stops at a folded snapshot CPY so cpy_batch can never swallow one. - The conv-state gather fold (GGML_CUDA_FUSE_GDN_CONV_GATHER=0 off): build_rs's GET_ROWS of the conv cache, feeding a CONCAT that the direct conv takes, is deferred like the GDN state gather and the direct kernel reads the cache row itself (row id = the gather's src[1]), holding each channel's three state values in registers before the snapshot writes because one of them is the write-back to that row. One sequence, at most 16 tokens, the CONCAT the gather's only reader, and only folded snapshot CPYs writing the cache before the conv. A CONCAT over a deferred gather that the direct conv does not take aborts rather than reading an unwritten GET_ROWS (the tripwire is asserted in write_tail and at the CONCAT). - The 1-token direct conv (GGML_CUDA_FUSE_GDN_CONV_DECODE=0 off), the ggml-org#249 re-submission: on RDNA4 / RDNA3_0 a 1-token conv takes the direct path only where it also elides the q/k norm pair (gdn_conv_norm_match() > 0) and where the decode-only ssm_prescan would not take the conv first. The data-dependent gate lives in the compute-time matchers (CONCAT, SSM_CONV, the ssm_conv_in stand-down and both folds), so they all decide the same way; graph_optimize uses the structural match (ggml_cuda_gdn_conv_pattern_at_concat), a superset, so its alloc deps cover every case the compute-time matchers take. The ssm_conv_in epilogue steps aside with `continue`, so it keeps scanning the window, and it keeps its CONCAT write where the direct conv is refused (the dense Q8_0 case that cost -1.1 % in ggml-org#249 is neutral again). - The shared-expert gate fold (GGML_CUDA_FUSE_SHEXP_GATE_SIGMOID=0 off): MUL_MAT(one gate row) -> SIGMOID -> MUL -> ADD runs in the mul_mat_vec_f launch the gate would take anyway (the same instantiation, ncols_dst and block size); every block recomputes the one gate row, and its slice of `y + x*sigmoid(g)` uses sigmoid_mul_add_f32's own op_sigmoid and its `#pragma clang fp contract(off)` expression. BF16 / F32 gate weights only (F16 can take a half accumulator), decode/verify band, and the gate, sigmoid and product may have no other reader. - The 2D memcpy fast path in ggml_cuda_cpy keeps its guard: a 2D copy of a view of a mirrored model weight is refused by the driver with "invalid argument" when the weight sits in the slab region, so the path is only taken while the padded span (height*pitch) fits the tensor's own allocation and the scalar kernels are the fallback (GGML_CUDA_SLAB=0 was the work-around). Measured (all four switches on vs all four off, one build, 1 GPU, llama-bench -r 5, two interleaved rounds; the folds are inert on a multi-device graph, GGML_CUDA_CONV_FUSION_MULTI=1 excepted): gfx1100 Qwen3.6-35B-A3B Q4_K_M tg128 142.3 -> 150.6 (+5.9 %), d4096 139.0 -> 146.9 (+5.6 %); gfx1201 Qwen3.6-35B-A3B 105.0 -> 108.4 (+3.3 %), d4096 +3.4 %; dense 27B UD-Q4_K_XL +1.7 %; Qwen3.5-4B Q8_0 neutral (103.6 vs 104.1, the ssm_conv_in epilogue keeps its CONCAT). Decomposed on gfx1201: 0003 alone +0.3 % (it is the enabler), +snapshot fold +2.2 %, +gather fold +1.0 %. The shared-expert gate fold is inert on every quantized model we hold (its gate weight must be BF16/F32) and measures +0.6 % on Flash-Next GSQ-IQ3_XXS tg128 (45.6 vs 45.3). Purity: greedy text byte-identical (35B-A3B 1 GPU, 4B Q8_0, 2-GPU -sm tensor -ncmoe 40, Flash-Next -sm layer); gate-prefill-logits.sh --ab "<all four off>" --ub 1 (the decode width, where the fold fires) 0.000000 KLD / 100.000 % same-top-p on 27B Q8_0; MTP on 35B-A3B (n_max 7) 182.4 vs 174.7 t/s with identical draft/accept counts (362/203); a --parallel 2 --kv-unified server run is clean (the multi-sequence batch declines the gather deferral). Flash-Next GSQ-IQ3_XXS `-sm tensor -ncmoe 48 -c 4096 -fa 1 -ctk q8_0 -ctv q8_0` now loads and decodes (it aborted on r182 in ggml_cuda_cpy at cpy.cu:479, device 1) and its text matches the GGML_CUDA_SLAB=0 arm.

Reelix added 2 commits March 15, 2023 23:32

Added install instructions for latest python3

ae43513

Added install instructions for the versions of `torch` and `sentencepiece` missing from the pip repo on the latest python3 - Used to get this working on Python 3.11.0

Made the file clearer for future python versions

11b3daf

Better future-python support instructions.

gjmulder added the documentation Improvements or additions to documentation label Mar 17, 2023

prusnak closed this Mar 18, 2023

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

spongioblast mentioned this pull request Sep 23, 2025

Intel Xeon Platinum 8592 Gadflyii/llama.cpp#5

Closed

woct0rdho pushed a commit to woct0rdho/llama.cpp that referenced this pull request Oct 17, 2026

Merge pull request ggml-org#182 from Nathanw1014/fix/speculative-pref…

b703f52

…ill-common-batch examples : port speculative-prefill to common_batch

Navigation Menu

Repository navigation

Added install instructions for latest python3 - #182

Added install instructions for latest python3#182
Reelix wants to merge 2 commits into
ggml-org:masterfrom
Reelix:patch-1

Reelix commented Mar 15, 2023

Uh oh!

bitRAKE commented Mar 15, 2023

Uh oh!

prusnak commented Mar 18, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Reelix commented Mar 15, 2023

Uh oh!

bitRAKE commented Mar 15, 2023

Uh oh!

prusnak commented Mar 18, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants